Skip to main content

Batch Assertions

Batch steps consume multiple records from an output topic in a single step and match each one against a corresponding expected file. By default, records and files are matched positionally (record[0] matches file[0], record[1] matches file[1], and so on). You can set batchMatchStrategy to lazy for order-independent 1-to-1 matching without replacement.

Raw batch: Then expected records from files​

Then expected records from files
  | topicAlias | expectedRecordsCount | files                            | expectedRecordKeys | batchMatchStrategy | consumerReadTimeout |
  | orders-out | 3                    | exp-1.json,exp-2.json,exp-3.json | k1,k2,k3           | positional         | 60                  |
ColumnTypeRequiredDescription
topicAliasstring✅Alias of a declared output topic
expectedRecordsCountinteger✅Exact number of records to collect before asserting
filesstring✅Comma-separated list of expected file paths (relative to assets dir), one per record
batchMatchStrategystring❌How collected records are matched against files (and expectedRecordKeys): positional (default) or lazy
expectedRecordKeysstring❌Comma-separated list of expected record keys (one per file in files, or a single key shared across all records in the batch), pre-filtered during polling and asserted alongside each file using batchMatchStrategy and keyMatchStrategy
expectedRecordKeystring❌Optional single key filter applied while collecting records
keyMatchStrategystring❌How expectedRecordKeys or expectedRecordKey is matched (exact, contains, starts_with, ends_with, regex)
consumerReadTimeoutinteger❌Max seconds to wait for all records
consumerDeltaTimeinteger❌How far back (seconds) to seek

Avro batch: Then expected records from files based on schema​

Then expected records from files based on schema
  | topicAlias | expectedRecordsCount | files                    | expectedRecordKeys | excludedKeys | batchMatchStrategy |
  | orders-out | 2                    | exp-1.json,exp-2.json    | k1,k2              | ts,eventId   | lazy               |
ColumnTypeRequiredDescription
topicAliasstring✅Alias of a declared output topic
expectedRecordsCountinteger✅Exact number of records to collect
filesstring✅Comma-separated list of expected JSON file paths
excludedKeysstring❌Comma-separated Avro field names to exclude from all records
batchMatchStrategystring❌How collected records are matched against files (and expectedRecordKeys): positional (default) or lazy
expectedRecordKeysstring❌Comma-separated list of expected record keys (one per file in files, or a single key shared across all records in the batch), pre-filtered during polling and asserted alongside each file using batchMatchStrategy and keyMatchStrategy
expectedRecordKeystring❌Optional single key filter applied while collecting records
keyMatchStrategystring❌How expectedRecordKeys or expectedRecordKey is matched (exact, contains, starts_with, ends_with, regex)
consumerReadTimeoutinteger❌Max seconds to wait for all records
consumerDeltaTimeinteger❌How far back (seconds) to seek

Timeout behaviour​

  • If the consumer collects fewer than expectedRecordsCount records before the timeout, the step fails with:

    Timed out after Nms waiting for 4 record(s), only 2 collected.

  • Increase consumerReadTimeout if your pipeline produces output slowly.

Single-row only​

Unlike single-record assertion steps (see Multi-row DataTables), batch steps accept exactly one DataTable row. That one row already describes an any-to-all match across expectedRecordsCount records collected from a single consumer call, adding more rows would be ambiguous, so a multi-row DataTable here is rejected immediately:

This step only supports a single DataTable row but 2 rows were provided.

If you need several independent batches, or several unrelated single records against the same topic, use separate step invocations, or the multi-row single-record assertion steps (raw matchers →, Avro matchers →) instead.

Batch matching strategies (batchMatchStrategy)​

StrategyAccepted DataTable valuesBehaviour
POSITIONALpositional, ordered, index, strictDefault. Record at index i must match file at index i (and expectedRecordKeys[i] when set).
LAZYlazy, fuzzy, unordered, any_order, any-orderOrder-independent 1-to-1 matching without replacement. Each consumed record claims the first unmatched expected file (and corresponding expectedRecordKeys[j] when set) whose content and key match.
# Lazy / unordered: records arriving as (3, 1, 2) still match files listed as (1, 2, 3)
Then expected records from files
  | topicAlias | expectedRecordsCount | files                            | expectedRecordKeys | batchMatchStrategy | consumerReadTimeout | consumerDeltaTime |
  | orders-out | 3                    | exp-1.json,exp-2.json,exp-3.json | k1,k2,k3           | lazy               | 30                  | 60                |

If your pipeline writes across multiple partitions and does not guarantee record order, set batchMatchStrategy to lazy (or use individual assertion steps per record with expectedRecordKey filtering).

See also​