Stages small enough to test
A pipeline can have correct components and still produce the wrong answer. The boundaries deserve their own tests.
Design note
Splitting a system into stages makes its behavior easier to inspect. It also introduces boundaries where assumptions can drift apart. Both effects belong in the design.
A useful decomposition gives each stage a limited responsibility and makes the information crossing its boundaries explicit. Passing the stage tests is necessary. It is not the same as demonstrating that the whole system works.
A hypothetical document pipeline
Imagine a small document search system with four stages: ingest, chunk, embed, and retrieve. This is an illustrative architecture, not a description of a search service running on this site.
Ingest identifies a document and records its version. Chunk turns that version into passages. Embed associates each passage with a vector from a particular model. Retrieve uses a query representation to select passages from a compatible index.
Suppose a revised document arrives while an earlier version is still being processed. Every stage might return a valid-looking result, yet retrieval could combine passages from both versions. A test showing that chunking returns nonempty strings would miss the problem entirely.
The missing property is coherence across the pipeline.
Make the boundary carry the assumption
A passage should travel with enough identity to answer: which source, which version, which chunking procedure, and which position in that source? A stored vector should also identify the embedding model or configuration that produced it.
These fields are not decorative metadata. They let the next stage reject an incompatible input instead of silently interpreting it under different assumptions. They also give a human a route back from a poor answer to the material that caused it.
The contract should define failure as clearly as success. An empty document, an unsupported format, and an interrupted read are different observations. Returning the same empty list for all three makes downstream reasoning harder.
A queue changes the failure model
A queue can absorb a temporary difference in processing speed. It cannot solve an indefinitely slower consumer. If input continues to arrive faster than embedding completes, the backlog still grows.
It also introduces delivery questions. If a worker finishes an embedding but crashes before acknowledging the job, a retry may repeat the work. The receiving stage needs a policy for recognizing the same item, replacing an obsolete version, or accepting an intentional recomputation.
That policy belongs in the contract. “Retry on failure” is incomplete until the system defines which effects are safe to repeat.
Test the joins and the journey
Start with a fixture for each stage: known input, expected output, and explicit failure cases. Then add boundary tests that send duplicate, stale, missing, and incompatible records through the joins.
Finally, test the complete task. Can a query recover the right passage from the current document, and can that passage be traced back to its source? Can the system explain an unavailable result without presenting an old one as current?
A useful comparison would vary the decomposition while holding the documents and queries fixed. Measure detection, recovery, latency, and operational complexity. More stages may improve isolation while making coordination worse.
The point of a small stage is that its behavior can be understood. The point of the system is that those understood behaviors still compose into something useful.