Engineering
Silent Data Pipeline Failures: Detection Patterns That Work
A pipeline that reports success while moving the wrong data is worse than one that crashes. Four detection patterns that catch partial and silent failures before finance does.
A silent pipeline failure is a run that reports success while producing incomplete or incorrect output. Detection requires asserting on the data rather than on the job: volume against expectation, freshness, distribution stability, and reconciliation against the source of truth.
- Job status is not data status. A green run tells you the process exited cleanly, nothing more.
- Assert on volume, freshness, distribution and reconciliation, in that order of cost-effectiveness.
- Alert on absence. Most silent failures produce too little data, not bad data.
Why green runs lie
A scheduled job exits zero. The orchestrator marks it successful. The dashboard is green. And the job processed eleven records instead of eleven thousand because an upstream API paginated differently, a credential expired midway and the error was caught and logged, or a filter inherited a stale date boundary.
Every one of those is a successful run by the only definition the orchestrator has. The failure is in the data, and nothing is looking at the data.
Four assertions worth making
1. Volume against expectation
The cheapest and highest-yield check. Compare row count to a rolling baseline for the same weekday and alert on a material deviation in either direction. Low volume catches truncation and silent credential failure; high volume catches duplicate loads, which corrupt downstream aggregates more quietly than missing rows.
2. Freshness
Assert that the maximum timestamp in the destination is within an expected window of now. This catches the case where the job runs successfully but processes nothing new: common when a watermark fails to advance, which is a failure that can persist for weeks while every run reports success.
3. Distribution stability
Track null rates per column, cardinality of key fields, and the share of rows falling in each category of important enumerations. A column whose null rate jumps from one percent to forty is a schema change upstream, and it is invisible to volume and freshness checks. This is the data layer's version of the schema drift detection worth running against third-party APIs.
4. Reconciliation against source
The most expensive and the only one that is conclusive. Periodically count the source and the destination over the same window and compare. Run it less often than the others, daily or weekly, but run it, because it catches the failures the first three miss.
Volume and freshness together cost very little and catch the majority of silent failures. If you implement nothing else, implement those two and alert on them per pipeline rather than per platform.
Common causes, and which assertion catches them
- Pagination change upstream: volume.
- Expired credential caught and logged: volume, freshness.
- Watermark fails to advance: freshness.
- Upstream schema change: distribution.
- Rate limit truncation under growth: volume, reconciliation.
- Duplicate load after a retry: volume (high side), reconciliation.
- Timezone boundary error: reconciliation.
The organisational half
Detection only matters if the signal reaches someone who will act. The usual failure is that data quality alerts go to a channel the data team mutes, while the people who feel the consequence (finance, operations, support) find out at month end.
Two practices help. Route data assertions to the owner of the integration, not the owner of the orchestrator. And treat manual reconciliation downstream as an unfiled incident, as described in SaaS-to-SaaS breakage.
Correlating a failed assertion with the upstream API error that caused it, and with the vendor ticket raised about it three weeks earlier, is the part that almost never happens manually. It is the specific gap Traxivo closes.
Where to put the assertions
Assertions placed in the wrong layer either miss failures or produce noise. A simple allocation:
- At ingestion: volume against baseline, and schema conformance. Catches upstream changes closest to their source, where the error message is still meaningful.
- After transformation: distribution checks such as null rates and key cardinality. Catches logic errors and silent type coercion.
- At the destination: freshness, and reconciliation against the source. Catches everything the first two missed, which is the point of having it.
Run ingestion and destination checks every cycle. Run reconciliation daily or weekly, since it is the expensive one. Route failures to the owner of the integration rather than to whoever maintains the orchestrator, because the orchestrator is almost never the thing at fault.
One practical note: assert on both the low and high side. A duplicate load after a partial retry inflates counts and corrupts aggregates far more quietly than missing rows, and a one-sided check will never see it.
Frequently asked questions
What is a silent data pipeline failure?
A pipeline run that reports success while producing incomplete or incorrect output, for example processing a fraction of the expected rows because a credential expired midway and the error was caught and logged. The orchestrator sees a clean exit, so nothing alerts.
How do you detect pipeline failures that report success?
By asserting on the data rather than the job: row volume against a rolling baseline, freshness of the maximum timestamp, stability of null rates and cardinality, and periodic reconciliation against the source system. Volume and freshness together are the cheapest and catch most cases.
Why alert on high volume as well as low?
Because duplicate loads, usually caused by a retry after a partial failure, inflate row counts and corrupt downstream aggregates more quietly than missing rows do. A deviation in either direction is worth investigating.
Stop rediscovering the same integration failure
Traxivo correlates the signals your tools already produce into one incident timeline, recognises a recurrence as a recurrence, and drafts the follow-up with the evidence attached. Nothing is sent without a named approver.
See how Traxivo works Browse use cases
