Running two classifiers on the same input produces a tempting monitoring signal: how often do they agree? It is easy to calculate, available without immediate human labels and useful for finding changes in behavior.

It is not accuracy. Two models can confidently make the same mistake. This article explains how to use agreement without turning it into a guarantee, and how to keep model-quality monitoring separate from the processing measurements in the current streaming experiment.

The missing variable is the answer

Let A(x) and B(x) be two model predictions. Agreement asks whether:

A(x) == B(x)

Accuracy requires a label y(x):

A(x) == y(x)

The first expression contains no evidence about the second. Different architectures can reduce some shared failure patterns, but they can still share training data, label definitions, preprocessing errors or a systematic blind spot.

Here is a deliberately small example:

Input Model A Model B Reviewed label
1 sport sport technology
2 sport sport technology
3 technology sport technology

The models agree on two of three inputs, and both agreed answers are wrong. Treating agreement as a confidence score would prioritize the wrong cases for automatic acceptance.

Disagreement is still useful

Disagreement creates a queue for investigation. It can reveal ambiguous examples, a changed category distribution, a preprocessing bug or one model behaving differently after an update. The signal is a reason to inspect, not a diagnosis by itself.

Keep the model versions, preprocessing version and input identifier with the comparison. Otherwise a jump in disagreement may be difficult to distinguish from a deployment that mixed old and new predictions.

A cross-tabulation of two models’ labels is also not a ground-truth confusion matrix. It shows where the models disagree with each other. Use names that make that distinction visible on dashboards and in alerts.

Confidence scores need their own evaluation

One model reporting 0.95 while another reports 0.70 does not prove the first is more likely to be correct. Confidence scales can differ across architectures and training procedures. Calibration asks whether scores correspond to observed correctness on representative labeled examples.

Without that evaluation, rules such as “prefer the more confident model” simply replace one unverified assumption with another. Even calibrated scores can become unreliable when the input population changes.

An operational policy might instead send selected disagreements for review, use a conservative fallback for high-impact cases, or delay action until a label arrives. The appropriate policy depends on the cost of the error, not only the model scores.

Sample agreements as well as disagreements

If every reviewed example comes from the disagreement queue, shared mistakes remain invisible. Include a representative sample of agreements, and avoid reporting review-set accuracy as population accuracy when the sampling strategy is biased.

Useful dimensions include category, language, input length and source. A stable global agreement rate can conceal degradation in a small but important segment. Track coverage as well: missing predictions, rejected inputs and timeouts should not silently disappear from the denominator.

When delayed labels become available, connect them back to the predictions made at the time, including their versions. Re-running a newer model on the input answers a different question from evaluating the original decision.

Quality and freshness are separate axes

A correct prediction can arrive too late to be useful. A fast prediction can be wrong. Measuring one does not establish the other.

The maintained Spark experiment deliberately uses one tiny, fixed classifier. It records ingress and processing timestamps while varying input pacing and an injected processing delay. The fixture labels verify that the expected path executed; four training texts do not establish useful generalization.

That experiment avoids mixing model disagreement with queueing behavior. Its companion article explains the measured backlog and source-admission limit. The older dual-model application remains in repository history, but the current branch should not be described as running a production disagreement-monitoring platform.

A practical monitoring split

Signal Useful question Additional evidence needed
Agreement rate Did the models’ relationship change? Representative reviewed labels
Confidence distribution Did score behavior change? Calibration and population context
Label distribution Did predicted categories shift? Source population and actual labels
Prediction age Are results arriving late? Stage timing and offered/processed rates
Missing-result rate Are some inputs excluded? Errors, timeouts and filtering policy

Schema checks can protect the structure of these records, and feature-input invariants can catch semantic changes before inference. Neither establishes that a classification is right. Use the data-contract example for that separate input boundary.

The strongest use of agreement is modest and practical: it helps decide where to look. Keep it beside labeled evaluation, coverage and freshness, and it becomes a useful operational signal without pretending to supply the ground truth you do not yet have.