A number is not an evaluation
Reported accuracy on a convenient test split predicts almost nothing about how a model behaves on new subjects, new hardware and conditions nobody sampled.
Published
A model reports 97% on held-out data. The question that matters is not whether the figure is correct — it usually is — but whether the split it was measured on resembles the situation the model will be deployed into. On sensor and scientific data, it frequently does not.
Where the number comes from
Randomly splitting rows is the default, and on physiological or instrument data it leaks. Windows from the same subject, the same session or the same hardware unit land on both sides of the split, and the model is rewarded for recognising the subject rather than the phenomenon. The headline figure then measures something nobody wanted to know.
Splitting by subject, by session, by device and by site gives lower numbers and far more information. If performance collapses when the split moves from row-level to subject-level, that collapse is the finding.
The questions worth answering
A more useful evaluation reports how the split was constructed and why; how performance varies across subgroups and operating conditions rather than only in aggregate; what happens when the input drifts out of distribution; and what the errors cost, separated by type, because a false negative and a false positive rarely carry the same consequence.
Aggregate metrics hide exactly the failures that matter. A system that works well on average and poorly on one identifiable group is not a system that works well.
Calibration and abstention
Where a person will act on the output, a calibrated probability is worth more than a marginally better point estimate, and a defined path to abstain is worth more than both. “I do not know” is a valid output and frequently the correct one. A model with no way to express it will produce a confident answer for an input that resembles nothing it was trained on.
The uncomfortable conclusion
Doing this properly sometimes shows that a learned model is not earning its place — that a signal-processing chain which can be reasoned about, tested exhaustively and explained to a regulator performs comparably and fails more predictably. That is a good result. It is cheaper to find it during evaluation than after deployment.