Ninety-eight per cent of a data programme happens before the dashboard
Institutions ask for a dashboard and discover they have bought a question. Here is the order of work that actually produces a number somebody is willing to sign.
Read
Applied AI
What a validation set looks like when the training data was collected somewhere else
An imported crop classification model reported 94 per cent accuracy and was wrong about a third of the fields we checked. The gap was not the model. It was the ground truth.
In short
- Accuracy quoted by a vendor is accuracy on their validation set, not yours.
- Ground truth collected in one agro-ecological zone rarely transfers to another.
- Budget for field validation before you budget for the model.
- A model that fails honestly is more useful than one that fails quietly.
A government agency asked us to review a crop classification system it had procured. The documentation reported 94 per cent overall accuracy. Field officers using it reported that it was frequently wrong. Both statements turned out to be true.
Where the number came from
The 94 per cent was measured against a validation set drawn from the same collection campaign as the training data: the same season, the same districts, the same handful of field teams using the same protocol. Under those conditions the figure is real. It is also close to meaningless as a prediction of performance anywhere else.
Three things differ once the model crosses a boundary. The spectral signature of a crop shifts with soil background, planting density and the timing of the rains. The class definitions drift, because what one survey calls intercropped maize another splits into two classes. And the imagery itself differs in atmospheric correction and in how much cloud the compositing step had to work around.
What we measured instead
We built an independent validation set: 740 field polygons across four districts, collected with GPS by officers who had not seen the model output, covering the crops that actually matter to the agency rather than the ones that classify easily.
Against that set, overall accuracy was 61 per cent. More usefully, the confusion was structured rather than random. Two classes accounted for most of it, and both were cases where the original class definition did not match local practice.
What we did about it
We did not throw the model away. We re-labelled two classes to match how the agency actually records them, retrained on a mixture of the original data and the new local polygons, and re-ran the measurement. Accuracy came to 83 per cent, which is a figure the agency can plan around because it knows exactly how it was produced.
The more important output was not the model. It was a validation protocol the agency now runs itself every season, so the next time performance drifts, somebody notices.
The general point
Every accuracy figure is a claim about a specific distribution. When a system crosses a border, the distribution changes and the claim expires. Re-measuring is not a sign that something went wrong with the procurement. It is the ordinary cost of deploying a statistical system somewhere new, and it belongs in the budget from the start.
