Your eval set is lying to you

Your eval set is lying to you, and it's lying slowly enough that nobody notices until a user does.

Here's the sequence. You build an eval set early, when the product is small and the traffic is imagined rather than real. You tune the model against it. Scores go up. You ship. Then real traffic starts hitting the system, and real traffic is never what you imagined — new phrasing, new edge cases, new ways users ask the same question, seasonal shifts, a feature launch that changes what people even come to ask about. Your eval set doesn't move. It was frozen the day you wrote it. The gap between what it measures and what production actually looks like grows every week, and because the eval score itself stays flat or even improves, nothing on your dashboard tells you the gap exists.

This is the standard failure mode in ML systems, not something unique to LLMs, but LLMs make it worse in two ways. First, the failure surface is language, so a subtly wrong answer often still looks fluent — a bad regression in a classifier throws an obviously wrong label, a bad regression in an LLM throws a confident, well-formatted, wrong paragraph. Second, teams tune directly against their own eval set, which turns the eval into a leaderboard the model quietly learns to game rather than a genuine measure of the task. Goodhart's law applies exactly as advertised: once the eval becomes the target, it stops being a good measure.

None of this shows up as a crash. It shows up as a slow erosion in the thing you actually care about — resolution rate, cost per correct answer, escalations to a human — while the offline number sits there looking fine. That's the quiet part. Nobody gets paged when an eval set drifts. Someone just eventually notices that support tickets are up, or that the "great" model is producing answers users don't trust, and by then the drift has been compounding for months.

Three signs your eval set has drifted, in practice:

• Your eval score has been flat or improving for months while a downstream business metric — resolution rate, cost, complaint volume — has moved in the wrong direction.
• Nobody on the team can tell you when the eval set's examples were last refreshed, or where they came from relative to current production traffic.
• The failures you see in production logs don't resemble the failures your eval set is built to catch. If real users are breaking the system in ways your test cases don't cover, the test cases aren't testing the system anymore.

The fix isn't a better one-time eval. It's treating the eval set as a live artifact that has to be re-sampled from production, not written once and defended forever. That means pulling real queries on a schedule, not just at launch. It means tracking a small number of production proxies alongside the offline score, so a divergence between "eval says great" and "production says worse" gets caught within days, not months. It means being suspicious of any metric that only ever goes up — real systems degrade in real ways, and an eval that never shows it isn't measuring the system, it's measuring itself.

We ran into a version of this building the doubt-solving engine for Utkarsh Classes, an edtech company handling 48 crore student queries a year. The target wasn't "beat a benchmark," it was teacher-level answer quality at production volume, sustained — which only means something if you're checking it against what the system is actually doing in the field, not against a fixed set of sample questions from six months prior. That discipline is also what let us cut the operating cost per query from around 70 paise to roughly 20-22 paise without quietly trading away quality to get there — you can only make that tradeoff safely if your measurement is honest about both sides of it.

This is why we build these systems to run on infrastructure the client owns and monitors end to end — their keys, their cloud, their models, fully documented. An eval set nobody can inspect or refresh is just another black box, and black boxes are exactly what silently drift. If you want a second opinion on whether your production numbers still match what your evals are telling you, that's a short, concrete conversation — not a sales pitch, just a look at the two numbers side by side.