If you are the person who spend days tweaking prompts and then test with three examples before shipping, you're basically flying blind.
Evals are the difference between an LLM demo and an actual product.
Without it, you can't reliably tell if your system got better or worse after a change.
What is "evals as code"
It's pretty simple.
Keep evals in your repo. Review them like any other code. Give them owners. If they fail, your CI should fail too.
Write evals for the stuff that's actually hard. The easy cases don't tell you much. The valuable ones are usually the weird edge cases that someone with domain knowledge thought of.
Run them before deployment and after deployment. CI catches regressions. Production catches drift.
A minimal example
There are plenty of eval frameworks. While none of them solve everything, the structure is usually pretty similar.
const suite = evalSuite({
cases: loadCases("./evals/cases.jsonl"),
runner: async (input) => {
const out = await model.complete(input);
return grade(out, input.expected);
},
thresholds: { passRate: 0.95, p95LatencyMs: 800 },
});
The cases are just JSONL. One test case per line.
That makes it easy for non-engineers to contribute. Someone can put examples into a spreadsheet, export them, and you're done.
If the pass rate drops below your threshold, the build fails.
Production drift is the real problem
Passing CI doesn't mean your app is still working well a week later.
Things change.
- Users start asking questions you never tested.
- The model provider updates a model without much warning.
- A prompt change quietly doubles your costs.
- Weird long-tail inputs start producing garbage.
That's why production evals matter.
Sample real traffic. Grade it in the background. Watch the metrics over time.
A right mindset is to treat evals as part of your product itself.