OpenAI is killing its Evals platform, and if your QA pipeline leans on it, you now have a hard deadline: read-only by October 31, 2026, dead by November 30, 2026. That’s not a far-off research footnote — it’s a migration task that belongs on your roadmap this quarter, before it becomes a fire drill next year.
Why it matters
Evals were supposed to be the thing that told you whether a model change — a new version, a prompt tweak, a provider switch — actually helped or quietly broke your product. If that testing layer lived inside OpenAI’s own tooling, you built a dependency on infrastructure that vendors can retire whenever they want. The lesson generalizes: anything load-bearing for shipping decisions needs to survive a vendor’s product roadmap, not depend on it.
What’s actually changing
- Evals platform retirement: Per OpenAI’s API docs, existing eval content goes read-only on October 31, 2026, and the service shuts down entirely on November 30, 2026.
- Pricing is getting noisier, not simpler. Headline token prices keep falling — some current-generation frontier models list at $0.10 per million input tokens and $0.50 per million output tokens, and independent trackers put a frontier token price index at 16, or 84% below its March 2023 baseline. One tracker even lists a paid API as low as $0.02 input / $0.00 output per million tokens.
- But unit economics are diverging from sticker price. Failed requests, retries, long-context tiers, reasoning tokens, and tool calls are now treated as material cost drivers by vendors and pricing guides alike — not rounding errors.
- Cross-provider eval tooling is gaining ground as teams look for consistent, external grading that doesn’t live inside one vendor’s console.
What this means if you ship product this quarter
Three concrete things change for teams building on LLM APIs right now:
1. Audit where your evals actually live
If your test suite, golden datasets, or regression checks sit inside OpenAI’s Evals UI or API, start exporting now. You have roughly a year before read-only mode and about 13 months before shutdown — that sounds comfortable until you remember migrations always take longer than planned, especially when they touch grading logic your team hasn’t touched in a year. Treat this like any vendor deprecation: inventory what’s stored, what’s custom (graders, rubrics, datasets), and what breaks if the API disappears tomorrow.
2. Move toward model-agnostic QA, even if you only use one model today
The practical fix isn’t finding a replacement vendor eval tool — it’s building (or adopting) an evaluation layer that doesn’t care which provider sits behind your API calls. That means your own dataset of representative prompts, your own scoring criteria, and a harness that can point at OpenAI, Anthropic, Google, or an open-weight model with minimal rewiring. This isn’t just risk management for the next deprecation — it’s what lets you actually A/B test models and catch silent regressions when a provider updates a model version under the same name.
3. Stop comparing providers on headline token price alone
A $0.10/$0.50 per-million-token model can still cost more per completed task than a pricier one, once you account for retry rates, context length needed to get a correct answer, and how many tool calls a task requires. Before your next pricing decision, build a real cost-per-successful-task number, not a cost-per-token one.
| What you’re measuring | What it misses |
|---|---|
| Input/output token price | Retries, failed calls, caching, batch discounts |
| Price per million tokens | Reasoning-token overhead, long-context tier pricing |
| Benchmark leaderboard score | Your actual task distribution and failure modes |
The reliability angle is now a product decision
Fallback planning — what happens when a provider has an outage, deprecates a model, or changes behavior after a silent update — used to be an infra concern. With eval infrastructure itself now provably mortal, it’s a product decision. If you can’t quickly verify that a fallback model performs acceptably on your real workload, you don’t have a fallback; you have a hope.
The honest counterargument
Building independent eval infrastructure is real engineering cost, and for small teams or early-stage products, OpenAI’s Evals might still be the fastest way to get basic testing in place before the shutdown date. Not every team needs multi-provider abstraction today — if you’re deeply committed to one model family with no near-term plan to switch, the migration cost of full model-agnostic QA may outweigh the benefit for now. The pragmatic move is proportional: export your test data regardless, but don’t over-engineer a cross-provider harness you won’t use.
Bottom line
The Evals shutdown is small on its own, but it’s a signal: evals, pricing, and reliability are converging into one problem — knowing what your AI product actually costs and actually does, independent of any single vendor’s tooling. Teams that treat that as infrastructure to own, not a feature to borrow, will spend less time scrambling at the next deprecation notice.