Your OpenAI Evals freeze on Saturday. Here's how to get them out, and keep testing on any model

OpenAI stops hosting its Evals service. As of Saturday, October 31st, your evals become read-only. As of Monday, November 30th, the Evals Dashboard and Evals API become inaccessible. Also on that day, the Agent Builder and reusable prompts (the v1/prompts API) are discontinued.

If you created test suites in Evals, the former date marks when you will lose the ability to change your evals, and the latter date when you won't be able to access them anymore. Below I describe what you can do and illustrate a way to preserve your evals and their last run together with graded outputs as your own, reusable tests you can run on any model.

Follow the OpenAI's suggestion and migrate to Promptfoo. In case of the Evals shutdown, OpenAI recommends users migrate to its newly purchased open-source evals framework, Promptfoo. The migration process differs depending on eval type and consists in exporting a Promptfoo configuration or rebuilding an eval from scratch by hand. If you use Promptfoo already, this is a viable way to go.

Export the results. You can export the results of a successfully completed run. The results files contain a history and not a test suite: by themselves, exported results cannot be rerun on other models.

Retrieve all information via the API. Up until the shutdown day, the Evals API returns information about your evals, their runs, and every run's output items: input messages you used to query the model and the returned results. Using this information, you can automatically build tests which you could use to validate new models by comparing their behavior with the output messages you already have at hand. Below I describe how it is done with LLMShift, the tool I built for this purpose myself.

Disclosure: I created LLMShift. It is freely available under the MIT license.

You need Python 3.9+ and your OpenAI API key. With uv there's nothing to install:

export OPENAI_API_KEY = sk-... uvx llmshift import openai-evals --api --all --split.llmshift/evals/ Enter fullscreen mode Exit fullscreen mode Or install llmshift with pip and remove the uvx. To pull specific evals, replace --all with --eval-id eval_abc --eval-id eval_def.

For each eval, LLMShift will load evaluation criteria and the latest run's datasource and the run's outputs: the inputs become test cases and outputs are baselines you can use to check whether a model answers differently the next time you run it. --split option generates tests files for every eval.

After the import, a fidelity check re-verifies the recorded outputs against the converted graders and calculates how often their verdict matches OpenAI’s recorded. With a mismatch, you’ll learn of it before you depend on it.

In addition, a few well-liked graders will cease operation. On 23 October, o3-mini, o4-mini and gpt-4.1-nano get shut down. Thus, for a score_model that relies on any of these, a replacement judge has to be assigned during the import.

Your evals are now test files in your repo. Apply them to the new model, be it OpenAI’s or not:

llmshift compare --tests.llmshift/evals/ --to openai:gpt-5.6-sol --judge openai:gpt-5.6-terra llmshift compare --tests.llmshift/evals/ --to anthropic:claude-sonnet-5-5 --judge openai:gpt-5.6-terra Enter fullscreen mode Exit fullscreen mode This report lists all the instances where there is a change: a difference in formatting, incorrect fact(s) output, different tonality, or a tool call. If the new model does not accept the option specified, the instance fails instead of just disappears. (For example, Anthropic’s latest version of its API does not accept temperature, top_p and top_k arguments at all).

Should cost-efficiency be your selection criterion, llmshift pick will apply several models against the same set of tests and recommend the most cost-effective one which passes them. (That way, you would save monthly costs).

llmshift init --github writes a config file and two workflows: checks on pull requests, and a nightly watch that alerts you when the live model starts answering differently. Or call the GitHub Action from your own workflow:

- uses: DeChuks-hub/llmshift@v0 with: command: compare args: --tests.llmshift/evals/ --to openai:gpt-5.6-sol --min-pass-rate 0.95 env: OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} Enter fullscreen mode Exit fullscreen mode Prior to the weekend, prior to 30 November 1) This week: run llmshift import while you can verify all in the dashboard and commit the test files, 2) Prior to 30 November: run llmshift compare against the next model you will be using, and fix anything that changed ( llmshift fix recommends adjustments to prompts), 3) If you were using reusable prompts (located in v1/prompts ), it is advised by OpenAI to include prompt text in your codebase. Do that, then let llmshift check test prompts for each pull request.

Is my data sent somewhere with llmshift? No – except when you specify a provider to llmshift compare. The import fetches eval data from the OpenAI API via your API key, and everything written is stored in your repository.

How about evaluations created within the open-source openai/evals repository? They also work with LLMShift: llmshift import openai-evals path/to/evals/registry.

Could I import an exported JSON file, rather than fetch via the API? Sure thing: llmshift import openai-evals eval.json --data items.jsonl.

What happens to a grader that cannot be imported? It is listed with the reason of being unconvertible in the import results summary. Nothing will be discarded.

Create an issue on GitHub. I will solve it before the shutdown.