OpenAI's COBOL Migration Score Jumped From 5.4% to 86.6%. The Tests Are Hidden

You hand a COBOL program to a model and ask for Java. The Java compiles. The tests you wrote pass. Then one batch run pays a customer one cent more than the mainframe would have, because the old code dropped the third decimal and the new code rounded it.

That cent is the bar OpenAI set in a post published on October 19 about coding in COBOL, Fortran and Visual Basic. In its COBOL migration eval, a migration only counts if every hidden test produces byte-for-byte the same observable output as the original. On that eval, OpenAI reports that GPT-6.1 Sol at low reasoning effort passed 86.6% of tasks, against 5.4% for GPT-5.6 Sol at the same setting ( OpenAI, TokenPost ). At high effort it reached 97.9%.

Those are large numbers from OpenAI's own evals, and the migration tests aren't published. Only one of the two evals is something you can rerun.

The post is a capability update, not a new product. Runtime Wire describes it the same way, and OpenAI's own wording is that you "can try our latest models through the API on coding tasks in the languages your team uses."

OpenAI frames the work around languages that run critical systems, citing COBOL's share of banking, card and ATM transactions and Fortran's use in weather forecasting and nuclear safety analysis ( OpenAI ).

On MultiLangBench at maximum reasoning effort, OpenAI reports these average test pass rates across the 20 languages ( OpenAI; the GPT-5.6 Sol and Astra averages and COBOL cells also appear in TokenPost ):

COBOL is Astra's lowest-scoring language, at 72.1% ( OpenAI ). Luna is the cheap option: OpenAI estimates $0.77 per sample against $3.48 for GPT-5.6 Sol, with much lower scores than Astra.

Price matters here because the two headline models sit far apart. GPT-6.1 Sol lists at $2 per million input tokens and $10 per million output. GPT-6 Astra lists at $10 and $50. For the migration eval, OpenAI says GPT-6.1 Sol ran at 61% of GPT-5.6 Sol's estimated API cost ( OpenAI ). TokenPost notes that this figure covers the eval only, not the cost of moving a production system.

MultiLangBench is on GitHub under Apache-2.0. Its README says the model must implement SQLite 3.40.1 in the requested language, with SQLite's documentation as the reference. The tests were converted from SQLite's TCL suite, about 24k of roughly 51k. Each test is an initial database, a user query and a validation query.

A few rules in the README are worth knowing before you compare your numbers with OpenAI's:

The score is a pair, API cost and test pass rate, so a cheaper model with a lower score is not automatically worse for your budget ( MultiLangBench ).

The migration eval is the half you can't rerun. Runtime Wire calls them "OpenAI-designed hidden tests," and the GitHub repo covers only the SQLite build task. So the 86.6% is OpenAI's measurement against OpenAI's own hidden tests. OpenAI also says migration output still needs expert review and testing against business rules and system dependencies before it goes to production ( OpenAI ).

What you can reuse from OpenAI's method is the pass rule: a migration counts only when the outputs match exactly. You can build that check on your own code before you trust any model with it.

Collect real inputs, run the old build and the migrated program on each one, and compare the bytes:

import subprocess import sys from pathlib import Path OLD = [ "./legacy/payroll " ] # the existing COBOL build NEW = [ " python3 ", " payroll.py " ] # the migrated program def run ( cmd, data ): result = subprocess. run ( cmd, input = data, capture_output = True, timeout = 60 ) return result. returncode, result. stdout failed = 0 for case in sorted ( Path ( " cases " ). glob ( " *.in " )): data = case. read_bytes () if run ( OLD, data )!= run ( NEW, data ): failed += 1 print ( f " DIFF { case. name } " ) print ( f " { failed } failing cases " ) sys. exit ( 1 if failed else 0 ) Enter fullscreen mode Exit fullscreen mode It compares exit codes and stdout as raw bytes, so trailing spaces, line endings and padded numbers all count. If your batch job writes files instead of printing, compare those files the same way.

The cases that break migrations are the quiet ones. OpenAI's example is COBOL's fixed-point arithmetic, which truncates extra decimal digits unless rounding is requested ( OpenAI ). In Python, the decimal module lets you say which behaviour you mean:

from decimal import Decimal, ROUND_DOWN, ROUND_HALF_UP amount = Decimal ( " 2 " ) / Decimal ( " 3 " ) cents = Decimal ( " 0.01 " ) truncated = amount. quantize ( cents, rounding = ROUND_DOWN ) rounded = amount. quantize ( cents, rounding = ROUND_HALF_UP ) Enter fullscreen mode Exit fullscreen mode Two thirds truncates to 0.66 and rounds to 0.67. A migrated program that reaches for float and round() takes the rounding path, and every test that only checks "close enough" will pass it. Seed your cases/ folder with values like these, plus empty records and maximum field widths. TokenPost makes the same point: changes to rounding or to how missing records are handled can alter a result even when the replacement appears to work.

A COBOL port is long agent work: read the program, write the new one, run the parity script, fix the diffs, run it again. Give it a branch of its own so the legacy tree stays untouched while the agent iterates.

I build Hermes IDE, so weigh this with that in mind. Hermes runs Claude Code, Codex and other command-line agents side by side, each in its own terminal and its own worktree ( README ). ⌘N starts a task on its own hermes/<name> branch, with the task as the agent's first prompt. That lets you hand the same migration ticket to two agents and run the parity script against both branches. I haven't run OpenAI's models on a COBOL migration myself.

Hermes IDE 2.3.4: Codex and Claude Code side by side, each in its own hermes/ worktree, with status, context use, cost and branch in the sidebar.

Hermes is free to download, source-available under BSL 1.1, and runs on macOS, Windows and Linux. It is not affiliated with OpenAI, Anthropic or any agent vendor.

OpenAI is not alone here: TokenPost points to IBM's COBOL-to-Java tool with comparison tests, and to Anthropic's published approach to migrating COBOL with Claude Code. Whichever agent you pick, the parity script decides whether the port ships.

If you own COBOL, Fortran or Visual Basic code, three steps cost little:

Do you have legacy code you would let a model migrate today, if the parity harness came back clean? Tell me in the comments.

A migration like this is an agent loop: write, run the check, read the diff, try again, and stop when the harness is green. The AI Agents Pocket Guide covers the patterns behind that loop, from tool calling to guardrails that keep an agent from declaring victory early. If you want to run two agents on the same ticket, Hermes IDE is free and I build it.