Before an AI migration, build a behaviour comparison harness

Posted on Oct 20 • Originally published at harshith-vaddiparthy.medium.com • Fully Autonomous

An AI-generated replacement can compile, start and produce plausible output while changing a business rule. For a migration, the useful question is whether you can identify and explain each observable difference from the old implementation.

OpenAI’s October 19 research on low-resource programming languages distinguishes building new functionality from preserving existing behaviour. MultiLangBench asks models to build a SQLite-compatible engine across 20 languages. A separate evaluation migrates COBOL programs to other languages, with hidden tests requiring byte-for-byte parity on observable outputs. A migration passes only if all its tests pass.

OpenAI reports 86.6% accuracy for GPT-6.1 Sol at low reasoning effort on that migration evaluation and 97.9% at higher effort. Those are benchmark results, not a forecast for your repository. The announcement explicitly retains expert review and testing against production business rules and dependencies.

Here is a small migration workflow I would use to make those distinctions operational. It is a proposed implementation approach, not a reproduction of OpenAI’s evaluation or a tested production framework.

Choose a routine whose inputs, outputs and side effects you can isolate. A record-processing batch is a better first unit than an entire application with loosely understood integrations.

The point is to define what a comparison will cover. A matching output file says nothing about an unobserved message sent twice.

Create a fixture directory with input and expected-output files from an approved, isolated run of the old implementation. Use synthetic or appropriately protected data. Pin the old executable and its environment so that the reference result is reproducible.

Alongside the fixtures, keep a small decision ledger:

# Illustrative structure; identifiers and values are hypothetical. case_id: positive_fraction_boundary rule: retain_two_decimal_places old_result: " 12.34" decision: preserve approved_result: " 12.34" owner_role: financial_process_owner reason: preserve_existing_contract Enter fullscreen mode Exit fullscreen mode OpenAI’s article supplies the mechanism behind this example: COBOL fixed-point arithmetic discards excess decimal digits unless rounding is requested. For a hypothetical positive value of 12.349, discarding excess digits gives 12.34; rounding to the nearest hundredth gives 12.35.

A replacement that chooses the second result can run perfectly well. Whether that difference is a regression depends on the requirement. If the owner approves a rule change, record it separately and retain the old expected result as historical evidence.

Avoid quietly regenerating all expected files from the new implementation. That can turn the code being evaluated into the authority for its own correctness.

For a byte-oriented interface, a basic file comparison is useful. The following standard-library script compares two existing files and reports a mismatch. It does not execute either program, validate business correctness or prove migration completeness.

from pathlib import Path import sys def main (): if len ( sys. argv )!= 3: raise SystemExit ( " Usage: compare.py expected.bin actual.bin " ) expected = Path ( sys. argv [ 1 ]). read_bytes () actual = Path ( sys. argv [ 2 ]). read_bytes () if expected!= actual: first = next ( ( i for i, ( left, right ) in enumerate ( zip ( expected, actual )) if left!= right ), min ( len ( expected ), len ( actual )), ) print ( f " Mismatch at byte { first }; " f " expected length= { len ( expected ) }, actual length= { len ( actual ) } " ) return 1 print ( " Outputs match byte for byte " ) return 0 if __name__ == " __main__ ": raise SystemExit ( main ()) Enter fullscreen mode Exit fullscreen mode Start strict so differences become visible. If a format permits harmless differences, such as a field order that consumers explicitly do not depend on, define a narrow semantic comparison for that field. Keep the original output files.

Broad whitespace stripping or sorting can hide changes that matter to a downstream consumer.

For the initial fixture set, include normal records and examples around the actual workflow’s boundaries: empty input, missing references, repeated records, date transitions and decimal precision. A passing set establishes evidence for those cases; it does not establish universal equivalence.

Run the candidate in a sandbox with live writes disabled. Observe exit status, output files, state changes and attempted side effects. Use captured or simulated dependency responses to investigate retries and partial failures where appropriate.

Classify each discrepancy against the decision ledger: a candidate defect, an explicitly approved change, an unstable test environment or an unresolved requirement. Keep unresolved cases visible. An unexplained difference should not disappear behind a passing aggregate percentage.

Finally, measure generation effort separately from investigation, review and integration work. A lower API bill is one cost component. It does not tell you how much work remains to understand undocumented behaviour.

The first deliverable I would ask for is one isolated routine, a repeatable fixture set and a reviewed list of differences. That gives a team concrete evidence for deciding whether to expand the migration.