Characterize Legacy Code Before the LLM Translates It

Posted on Nov 3 • Originally published at chainofthought.show

Legacy modernization is mostly archaeology. A model can read a sprawling, undocumented codebase, explain modules, map dependencies, translate languages, and draft migrations, compressing days of careful reading into a reviewable draft. The catch is the business rules that live outside the syntax: say, an exception that exists because of a contract signed in 2011, the kind of rule that exists for a reason no doc explains. A model reads the code, not the intent, so it can fluently reproduce structure while getting a critical behavior wrong. A subtly wrong migration is worse than an obviously broken one, because it ships.

Rachelle Palmer of MongoDB made the case on Chain of Thought that modernization needs both speed and proof that the new code behaves like the old. You want the LLM to do the slow work (reading, explaining, drafting) and the engineer to own verification and the decisions that require understanding why the old system behaves the way it does. Before any translation starts, pin the behavior you must preserve.

A characterization test captures the current output of a legacy function without judging whether that output is correct by spec. It is a snapshot: given this input, the system returns this result. Once frozen, any change the LLM produces that breaks the test is either a bug in the translation or a real design change that needs a human sign-off.

I would start with discrete legacy functions (a pricing calculator, a formatter, a validation rule) where inputs and outputs are easy to observe, and expect more work on UI-heavy flows or stateful systems without clean boundaries. Pick the functions that encode the business logic you are most afraid of losing.

Write a characterization suite first. Use the model to help draft tests by feeding it the legacy source and asking for representative input-output pairs, but review every one. Then freeze the suite in version control and do not change it during translation. Let the LLM translate, run the suite, and treat any failure as a design decision, not a nuisance to suppress.

In order to prove equivalence between a legacy application and a replacement application, you actually need to generate effectively equivalent tests between your legacy app and your new modernized app.

Rachelle Palmer, MongoDB, on Chain of Thought ep 14

That is the discipline: equivalent tests on both sides, signed off by a human.

Palmer puts the model's share of modernization work at about 80 percent, with the remaining 20 percent purely manual. She does not itemize that 20 percent, and she says equivalent tests can themselves be generated with LLMs. In the workflow I am proposing, the human share includes deciding which behaviors matter, reviewing each failure after translation, and deciding whether a change is acceptable.

I do believe the eighty twenty rule applies here, which is that you can get LLMs to assist with 80, but you still have this remaining 20% that's like purely manual effort.

This is the opposite of vibe coding, where you accept generated code by how it runs rather than by reading it. That loop is fine for a prototype. Code that might still carry an exception from, say, a 2011 contract deserves the frozen suite. The AI coding topic page and the best episodes on AI coding cover what changes when models write more of the code.

This is an illustration of the order of operations, not production code.

# Sketch: freeze, translate, compare legacy_tests = run_characterization ( legacy_module ) new_code = llm_translate ( legacy_module, spec = docs ) new_tests = run_characterization ( new_code ) diffs = compare ( legacy_tests, new_tests ) # Any diff is a design decision. Human signs off. Enter fullscreen mode Exit fullscreen mode I would not run this unsupervised. The sketch shows the order, not the review step. Review every diff.

The longer explainer, with the episode clips, is on Chain of Thought. It draws on this episode.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.