The problem
Most COBOL migration tools translate code. Very few can show that the translation is equivalent. COBOL numeric semantics are subtle: PICTURE clauses, packed decimals (COMP-3), truncation, ROUNDED, ON SIZE ERROR and intermediate precision all change the bytes a program writes.
What I built
- A Python runtime for COBOL data and arithmetic semantics: PICTURE, USAGE DISPLAY / COMP-3 / COMP / COMP-5, MOVE, numeric editing, ROUNDED, ON SIZE ERROR, STRING / UNSTRING / INSPECT. Every rule is compared byte for byte with GnuCOBOL; cases it does not model are refused with an error instead of guessed.
- A differential harness that compiles the COBOL program with cobc, runs it and the Python translation on the same inputs (regression vectors plus Hypothesis-generated ones derived from record layouts), compares stdout and output files byte for byte and reports the first difference by line and field.
- A pre-registered benchmark of LLM-produced translations, measured with the harness.
Results
| check | result |
|---|---|
| Runtime vs GnuCOBOL 3.1.2 | 9,769 generated cases in 33 rules: 9,652 agree, 0 disagree, 117 refused as unsupported |
| Reference translations | 10 of 10 pass regression and boundary vectors |
| Mutation testing | 67 mutants, 9 equivalent; 52 killed, 6 missed |
| GnuCOBOL 3.1.2 vs 3.2 | the compilers themselves differ in 104 cases |
Benchmark: LLM translations of COBOL
Translations were scored on held-out inputs never shown to the model. One call with the runtime library passed 92% of attempts on the external corpus; with up to three repair rounds driven by harness feedback, 100%. All 408 passing translations also pass on GnuCOBOL 3.2.
| condition | held-out pass, external corpus |
|---|---|
| one call, with runtime | 83/90 (92.2%) |
| plus up to 3 repair rounds | 90/90 (100%) |
| one call, stdlib only | 82/90 (91.1%) |
Why it matters
It turns a migration from a leap of faith into a measurable process: every translated program carries a byte-level equivalence report, and every unsupported construct is flagged instead of silently mistranslated.