Dialect Recon

An agent is given the manual for a small programming language, forty example programs with the exact output each one produced, and a button. The button runs any program the agent writes and shows what came out. It can be pressed two hundred times (eighty, on half the tasks).

The button does not run the language in the manual. It runs a dialect: the same language, changed in four to sixteen places. Lists might start at 1. print might be spelled say. A function might remember its variables as they were when it was written, not as they are when it runs. Nobody tells the agent which.

The agent must write an interpreter that matches the button byte for byte on three hundred programs it never sees.

This is Dialect Recon, an environment we built to train and measure one skill: working out how a system really behaves when its documentation is wrong. We ran two frontier coding agents on it. For each hidden change in each task they attempted, we checked whether their final interpreter got it right.

Source: our measurement. A change that appears in a task attempted three times counts three times.

Both agents read the evidence in front of them perfectly. Neither was good at going out to get the evidence that was not there. That gap is what Dialect Recon measures, and it is what this post is about: how the environment works, what we found and fixed while building it, and what the agents did.

IReading versus asking

Writing an interpreter for a documented language is no longer hard for a frontier agent.

• Given only the README of MNM Lang, a language invented in March 2026, Claude Code (Claude Opus 4.6) solved all 26 of its challenges (Acher, 2026).
• On esoteric languages such as Brainfuck, frontier models writing code directly score 0 to 11% (Sharma & Chopra, 2026). Coding agents that can run their code solve 64 (Claude Opus 4.6) and 79 (GPT-5.4) of the same 80 Brainfuck problems (Sharma, Thorat & Chopra, 2026, Table 1).

When the rules are written down, the agent's job is to follow them, and agents are now very good at that.

Real engineering often gives you a different job. An old interpreter has corner cases nobody wrote down. A vendor's SQL is almost standard SQL. The only specification of a network protocol that matters is the server on the other end. The documentation is a starting guess and the running system is the truth. The skill is noticing where the guess is wrong, designing the experiment that pins down the difference, and fixing the code without breaking what already works.

Research shows models struggle with this, though always with some help:

Each study tells the model what changed, hands it the new rules, or hides something small. Two recent benchmarks ask agents to rebuild whole programs from their behaviour (MirrorCode, ProgramBench), but those programs are real and public, so a model may partly remember them.

Dialect Recon removes all of that help. The rules of a whole language are hidden behind a black box, experiments are limited, and grading uses programs the agent has never seen.

The idea itself is older than language models. In programming-language courses at Brown, students are given "mystery languages": variants of a language that differ in hidden ways, which they must tell apart by writing experiments (Pombrio, Krishnamurthi & Fisler, SNAPL 2017). Dialect Recon asks the same thing of an agent, at a scale of four hundred languages and with an exact grader.

IIFour hundred languages nobody has seen

Every dialect starts from one small base language: integers, strings, booleans, nil, lists, functions with closures, if, while, for, and print. A program ends normally (exit code 0), prints <<error>> and exits with code 1 on a runtime error, or exits with code 2 if it does not parse. The manual describes all of it.

A dialect is made by turning dials. There are 48 of them in eight families, and each changes one rule of the language:

The scope dials have a research pedigree. Lu and Krishnamurthi collected the misconceptions students hold about variables, mutation, scope and closures, and turned each one into a misinterpreter: an interpreter that runs code the way a student with that misconception expects (OOPSLA 2024, Distinguished Paper). Several of our dials are those misinterpreters written as settings: variables passed by reference, one global scope for everything, let that reassigns instead of declaring, closures that copy their environment. These are the assumptions a careful engineer makes without noticing, and here each one can be the truth.

A dialect turns 4 to 16 dials (median 10, our measurement over all 400), taken from at least four families. The environment has 400 dialects. None existed before we generated it, so no model can have seen one in training. Each also has an exact answer key: the dialect is whatever our reference interpreter does with those dials set.

Most programs cannot tell a difference

Take the division dial. These three programs look like fair tests of it:

print 6 / 3;      # 2 in both languages
print 7 / 2;      # 3 in both languages
print -7 / 2;     # -3 in the base language, -4 in the dialect

Only the last one can tell the two languages apart. Rounding only matters when the result is negative and inexact. The same is true of almost every dial: it changes behaviour only in a narrow corner.

That is why our first attempt, generating programs at random, failed. 88 to 94% of random programs printed exactly the same output with a dial on or off (our prototype measurement). They almost never divided a negative number, read past the end of a list, or captured a loop variable in a closure.

So every program is written to ask a question. For each dial we wrote small templates that reach its corner, and mixed them into random, well-typed code. Then we test each program: run it on the dialect, then again with one dial switched back to the base language. If the output changes, the program is sensitive to that dial, like the -7 / 2 line above. Every dial in every dialect controls at least 20 of the 300 graded programs (our measurement), so each dial an agent misses costs a known number of points.

The dials also interfere. With "lists start at 1" switched on, our closure templates produced no sensitive program in 30,830 tries (prototype measurement). They indexed lists from 0, crashed on the first index, and never reached the closure they were written to test. The fix was to render every template in the dialect's own rules. We also found 13 pairs of dials that hide each other's effects, and those pairs are never drawn together.

IIIHiding the evidence

For each dialect we generate 60 example programs and show the agent 40. The 40 are chosen so that two to four of the dialect's changes never appear in any of them. Those changes cannot be read off the examples. The agent has to suspect them, write a program that would reveal them, and spend a button press on it.

Here is a change the examples do show. One dialect gives the agent an example much like this:

let q = [14];
print q[1];
print 2 * 3 + 4 * 5;

The manual predicts an error on line 2, since a one-element list has nothing at index 1. The dialect prints 14 and then 70. Two changes are visible: lists start at 1, and + binds tighter than *, so the last line means 2 * (3 + 4) * 5.

Here is a change the examples in that same dialect never show:

let r = 1;
fn k() {
  return r;
}
r = 5;
print k();

The manual says this prints 5. The dialect prints 1, because its functions snapshot their variables when they are defined. None of the 40 examples does anything like this. The only way to find out is to wonder about it and ask.

Hidden changes are always about meaning, never spelling. A renamed keyword shows up in almost every example, and a blind guess at one gets back only "parse error", which teaches nothing.

Where the points are

For every dialect we measure how far each kind of knowledge gets you on its 300 graded programs:

Source: our measurement. Trusting the manual scores nothing. Reading the examples perfectly scores about half. The rest is only reachable by experiment. We call the middle number the examples-only score, and every point above it is discovery.

Figure 1. Each column is one dialect. Dark: what the manual alone earns. Mid: what the examples add. Light: what only experiments can reach.

Why a list of dials is not enough

With a fixed menu of 48 dials, an agent could try to learn the menu and test each dial once. In our prototype, a script that knew every dial did exactly that: one test per dial, and a perfect score on every dialect.

Then we gave it a dialect with 16-bit integers, a width it had never seen. Its single width test was designed to tell 32-bit, 64-bit and unlimited integers apart, by adding numbers near two billion. In a 16-bit language, numbers wrap around much earlier: print 30000 + 30000; prints -5536, not 60000. The answer matched none of the options the script knew, so it picked the nearest one, 32-bit, and every program that touched a number above 32,767 came out wrong. Its score fell to about 120 of 300 (prototype measurement).

So 15 dial settings are reserved for the dialects we evaluate on, including keywords such as puts and val, a nil that prints as <nil>, and division by zero that returns its left operand. An agent that has memorised the menu should fail on them. An agent that knows how to experiment should not.

IVAn exact grader

The reward is one number: how many of the 300 hidden programs the agent's interpreter reproduces exactly, output bytes and exit code both, from 0 to 300. There is no fuzzy matching and no judge model. A program matches or it does not.

Because every dial controls its own set of programs, the score rises in steps as the agent discovers the language. Miss one dial and you lose exactly the programs that depend on it. That also lets us take any submission and say which changes it found and which it missed, which is how the tables in this post were made.

The button reports what happened, never why. It returns the program's output and exit code. A runtime error comes back as <<error>> and code 1, with no message. If it answered "index 3 is out of range", the agent could learn the indexing rule from the error text, and we would be measuring reading again. Reporting only behaviour forces the agent to reason from behaviour.

The dialect runs outside the agent's machine, and the agent has no network. On every task we check the extremes before trusting any score: the reference interpreter scores 300, an empty submission scores 0, and an interpreter that follows the manual scores close to 0.

VWhat testing found

Three findings changed the environment more than any other.

The score depended on the machine. Our first design limited each program's wall-clock time. On a busy grading machine a correct interpreter could run out of time, so the same submission could score differently on two runs. Worse, a submission that hit the limit on every program could hold the grader for 340 programs × 60 seconds, almost six hours, longer than the agent's whole episode. We removed wall-clock time from scoring entirely. Each program now gets 2 seconds of CPU time, which does not change with machine load.

The button could stop working. The language allows up to 200 nested function calls. The service behind the button used a lower Python recursion limit than our reference interpreter, so a program recursing 199 levels deep, well within the rules, crashed it for the rest of the episode. Agents test documented limits, so this was common: the button died in 13 of 22 graded Sonnet runs and 5 of 34 Codex runs (our measurement). We gave the service the same limit as the interpreter and made it survive any request, then replayed all 136,000 graded programs in the dataset through it. Every one returned its recorded answer. The agent results below were measured before this fix, so they are a lower bound.

What we tested was not what users download. Our tests ran each task from our own copy. The public registry delivers files differently, and from there the grader failed before it started. Every check had passed and nothing worked. We now run every check on the downloaded task, exactly as users get it.

VIAttacking our own grader

An exact grader is only useful if nothing but a correct interpreter can earn its reward. So as part of building each task, a separate red-team process tries to earn reward without solving it. It writes submissions that do nothing, submissions that guess, and submissions that try to learn more about the hidden dialect than the rules allow. Each attempt is run against the real grader and scored. Its claims are never taken on trust.

Three findings stand out.

1. Some trivial strategies fail by design. A submission that prints the same thing every time fails admission, because the 40 examples have at least 35 different outputs. A submission that memorises the 40 example answers passes admission but scores almost nothing, because no example program appears among the 300 graded ones. Neither needed a fix. They fail because of how the data is built.

2. A perfect score cannot be told apart from a perfect shortcut. A red-team attempt counts as an exploit only if it scores more than the honest reference solution. We added that rule after another environment, where attempts that did the real work and added a useless extra step tied the reference and were wrongly flagged as cheating. The rule works almost everywhere. Dialect Recon is the exception, because the reference already scores the maximum, 300. A shortcut here can at best tie it, and a tie is invisible to the rule. We did not replace the rule with an automatic guess. Any attempt that exactly ties the reference is now flagged for a person to review.

3. The red team has to be tested too. Continued testing found a route through the grading boundary that could reach the maximum score without interpreting any program. It had been missed because an earlier round of red-team attempts had been written to a location the grader never read. Those attempts were recorded as failures, but they had never actually run. We closed the route: untrusted code now runs with no access to anything but the one program it is given. We fixed the routing so every attempt is graded for real. Then we re-checked everything. The shortcut now scores 0, the reference still scores 300, and all 34 agent submissions we re-graded scored exactly what they scored before. None of the agent runs in this post used the route.

The lesson from all three: a grader is software, and it needs its own adversary for as long as the environment exists.

VIIResults

We ran both agents on the same ten tasks, drawn with a fixed seed, with up to four attempts per task. Claude Sonnet 5 ran through Claude Code, GPT-5.6-terra ran through the Codex CLI at its highest reasoning setting. Every number below is our measurement from the raw runs. We re-graded all 53 finished submissions ourselves and every one reproduced its recorded score exactly.

For comparison, on these ten tasks the manual alone scores 4.6 on average and the examples alone score 129.3. Both agents clear the examples-only line on average, and both reach a perfect score at least twice, so the task is solvable from scratch in two hours. Sonnet scores higher on average and higher on 8 of the 10 tasks.

Figure 2. Each row is one task. Grey tick: manual-only score. Dark tick: examples-only score. Dots: individual runs, with ×n where several runs scored the same.

Side by side, task by task

Source: our measurement. Tasks are ordered by their examples-only score.

Three patterns stand out.

Nobody beat Task 1. Both of its hidden changes (ranges that include their end value, and arguments passed by reference) raise no error and are never hinted at. All six runs, from both models, scored exactly 108.

Codex repeats itself. On Task 7 it scored 175 four times, on Task 1 108 four times, on Task 6 169 three times, and in each case the replay shows the same changes found and missed. Sonnet varies more between attempts on one task (Task 3: 114, 204, 300), which gives it more chances at a perfect run.

A single experiment decides large gaps. On Task 2 the only Codex run that divided by zero scored 214, and the two that never tried scored 110. On Task 10 one extra probe of a top-level return separates 217 from 149. The difference between runs is rarely skill at interpreting. It is whether the agent asked one specific question.

Which changes get found

Both agents found every change that the examples show, in every finished run (one Codex run implemented one of them only partly, described below). Every change they missed completely was one the examples hide. Among the hidden ones, a clear split appears:

Figure 3. For each hidden change, the share of runs that implemented it correctly. Labels show found of total.

How agents find hidden changes explains the split. The manual lists its runtime errors in one table, and the most productive habit in both models' transcripts was to run each listed error through the button and compare. A dialect change that turns one of those errors into a normal result (division by zero returning a value, a top-level return that ends the program quietly, comparing a number with a string giving false) gets found this way.

The changes found least often raise no error at all. Arguments passed by reference (0 of 3 for Sonnet, 0 of 7 for Codex), lists copied when passed to a function, a string index returning a character code, arguments evaluated right to left. Nothing in the manual or the examples points at them. To find one, an agent has to imagine it first.

VIIIHow each model works

We read every transcript: 22 graded Sonnet runs and 36 Codex runs. Both models follow the same broad arc (read, compare, build, test, stop), but they order it differently and spend their experiments very differently.

Source: our reading of every transcript. Probe counts are exact where the agent printed the server's remaining budget, and counted from commands elsewhere.

Figure 4. Minutes each finished run worked before declaring itself done. The limit was 120.

Both stop far too early. No run used more than 51 minutes of 120, and no run used its whole probe budget. The median Sonnet run used a third of it, the median Codex run a sixth. Codex runs never discussed how many hidden changes might remain, although the instructions say there are between four and sixteen. Several Sonnet runs misread what is graded: the instructions say the withheld programs are checked, but not that the score counts exact matches on them, and some runs concluded that only crash-avoidance was scored and stopped as soon as the 40 examples passed.

The best runs compare, the worst runs assume. In every perfect run the agent sent questions about meaning (evaluation order, scope, comparisons) to the button and to its own interpreter, side by side, and treated every disagreement as a lead. The runs stuck on the examples-only score either never asked the question that mattered, or asked it only of their own interpreter.

Moments from the transcripts

Each block below is copied from the recorded transcript. Times are seconds from the start of the run.

cat > /tmp/e14.src << 'EOF'
print 1 < 2 < 3;
EOF
python3 /tmp/probe2.py /tmp/e14.src
echo "=== mine ==="
python3 submission/interp.py /tmp/e14.src; echo status=$?
--- result ---
false
exit_status=0 remaining=177
=== mine ===
<<error>>
status=1

fn P(tag) { echo tag; return tag; }
fn two(a, b) { return a; }
echo two(P(1), P(2));
echo [P(3), P(4), P(5)];
...
--- result ---
1
2
1
5
4
3
[3, 4, 5]

let fs = [];
for (i in 0 .. 3) { fn g() { return i; } push(fs, g); }
print fs[0]();
print fs[2]();
--- result (probe, then own interpreter) ---
{"remaining": 150, "status": 0, "stdout": "2\n2\n"}
0
2

print 200 / 3;
print 200 % 3;
--- result ---
STATUS=0 REMAINING=48
-19
-2

for f in e1 e4 e5 e8; do echo "=== $f ==="; python3 t/myprobe.py t/$f.src; done

print 1 < "a";
=== cmpmix ===
<<error>>
status=1

print "abc"[-1];
print "abc"[-4];
--- output
<<error>>

sed -i '180c\  if i<0 and type(a)is list:i+=len(a)' submission/interp.py

print 1 / 0;
---
REF
1
OURS status=1
<<error>>

let a = 1;
if (true) { let a = 2; }
print a;
---
2

let a = [1, 2];
a[1] = 9;
print a;
a[-1] = 8;
print a;
{'remaining': 71, 'status': 1, 'stdout': '[1, 9]\n[8, 9]\n<<error>>\n'}

fn f() { print 7; return 1; }
print 1;
return f();
--- probe vs ours
1
7
[ours]
1
<<error>>
status=1

IXDid either model try to cheat?

No. We searched every command, every file read and every file written in all 62 transcripts (26 Sonnet, 36 Codex) for any attempt to reach the hidden programs, the dialect's settings, the reference interpreter, the grader or the network, or to influence the score. We found none in either model.

The few commands that looked outside the task folder were diagnostic: locating the task directory, checking why ./probe would not run, and checking whether the button's service was still alive after it stopped answering. Some runs wrote their own client for the button's socket to see the exit code and remaining budget. That uses the documented interface, and the service still enforced the budget. Every submission we examined is a genuine interpreter that reads only the program it is given.

Takeaways

1. Hide the rules, grade on unseen inputs. A generated language with an exact grader cannot have been seen in training, and its answer key is never wrong. The difficulty comes from which evidence is withheld, not from the language.
2. Measure the floors first. The manual-only and examples-only scores give every other number its meaning. Without them, 150 out of 300 could be excellent or could be nothing.
3. Frontier agents read well and ask poorly. Both models found every change the examples show and most of the changes the manual's error table points to. Both missed most changes that nothing hints at.
4. One question often decides a run. A single division by zero, a single top-level return, a single side-by-side comparison separated runs by 68 to 151 points.
5. Agents stop when they feel done. No run used half its time, and the median run used a third (Sonnet) or a sixth (Codex) of its experiment budget. The behaviour worth training is simple to state: keep experimenting until the evidence, not the feeling, says you are finished.
6. A grader needs its own adversary. The most serious issues we found were in the grading machinery, not the language, and they were found by continuing to attack it.

References

• Acher, M. (2026). Can Coding Agents Program in M&Ms Language? blog.mathieuacher.com
• Lu, K.-C., & Krishnamurthi, S. (2024). Identifying and Correcting Programming Language Behavior Misconceptions. OOPSLA 2024. doi:10.1145/3649823
• Pombrio, J., Krishnamurthi, S., & Fisler, K. (2017). Teaching Programming Languages by Experimental and Adversarial Thinking. SNAPL 2017. doi:10.4230/LIPIcs.SNAPL.2017.13
• Sharma, A., & Chopra, P. (2026). EsoLang-Bench: Evaluating Genuine Reasoning in Large Language Models via Esoteric Programming Languages. arXiv:2603.09678
• Sharma, A., Thorat, S., & Chopra, P. (2026). Frontier Coding Agents Use Metaprogramming to Adapt to Unfamiliar Programming Languages. arXiv:2606.10933
• Thimmaiah, A., Zhang, J., Srinivasa, J., Li, J. J., & Gligoric, M. (2025). LLMs Lean on Priors, Not Programming Language Semantics (PLSemanticsBench). arXiv:2510.03415
• Wei, A., et al. (2025). CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesis. arXiv:2503.23145
• Wu, Z., et al. (2023). Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks. NAACL 2024. arXiv:2307.02477
• Yin, C., et al. (2025). Investigating Advanced Reasoning of LLMs via Black-Box Environment Interaction (ORACLE). arXiv:2508.19035
• Adamczewski, T., et al. (2026). MirrorCode: AI can rebuild entire programs from behavior alone. arXiv:2606.30182
• Yang, J., et al. (2026). ProgramBench: Can Language Models Rebuild Programs From Scratch? arXiv:2605.03546