Dialect Recon

Can an agent learn a language by experimenting on it?

Attoloop ResearchSeptember 2026

An agent is given the manual for a small programming language, forty example programs with the exact output each one produced, and a button. The button runs any program the agent writes and shows what came out. It can be pressed two hundred times (eighty, on half the tasks).

The button does not run the language in the manual. It runs a dialect: the same language, changed in four to sixteen places. Lists might start at 1. print might be spelled say. A function might remember its variables as they were when it was written, not as they are when it runs. Nobody tells the agent which.

The agent must write an interpreter that matches the button byte for byte on three hundred programs it never sees.

This is Dialect Recon, an environment we built to train and measure one skill: working out how a system really behaves when its documentation is wrong. We ran two frontier coding agents on it. For each hidden change in each task they attempted, we checked whether their final interpreter got it right.

Claude Sonnet 5 (high)19 runs
hidden changes the examples show, found
106 / 106
hidden changes only an experiment reveals, found
21 / 48
GPT-5.6-terra34 runs
hidden changes the examples show, found
204 / 204
hidden changes only an experiment reveals, found
25 / 90

Source: saved regrade records and transcript review. A change that appears in a task attempted three times counts three times. Some changes have no isolated test programs, so their classification also depends on the transcript. One example-visible change was only partly implemented.

Both agents found every change shown in the examples, though one implementation was incomplete. They missed many of the changes that required a new experiment. We built Dialect Recon to measure that gap. The runs also exposed problems in the environment that we had to fix before trusting it.


IReading versus asking

Coding agents can do well in unfamiliar languages when they have the rules and can test their code. Given the README of MNM Lang, a language invented in March 2026, Claude Opus 4.6 running through Claude Code solved all 26 of its challenges, with the author pointing out wrong outputs during debugging (Acher, 2026).

On esoteric languages such as Brainfuck, frontier models writing code directly score 0 to 11% (Sharma & Chopra, 2026). Coding agents that can run their code solve 64 (Claude Opus 4.6) and 79 (GPT-5.4) of the same 80 Brainfuck problems (Sharma, Thorat & Chopra, 2026, Table 1).

Real engineering often gives you a different job. An old interpreter has corner cases nobody wrote down. A vendor's SQL is almost standard SQL. The only specification of a network protocol that matters is the server on the other end. In those cases, an engineer has to find where the running system differs from the documentation, test the difference, and account for it in their code.

Related studies have tested how models handle changed or hidden rules:

Study Setting Result
Wu et al., 2023, Table 20 Python with 1-based lists. The model is told about the change. GPT-4 predicting program output, with chain of thought: 73.5% → 24.8%
Thimmaiah et al., 2025 Familiar operators with new meanings. The new rules are supplied. Models that reach 90% on normal semantics drop by up to 40 to 60 points
Wei et al., 2025 (CodeARC) Recover one hidden function by querying it Best of 18 models: 52.7% on anonymized functions
Yin et al., 2025 (ORACLE) Work out black-box environments by interacting o3 exceeded 70% on most easy environments, but fell below 40% on some hard tasks

These studies cover supplied rule changes, hidden functions and black-box environments. Two recent benchmarks ask agents to rebuild whole programs from their behaviour (MirrorCode, ProgramBench), but those programs are real and public, so a model may partly remember them.

In Dialect Recon, an agent has to recover the rules of a whole language from examples and a limited number of experiments. Grading uses programs the agent has never seen.

The idea itself is older than language models. In programming-language courses at Brown, students are given "mystery languages": variants of a language that differ in hidden ways, which they must tell apart by writing experiments (Pombrio, Krishnamurthi & Fisler, SNAPL 2017). Dialect Recon asks the same thing of an agent, at a scale of four hundred languages and with an exact grader.


IIFour hundred generated languages

Every dialect starts from one small base language: integers, strings, booleans, nil, lists, functions with closures, if, while, for, and print. A program ends normally (exit code 0), prints <<error>> and exits with code 1 on a runtime error, or exits with code 2 if it does not parse. The manual describes all of it.

A dialect is made by turning dials. There are 48 of them in eight families, and each changes one rule of the language:

Family Example dial Base language Dialect
Syntax keyword spelling print 5; say 5;
Arithmetic division rounding -7 / 2 is -3 -7 / 2 is -4
Indexing first index [10, 20][1] is 20 [10, 20][1] is 10
Scope when closures read variables reads the current value reads the value at definition
Evaluation order argument order left to right right to left
Types mixed comparison 1 < "a" is an error 1 < "a" is false
Printing boolean form true True
Errors undefined variables an error nil

The scope dials come from work by Lu and Krishnamurthi. They collected the misconceptions students hold about variables, mutation, scope and closures, and turned each one into a misinterpreter: an interpreter that runs code the way a student with that misconception expects (OOPSLA 2024, Distinguished Paper). Several of our dials are those misinterpreters written as settings: variables passed by reference, one global scope for everything, let that reassigns instead of declaring, closures that copy their environment. These let us test whether an agent checks familiar behaviour such as scope and argument passing.

A dialect turns 4 to 16 dials (median 10, our measurement over all 400), taken from at least four families. The environment has 400 dialects. We generated their rule combinations instead of choosing existing languages. That alone does not establish what a model has seen in training. Each dialect has an exact answer key: it is whatever our reference interpreter does with those dials set.

Most programs cannot tell a difference

Take the division dial. These three programs look like fair tests of it:

print 6 / 3;      # 2 in both languages
print 7 / 2;      # 3 in both languages
print -7 / 2;     # -3 in the base language, -4 in the dialect

Only the last one can tell the two languages apart. Rounding only matters when the result is negative and inexact. The same is true of almost every dial: it changes behaviour only in a narrow corner.

That is why our first attempt, generating programs at random, failed. 88 to 94% of random programs printed exactly the same output with a dial on or off (our prototype measurement). They almost never divided a negative number, read past the end of a list, or captured a loop variable in a closure.

For each dial we wrote small program templates that test its corner case, and mixed them into random, well-typed code. Then we test each program: run it on the dialect, then again with one dial switched back to the base language. If the output changes, the program is sensitive to that dial, like the -7 / 2 line above. Every dial in every dialect controls at least 20 of the 300 graded programs (our measurement), so we can measure which programs test each rule.

The dials also interfere. With "lists start at 1" switched on, our closure templates produced no sensitive program in 30,830 tries (prototype measurement). They indexed lists from 0, crashed on the first index, and never reached the closure they were written to test. The fix was to render every template in the dialect's own rules. We also found 13 pairs of dials that hide each other's effects, and those pairs are never drawn together.


IIIHiding the evidence

For each dialect we generate 60 example programs and show the agent 40. The 40 are chosen so that two to four of the dialect's changes never appear in any of them. Those changes cannot be read off the examples. The agent has to suspect them, write a program that would reveal them, and spend a button press on it.

Here is a change the examples do show. One dialect gives the agent an example much like this:

let q = [14];
print q[1];
print 2 * 3 + 4 * 5;

The manual predicts an error on line 2, since a one-element list has nothing at index 1. The dialect prints 14 and then 70. Two changes are visible: lists start at 1, and + binds tighter than *, so the last line means 2 * (3 + 4) * 5.

Here is a change the examples in that same dialect never show:

let r = 1;
fn k() {
  return r;
}
r = 5;
print k();

The manual says this prints 5. The dialect prints 1, because its functions snapshot their variables when they are defined. None of the 40 examples does anything like this. The only way to find out is to write a program that tests when the function reads those variables.

Hidden changes are always about meaning, never spelling. A renamed keyword shows up in almost every example, and a blind guess at one gets back only "parse error", which teaches nothing.

Where the points are

For every dialect we measure how far each kind of knowledge gets you on its 300 graded programs. Across all 400 dialects, the manual alone gives a median score of 0. Applying everything the 40 examples show gives a median of 143. Recovering all the rules gives 300. We call the score from applying the examples perfectly the examples-only score. Going beyond it requires discovering changes the examples do not show.

The examples get you about halfwayMinimum to maximumMedianFull score
Median
Manual onlyRange 0 to 290
With all 40 examplesRange 46 to 172143
All rules recoveredFull score300
0100200300

Hidden programs reproduced (of 300)

Show the data
DialectManual aloneWith the examplesExperiment only
1546254
2067233
3471229
4177223
5080220
6081219
7282218
8083217
9383217
10084216
11084216
12084216
13085215
14485215
15585215
16786214
17087213
18088212
19088212
20290210
21390210
22390210
23191209
24791209
25092208
26292208
27093207
28094206
291394206
30096204
31096204
321196204
33097203
34097203
350100200
360100200
371100200
3815100200
395101199
400102198
410102198
420102198
4310102198
449105195
4515105195
460107193
470108192
480109191
4915110190
5015111189
517112188
520113187
539113187
540114186
551114186
562114186
570116184
580116184
594116184
608116184
610117183
620118182
630118182
641118182
651118182
668118182
6713118182
6825120180
690121179
700121179
7115122178
722123177
733124176
740125175
750125175
763125175
775125175
7816125175
791126174
8015127173
810128172
820128172
835128172
8410128172
850129171
862129171
876129171
8813129171
890130170
900130170
910130170
920130170
930130170
940130170
953130170
969130170
9714130170
980131169
9910131169
10029131169
1010132168
1020132168
1039132168
1040133167
1051133167
1066133167
1070134166
1080134166
1090135165
1100135165
1110135165
1123135165
11315135165
1140136164
1150136164
1164136164
1178136164
1180137163
1190137163
1200137163
1210137163
1220137163
1230137163
1240137163
1250137163
1260138162
1270138162
1280138162
1290138162
1300138162
1310138162
1320138162
1330138162
1340138162
1350138162
1361138162
13716138162
1380139161
1390139161
1400139161
1411139161
1421139161
1433139161
1446139161
1459139161
1460140160
1470140160
1480140160
1490140160
1500140160
1512140160
1523140160
1533140160
15413140160
15515140160
1560141159
1570141159
1580141159
1590141159
1600141159
1610141159
1620141159
1630141159
1640141159
1657141159
16612141159
16714141159
16815141159
1690142158
1700142158
1710142158
1720142158
1730142158
1740142158
1750142158
1760142158
1771142158
1783142158
1795142158
1805142158
1816142158
18211142158
18313142158
18414142158
18516142158
1860143157
1870143157
1880143157
1890143157
1900143157
1910143157
1920143157
1930143157
1940143157
1950143157
1961143157
1971143157
1982143157
1994143157
2007143157
2019143157
2020144156
2030144156
2040144156
2050144156
2060144156
2070144156
2080144156
2090144156
2100144156
2110144156
2120144156
2131144156
2144144156
2154144156
2165144156
2178144156
21814144156
2190145155
2200145155
2210145155
2220145155
2230145155
2240145155
2250145155
2260145155
2270145155
2280145155
2290145155
2303145155
2313145155
2323145155
2333145155
2343145155
2355145155
2366145155
2378145155
2389145155
23915145155
24015145155
24116145155
2420146154
2430146154
2440146154
2450146154
2460146154
2470146154
2480146154
2490146154
2500146154
2510146154
2520146154
2530146154
2540146154
2550146154
2560146154
2570146154
2580146154
2590146154
2600146154
2610146154
2620146154
2631146154
2642146154
2652146154
2663146154
2676146154
2689146154
2690147153
2700147153
2710147153
2720147153
2730147153
2740147153
2750147153
2760147153
2770147153
2780147153
2790147153
2800147153
2810147153
2820147153
2830147153
2840147153
2850147153
2860147153
2870147153
2880147153
2890147153
2901147153
2911147153
2921147153
2934147153
2944147153
2954147153
2964147153
2975147153
2985147153
2997147153
3008147153
3010148152
3020148152
3030148152
3040148152
3050148152
3060148152
3070148152
3080148152
3090148152
3100148152
3110148152
3120148152
3130148152
3140148152
3150148152
3160148152
3171148152
3181148152
3192148152
3204148152
3214148152
3224148152
3235148152
3245148152
3255148152
3266148152
32711148152
32812148152
3290149151
3300149151
3310149151
3320149151
3330149151
3340149151
3350149151
3360149151
3370149151
3380149151
3390149151
3400149151
3411149151
3421149151
3432149151
3442149151
3452149151
3463149151
3474149151
3486149151
3498149151
35011149151
3510150150
3520150150
3530150150
3540150150
3550150150
3560150150
3570150150
3580150150
3590150150
3600150150
3610150150
3620150150
3630150150
3640150150
3650150150
3660150150
3670150150
3680150150
3690150150
3700150150
3710150150
3720150150
3730150150
3740150150
3750150150
3760150150
3770150150
3781150150
3792150150
3803150150
3814150150
3824150150
3835150150
3845150150
3855150150
3866150150
3876150150
3886150150
3896150150
3907150150
3918150150
3928150150
39313150150
39414150150
39517150150
39617151149
3970159141
39811166134
3990172128
4001172128

Figure 1. Source: our measurement over all 400 dialects. Lines span the lowest to highest score, circles mark the medians, and the square marks the full score. The examples-only score includes what the manual earns. The remaining points require experiments. The table contains every dialect's scores.

Why a list of dials is not enough

With a fixed menu of 48 dials, an agent could try to learn the menu and test each dial once. In our prototype, a script that knew every dial did exactly that: one test per dial, and a perfect score on every dialect.

Then we gave it a dialect with 16-bit integers, a width it had never seen. Its single width test was designed to tell 32-bit, 64-bit and unlimited integers apart, by adding numbers near two billion. In a 16-bit language, numbers wrap around much earlier: print 30000 + 30000; prints -5536, not 60000. The answer matched none of the options the script knew, so it picked the nearest one, 32-bit, and every program that touched a number above 32,767 came out wrong. Its score fell to about 120 of 300 (prototype measurement).

So 15 dial settings are reserved for the dialects we evaluate on, including keywords such as puts and val, a nil that prints as <nil>, and division by zero that returns its left operand. An agent that has memorised the menu should fail on them. An agent that knows how to experiment should not.


IVAn exact grader

The reward is one number: how many of the 300 hidden programs the agent's interpreter reproduces exactly, output bytes and exit code both, from 0 to 300. There is no fuzzy matching and no judge model. A program matches or it does not.

We record which programs respond to each dial, so we can inspect a submission's errors alongside its transcript to judge which changes it found. Some programs depend on several dials, so their contributions cannot simply be added. The tables use that combined evidence, with isolated tests where available.

The button reports what happened, never why. It returns the program's output and exit code. A runtime error comes back as <<error>> and code 1, with no message. If it answered "index 3 is out of range", the agent could learn the indexing rule from the error text, and we would be measuring reading again. With only the output and exit code, the agent has to test what caused the error.

The dialect runs outside the agent's machine, and the agent has no network. On every task we check the extremes before trusting any score: the reference interpreter scores 300, an empty submission scores 0, and an interpreter that follows the manual scores close to 0.


VWhat testing found

Testing turned up problems in timing, the probe service and the way tasks were packaged.

The score depended on the machine. Our first design limited each program's wall-clock time. On a busy grading machine a correct interpreter could run out of time, so the same submission could score differently on two runs. Worse, a submission that hit the limit on every program could hold the grader for 340 programs × 60 seconds, almost six hours, longer than the agent's whole episode. We changed the scoring limit to 2 seconds of CPU time per program, so waiting for a busy host no longer consumes that allowance.

The button could stop working. The language allows up to 200 nested function calls. The service behind the button used a lower Python recursion limit than our reference interpreter, so a program recursing 199 levels deep, well within the rules, crashed it for the rest of the episode. Agents test documented limits, so this was common: the button died in 13 of 22 graded Claude Sonnet 5 runs and 5 of 34 GPT-5.6-terra runs (our measurement). We gave the service the same limit as the interpreter and made it survive any request, then replayed all 136,000 example and hidden programs in the dataset through it. Every one returned its recorded answer. The agent results below were measured before this fix. We have not rerun the agents to measure how much the failures affected their scores.

What we tested was not what users download. Our tests ran each task from our own copy. The public registry delivers files differently, and from there the grader failed before it started. Every check had passed and nothing worked. We now run every check on the downloaded task, exactly as users get it.


VIAttacking our own grader

We ask agents to try to earn points without doing the task. A separate model reviews their submissions, the grader and the recorded results for evidence of unearned reward. Where a control is available, we also score the submission with the suspected attack removed to see what it contributed.

In Dialect Recon's earlier checks, printing the same thing every time failed admission: the 40 examples have at least 35 different outputs. Memorising those answers passed admission but scored almost nothing, because none of the examples appears among the 300 graded programs.

One attempt did reach the maximum score without interpreting any program. An earlier round had missed it because the submissions never reached the grader. We fixed that test setup and isolated untrusted code from the hidden answers, giving it only the program being graded. After the fix, the shortcut scored 0, the reference scored 300, and all 34 agent submissions we re-graded kept their earlier scores. None of the agent runs in this post used the shortcut.

Source: our Dialect Recon validation record and subsequent red-team implementation history.


VIIResults

We ran both agents on the same ten tasks, drawn with a fixed seed, with up to four attempts per task. Claude Sonnet 5 (high) ran through Claude Code, GPT-5.6-terra ran through the Codex CLI at its highest reasoning setting. Every number below is our measurement from the raw runs. We re-graded all 53 finished submissions ourselves and every one reproduced its recorded score exactly.

Claude Sonnet 5 (high) GPT-5.6-terra
Finished runs 19 34
Mean score (of 300) 199 171
Median score 182 169
Perfect runs (300) 4 2
Runs exactly on the examples-only score 7 13
Runs below it 0 1
Median working time (limit 120 min) 19 min 16 min

For comparison, on these ten tasks the manual alone scores 4.6 on average and the examples alone score 129.3. Both agents clear the examples-only line on average, and both reach a perfect score at least twice, so the task is solvable from scratch in two hours. Claude Sonnet 5 scores higher on average and higher on 8 of the 10 tasks.

Many runs land exactly on the examples-only lineClaude Sonnet 5 (high)GPT-5.6-terraExamples-only scoreManual-only score
Task 1×2×4
Task 2×2
Task 3
Task 4×2
Task 5×2
Task 6×3
Task 7×4
Task 8×3×2
Task 9×2×2
Task 10×2×2
0100200300

Hidden programs reproduced (of 300)

Show the data
TaskExamples-onlyManual-onlyClaude Sonnet 5 (high) runsGPT-5.6-terra runs
Task 11080108, 108108, 108, 108, 108
Task 211015214110, 110, 214
Task 31141114, 204, 300102, 114, 204, 300
Task 41280231128, 128, 168
Task 51296245, 300245, 245, 300
Task 61300300169, 169, 169
Task 71331175, 234175, 175, 175, 175
Task 814516145, 145, 145145, 211, 211
Task 91474182, 182147, 182, 182
Task 101493149, 300149, 149, 217, 217

Figure 2. Source: saved regrade scores and task baseline receipts. Each row is one task. Short tick: manual-only score. Tall tick: examples-only score. Circles and squares: individual runs, with ×n where several runs scored the same.

Side by side, task by task

Task Hidden changes (not in examples) Examples-only score Claude Sonnet 5 runs GPT-5.6-terra runs
1 6 (2) 108 108, 108 108, 108, 108, 108
2 9 (3) 110 214 110, 110, 214
3 7 (2) 114 114, 204, 300 102, 114, 204, 300
4 16 (4) 128 231 128, 128, 168
5 10 (3) 129 245, 300 245, 245, 300
6 9 (3) 130 300 169, 169, 169
7 9 (3) 133 175, 234 175, 175, 175, 175
8 5 (2) 145 145, 145, 145 145, 211, 211
9 9 (3) 147 182, 182 147, 182, 182
10 8 (2) 149 149, 300 149, 149, 217, 217

Source: our measurement. Tasks are ordered by their examples-only score.

Nobody beat Task 1. Both of its hidden changes (ranges that include their end value, and arguments passed by reference) raise no error and are never hinted at. All six runs, from both models, scored exactly 108.

GPT-5.6-terra often repeated the same result. On Task 7 it scored 175 four times, on Task 1 108 four times, on Task 6 169 three times, and in each case the replay shows the same changes found and missed. Claude Sonnet 5 varies more between attempts on one task (Task 3: 114, 204, 300), which gives it more chances at a perfect run.

On Task 2 the only GPT-5.6-terra run that divided by zero scored 214, and the two that never tried scored 110. On Task 10 one extra probe of a top-level return separates 217 from 149. In these runs, the score depended on whether the agent tested that behaviour.

Which changes get found

Both agents found every change that the examples show, in every finished run (one GPT-5.6-terra run implemented one of them only partly, described below). Every change they missed completely was one the examples hide. Some of those changes were found far more often than others:

Some hidden changes are almost always found. Others never are.Claude Sonnet 5 (high)GPT-5.6-terra
Sonnet 5GPT-5.6-terra
0%25%50%75%100%

Share of dialects with this hidden change where the run found it

Show the data
Hidden changeClaude Sonnet 5 (high) foundGPT-5.6-terra found
list items evaluated right to left1/13/3
variables scoped to the function2/23/3
and/or return an operand2/24/4
negative indexes wrap or clamp2/22/3
an error erases earlier output2/32/4
return at top level ends the program1/22/4
ranges include their end2/43/7
division by zero gives a value2/21/6
one global scope for everything2/42/7
let on an existing name reassigns it0/32/3
number vs string compares false2/41/7
reading past the end gives nil2/60/10
division rounds down1/30/6
arguments passed by reference0/30/7
lists copied into functions0/20/3
and/or evaluate both sides0/10/3
arguments evaluated right to left0/10/3
extra arguments silently dropped0/10/3
string index gives a character code0/20/4

Figure 3. Source: saved per-change regrades and transcript review. For each hidden change, the share of runs that implemented it correctly. Labels show found of total.

How agents find hidden changes explains the split. The manual lists its runtime errors in one table, and the most productive habit in both models' transcripts was to run each listed error through the button and compare. A dialect change that turns one of those errors into a normal result (division by zero returning a value, a top-level return that ends the program quietly, comparing a number with a string giving false) gets found this way.

The changes found least often raise no error at all. Arguments passed by reference (0 of 3 for Claude Sonnet 5, 0 of 7 for GPT-5.6-terra), lists copied when passed to a function, a string index returning a character code, arguments evaluated right to left. Nothing in the manual or the examples points at them. To find one, an agent has to imagine it first.


VIIIHow each model works

We read every transcript: 22 graded Claude Sonnet 5 runs and 36 GPT-5.6-terra runs. Both models read the manual, compared outputs, built an interpreter and tested it, but differed in when and how they used their experiments.

Behaviour Claude Sonnet 5 (high) GPT-5.6-terra
First move after reading Probes first. In 21 of 22 graded runs the first probe comes before any interpreter code, at about 2 minutes. Builds first. Writes a complete interpreter, makes all 40 examples pass, then probes.
Probes used (median) 33.5, about a third of the budget 17.5, about a sixth of the budget
Most probes in one run 152 of 200 83 of 200
Probe design Tracer functions, powers-of-two scans, binary searches over the socket, side-by-side checks against its own interpreter Batches of one-line questions, and an error-table sweep in its best runs
Same task, repeated Scores vary between attempts Often the identical score and the identical changes found
Final message A detailed summary, sometimes claiming checks it did not run Always begins "Implemented submission/interp.py… matches all 40 visible examples". Never mentions what was left untested.

Source: our reading of every transcript. Probe counts are exact where the agent printed the server's remaining budget, and counted from commands elsewhere.

No run came close to the limitClaude Sonnet 5 (high)GPT-5.6-terraMedian
time limit
Claude Sonnet 5 (high)median 19 min
GPT-5.6-terramedian 16 min
0306090120

Minutes the agent worked before declaring itself done

Show the data
ModelRunsFastest (min)Median (min)Slowest (min)
Claude Sonnet 5 (high)19101951
GPT-5.6-terra3491628

Figure 4. Source: saved run timestamps. Minutes each finished run worked before declaring itself done. The limit was 120.

Both models stopped with time and experiments left. No run used more than 51 minutes of 120, and no run used its whole probe budget. The median Claude Sonnet 5 run used a third of it, the median GPT-5.6-terra run a sixth. GPT-5.6-terra runs never discussed how many hidden changes might remain, although the instructions say there are between four and sixteen. Several Claude Sonnet 5 runs misread what is graded: the instructions say the withheld programs are checked, but not that the score counts exact matches on them, and some runs concluded that only crash-avoidance was scored and stopped as soon as the 40 examples passed.

In every perfect run the agent sent questions about meaning (evaluation order, scope, comparisons) to the button and to its own interpreter, side by side, and treated every disagreement as a lead. The runs stuck on the examples-only score either never asked the question that mattered, or asked it only of their own interpreter.

Moments from the transcripts

Each block below is copied from the recorded transcript. Times are seconds from the start of the run.

Claude Sonnet 5, Task 10: one comparison turns 149 into 300

While checking comparisons, the agent ran the same program through the button and through its own interpreter (848 s):

cat > /tmp/e14.src << 'EOF'
print 1 < 2 < 3;
EOF
python3 /tmp/probe2.py /tmp/e14.src
echo "=== mine ==="
python3 submission/interp.py /tmp/e14.src; echo status=$?
--- result ---
false
exit_status=0 remaining=177
=== mine ===
<<error>>
status=1

Its next message (894 s): "…the dialect may silently change other error behaviors too. Let me systematically test each documented error condition against the real probe." That sweep found the task's second hidden change. Another Claude Sonnet 5 run on the same task never sent these probes and scored 149, exactly the examples-only score.

Claude Sonnet 5, Task 6: tracer functions reveal evaluation order

A single 349-byte probe (594 s) uses a function that prints its argument, to show the order in which things are evaluated:

fn P(tag) { echo tag; return tag; }
fn two(a, b) { return a; }
echo two(P(1), P(2));
echo [P(3), P(4), P(5)];
...
--- result ---
1
2
1
5
4
3
[3, 4, 5]

Arguments run left to right, but list items run right to left (5 4 3). No example shows this, and no error table would lead to it. This run scored 300. Another Claude Sonnet 5 run on the same task, which ended early and is not among the finished runs, never tested evaluation order this way and scored 224.

Claude Sonnet 5, Task 3: "no scoping at all"

The agent compared the button's answer with its own interpreter on a closure-in-a-loop program (637 s):

let fs = [];
for (i in 0 .. 3) { fn g() { return i; } push(fs, g); }
print fs[0]();
print fs[2]();
--- result (probe, then own interpreter) ---
{"remaining": 150, "status": 0, "stdout": "2\n2\n"}
0
2

After two more probes it concluded (757 s): "This is a fundamental discovery … the dialect has no lexical/block scoping at all", just one flat global namespace. It rewrote the interpreter and scored 300. It is the only Claude Sonnet 5 run on this task to find that change, which controls 96 hidden programs.

Claude Sonnet 5, Task 4: one output line reveals two hidden changes

The agent first found that integers wrap at 8 bits (print 128; prints -128). Later (493 s):

print 200 / 3;
print 200 % 3;
--- result ---
STATUS=0 REMAINING=48
-19
-2

200 wraps to −56, and −56 / 3 rounds down to −19 instead of toward zero (−18). One line exposed both the 8-bit width and a hidden rounding change.

Claude Sonnet 5, Task 8: the right test, written and never sent

The agent wrote eight small programs, one per documented error, and ran all eight on its own interpreter. One of them, e7, read past the end of a three-element list, exactly the dialect's hidden change. Then (691 s):

"Now let's verify these against the real dialect via probe (combining calls isn't possible since errors halt execution, so I'll check a representative subset to confirm error format/status)"

for f in e1 e4 e5 e8; do echo "=== $f ==="; python3 t/myprobe.py t/$f.src; done

e7 was not in the subset. The run stopped at 746 s with 55 of 80 probes unused, and scored 145, the examples-only score. The missed change controls 89 hidden programs.

Claude Sonnet 5, Task 5: asking its own interpreter instead of the button

The agent tested a mixed comparison, but only locally (787 s):

print 1 < "a";
=== cmpmix ===
<<error>>
status=1

"All correct, matching the spec's documented examples exactly."

On the same task another Claude Sonnet 5 run sent print 1 < true; to the button and got false. That run scored 300. This one scored 245. Comparing against the button caught the change that the local test missed.

GPT-5.6-terra, Task 3: one hidden change hides another, and the run falls below the floor

The agent probed negative string indexes (about 467 s):

print "abc"[-1];
print "abc"[-4];
--- output
<<error>>

The first line really printed c. The second index was out of range and raised an error, and this dialect has a second hidden change: an error erases everything printed before it. So the c vanished. The agent concluded that negative indexes do not work on strings and restricted them to lists:

sed -i '180c\  if i<0 and type(a)is list:i+=len(a)' submission/interp.py

It lost 12 programs that an examples-only interpreter gets right and scored 102, the only run of either model below the examples-only score. Another GPT-5.6-terra run tested "abc"[-1] and "abc"[-3], both in range, saw no error, and scored 300.

GPT-5.6-terra, Task 2: one division by zero, +104 programs
print 1 / 0;
---
REF
1
OURS status=1
<<error>>

Its next probe divided and took remainders of 7, -7 and 0 by zero, and all returned the left operand. The agent: "One additional dialect rule surfaced: division and remainder by zero do not error", and both return the left operand. This run scored 214. The other two GPT-5.6-terra runs on the task never divided by zero and scored exactly 110.

GPT-5.6-terra, Task 8: the same question, asked two ways

Two runs tested whether let inside a block creates a new variable. One printed the outer variable afterwards:

let a = 1;
if (true) { let a = 2; }
print a;
---
2

It concluded that let rebinds an existing name, and scored 211. The other read the variable only through a closure inside the block, which prints the same value under either rule, reported that scope "matches the base language", and scored 145, the examples-only score.

GPT-5.6-terra, Task 9: evidence against its own hypothesis, ignored

The agent decided negative indexes wrap Python-style. A later probe (about 640 s) said otherwise:

let a = [1, 2];
a[1] = 9;
print a;
a[-1] = 8;
print a;
{'remaining': 71, 'status': 1, 'stdout': '[1, 9]\n[8, 9]\n<<error>>\n'}

Python-style wrapping would give [1, 8]. The dialect gave [8, 9]: it clamps -1 to the first element. The agent kept its original rule, scored 0 of 56 on that change, and finished on the examples-only score of 147. Both other GPT-5.6-terra runs on the task found the clamp and scored 182.

GPT-5.6-terra, Task 10: one probe worth 68 programs
fn f() { print 7; return 1; }
print 1;
return f();
--- probe vs ours
1
7
[ours]
1
<<error>>
status=1

The agent: "a top-level return is valid, evaluates its expression, and ends the program successfully. I’m incorporating that behavior now." The two GPT-5.6-terra runs that sent this probe scored 217. The two that did not scored 149. The whole gap is this one question.


IXDid either model try to cheat?

We found no evidence of it. We searched every command, every file read and every file written in all 62 transcripts (26 Claude Sonnet 5, 36 GPT-5.6-terra) for any attempt to reach the hidden programs, the dialect's settings, the reference interpreter, the grader or the network, or to influence the score. We found none in either model.

The few commands that looked outside the task folder were diagnostic: locating the task directory, checking why ./probe would not run, and checking whether the button's service was still alive after it stopped answering. Some runs wrote their own client for the button's socket to see the exit code and remaining budget. That uses the documented interface, and the service still enforced the budget. Every submission we examined is a genuine interpreter that reads only the program it is given.


Conclusion

What stood out to us was how often the agents stopped without testing the hidden changes. They found every change shown by the 40 examples, with one partial implementation, but often left behaviour such as argument passing and evaluation order unchecked. No run used even half of its two hours. The median run used a third (Claude Sonnet 5) or a sixth (GPT-5.6-terra) of its experiments.

The transcripts helped explain the scores. On Task 2, a division-by-zero probe was worth 104 programs. On Task 10, a top-level return was worth 68. Comparing the button's answer with the agent's own interpreter exposed differences that testing the interpreter against the manual could not. We want to train agents to make those comparisons more often, including when nothing has gone visibly wrong.

The manual-only and examples-only scores let us see what the agents had actually discovered. We also had to fix the grader and check that our red-team submissions had reached it. The results here still describe the earlier agent runs, including the probe-service failures.


References

  • Acher, M. (2026). Can Coding Agents Program in M&Ms Language? blog.mathieuacher.com
  • Lu, K.-C., & Krishnamurthi, S. (2024). Identifying and Correcting Programming Language Behavior Misconceptions. OOPSLA 2024. Author publication page, DOI 10.1145/3649823
  • Pombrio, J., Krishnamurthi, S., & Fisler, K. (2017). Teaching Programming Languages by Experimental and Adversarial Thinking. SNAPL 2017. doi:10.4230/LIPIcs.SNAPL.2017.13
  • Sharma, A., & Chopra, P. (2026). EsoLang-Bench: Evaluating Genuine Reasoning in Large Language Models via Esoteric Programming Languages. arXiv:2603.09678
  • Sharma, A., Thorat, S., & Chopra, P. (2026). Frontier Coding Agents Use Metaprogramming to Adapt to Unfamiliar Programming Languages. arXiv:2606.10933
  • Thimmaiah, A., Zhang, J., Srinivasa, J., Li, J. J., & Gligoric, M. (2025). LLMs Lean on Priors, Not Programming Language Semantics (PLSemanticsBench). arXiv:2510.03415
  • Wei, A., et al. (2025). CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesis. arXiv:2503.23145
  • Wu, Z., et al. (2023). Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks. NAACL 2024. arXiv:2307.02477
  • Yin, C., et al. (2025). Investigating Advanced Reasoning of LLMs via Black-Box Environment Interaction (ORACLE). arXiv:2508.19035
  • Adamczewski, T., et al. (2026). MirrorCode: AI can rebuild entire programs from behavior alone. arXiv:2606.30182
  • Yang, J., et al. (2026). ProgramBench: Can Language Models Rebuild Programs From Scratch? arXiv:2605.03546

Name a skill your model is missing.

Get in touch

©2026 Attoloop
Built in India