PermitForge

An agent is handed a plant that already meets its air-quality limits. Its stacks are at their tallest allowed height and every one has the strongest control the task offers. That design is the draft. The job is to make the plant cheaper than the draft without letting the air anywhere around it get too dirty.

The agent can change the control on each stack and pick its height. It can ask what a design costs as often as it likes. Finding out what reaches the air around the plant is the scarce part: it gets 30 evaluations, then has to submit one design for an independent physics check.

This is PermitForge. The plants are synthetic. The weather is real. The grader is the same dispersion model US regulators use. We built it, ran four frontier models on it, then trained a 27B open model on it and measured what changed.

Source: the saved evaluation episodes, recomputed in the figure data. Means give each task equal weight after averaging its repeats. "Saved" is the fraction of the draft's annualized cost that the submitted design removes while still passing every standard.

IWhy this job

We wanted an environment with three properties.

The agent has to search. A plant has three controls and up to eight heights for each of its stacks, which is 576 designs for the smallest plant and millions for the largest, and the cheapest one that still passes cannot be read off the inputs. The agent has to try designs, read what the physics says about each, and decide what to try next, with a limited number of tries.

The grade comes from physics. A submitted design is run through the dispersion model regulators use. There is no judge model and no stored answer to grade against.

The score means something outside the benchmark: dollars per year saved while the modeled air stays inside the standards.

Air permits have all three, and PermitForge borrows the US rules. Kilns, boilers and other industrial equipment release pollutants, and in the US a plant needs a permit before a new source is built or an old one is modified (New Source Review). The limits are the National Ambient Air Quality Standards, NAAQS (EPA's table). They are limits on the air people breathe, not on a single stack. That is what makes the job a search: a plant can meet them with strong controls, tall stacks, or some mix, and the cheapest mix depends on the weather around that plant.

Before a large project, the applicant has to show with a dispersion model that the plant will not push the air over a standard anywhere around it. AERMOD is EPA's preferred model for this kind of source under the US modeling guideline, 40 CFR Part 51 Appendix W. A consultant sets up the sources, the points where the air is checked and several years of weather, runs the model, finds the standard and the location closest to the limit, changes a control or a stack, and runs it again. Each run takes minutes, so runs are rationed. The agent's 30 evaluations stand for that constraint.

Height has a limit, and PermitForge borrows that too. Under 40 CFR 51.118, an emission limit cannot take credit for stack height above Good Engineering Practice, GEP, which depends on the buildings nearby (40 CFR 51.100(ii)). The rule limits credit, not the physical stack. PermitForge makes it the top of each stack's menu, so an agent cannot buy its way out with height alone.

PermitForge keeps that loop and drops the rest. The benchmark checks eight modeled standards and nothing else: no case-by-case review of control technology, no negotiation with an agency, no permit at the end. Passing those standards is the benchmark's definition of feasible, nothing more.

IIThe job, from the ground up

A plant and its stacks. A plant here is a few buildings with 2 to 6 stacks. A stack is a chimney: a boiler, a kiln or a turbine burns fuel, and the exhaust goes up the stack and out into the air. Each stack releases three pollutants. Nitrogen oxides (NOx) come from burning at high temperature and partly turn into nitrogen dioxide (NO₂) in the air, which is the form the standards limit. Sulfur dioxide (SO₂) comes from sulfur in the fuel. Fine particles (PM₂.₅) are soot and dust small enough to breathe in. How much of each comes out of a stack with no control at all is fixed by the equipment: a kiln stack in our example releases 92 grams of NOx every second.

A control. A control is equipment that removes part of a pollutant before it leaves the stack: a catalyst for NOx, a scrubber for SO₂, a filter for particles. In PermitForge every stack has the same three choices and they always come as a package:

Source: the task contract. These are synthetic packages, not named commercial technologies.

So the kiln stack that releases 92 g/s of NOx with no control releases 26.5 g/s with moderate control and 4.6 g/s with maximum. Controls cost money every year (buying the equipment, running it, maintaining it, all rolled into one annual figure), and the price differs from stack to stack: on our example plant, maximum control costs between $131,000 and $424,000 a year per stack, and moderate control costs 55% of that.

A height. The second lever is how tall the stack is. Exhaust leaving a taller stack has more room to spread out and dilute before any of it reaches the ground, so the air near the plant gets cleaner. A taller stack also costs money: in PermitForge, one price per metre per year above the stack's minimum height ($4,501 per metre on the example plant). And there is a catch near buildings. Wind blowing over a building leaves a turbulent wake behind it, and a short stack's exhaust gets caught in that wake and pulled down to the ground. That is downwash. Raising a stack lifts the exhaust out of the wake, which is one reason the first metres of height help so much. Each stack can be set to one of up to eight heights between its minimum and its GEP limit.

A receptor. The model does not check "the air" in general. It checks the air at specific points called receptors: a calculation point, not a sensor. Our plants have 184 of them, close together along the plant's fence and more spread out up to five kilometres away. A design has to pass at every one of them.

Background. The plant is not the only thing polluting. Traffic, other industry and natural sources put pollution into the air already. That amount, the background, is added to whatever the plant contributes before the sum is compared with the limit.

A standard, a margin, a binding receptor. Each standard is a limit on the concentration of one pollutant in the air, in micrograms per cubic metre (µg/m³), together with a rule for turning five years of hourly values into one number, the design value. There are eight standards here: hourly and annual NO₂, daily and annual PM₂.₅ (the annual one twice, with a stricter limit that protects health and a looser one that protects crops and visibility), and hourly, three-hour and annual SO₂. For each standard, the receptor whose design value comes closest to the limit is the binding receptor, and the distance left under the limit there is the margin. A design is feasible when every margin is zero or more. Chapter III shows how a design value is computed.

The draft and the cost. Every task starts from the draft: every stack on maximum control at its tallest allowed height. It always passes, and it is the most expensive design there is. A design's cost is the sum of its control prices plus the price of the height bought above each stack's minimum. The agent's job is to find a cheaper design that is still feasible.

Let's take an example

Take a kiln plant at the KABQ station (Albuquerque weather) with six stacks and three buildings. The draft costs $3,569,382 per year. Here is what the stacks release with no control, and what each control costs on each of them.

Source: the scenario recorded in the showcase episode, in the figure data. Uncontrolled rates in grams per second, with NOx counted as NO₂ mass before chemistry. Prices are annualized.

Source: the KABQ kiln task, saved geometry and the best submission among its model repeats, from GPT-6.1 Sol (medium), as checked by the full dispersion model. Draft receptor values were reconstructed from the saved unit fields and checked against the native draft value. The static views crop to the near-fence area. The interactive view and the table include every receptor. This is a design-value map, not an hourly plume.

Here is the draft beside the best design any model submitted for this plant:

Source: the same scenario and the best native-verified submission in the main panel, from GPT-6.1 Sol (medium). Per-source costs use the benchmark's exact cost formula, rounded to whole dollars. The figure data keeps the exact ledger.

The recorded trials show why the choices differ from stack to stack. S2 keeps maximum control but drops to its minimum height: strong control on this stack is cheap, $137,777 a year in total, while the height it would otherwise need would have cost $184,282 on its own at an earlier 52.5 m trial. S4 goes the other way: no control at 41 m costs $139,495, already below its $225,867 moderate-control price, so it buys height instead of equipment. S6 needs no control at 57 m, because its $218,177 of extension is cheaper than even moderate control at $233,310. The model's last evaluation tried S6 one step lower, at 49 m, with everything else as submitted: the hourly NO₂ margin at receptor 91 went from +1.87 to -1.83 µg/m³, so it submitted the 57 m design. Earlier trials that lowered S1, S3 or S5 on their own, with other stacks at other settings, failed at other receptors.

Source: showcase evaluations 3, 8, 9, 12, 13, 23, 24, 26, 27 and 28, the submitted design and the exact per-source ledger, all in the figure data. Feasibility belongs to the whole plant, not to one stack, and these trials do not show that every neighbouring design fails.

The best submitted design costs $1,376,183, a saving of 61.44%. What holds it there is hourly NO₂ at receptor 91 on the south fence: a modeled value of 142.77 µg/m³ plus 43.52 µg/m³ of background leaves 1.87 µg/m³ under the limit.

Source: the showcase's native submission and draft cost. "Best" means the best observed in the main panel, not a proven optimum.

Nothing else about the plant can change. Moving a stack, changing its exhaust temperature or moving the receptors would be a different problem.

IIIWhy it is hard

The weather does not give a design one concentration. Wind direction, wind speed, how well the air mixes and the temperature of the exhaust change where the pollution goes, hour by hour. PermitForge uses five years of real hourly weather, 2020 to 2024, from one weather station per plant. A cheap design that is fine on an ordinary day can fail during one unusual combination of wind and still air.

The standards handle this with the design value. The hourly NO₂ standard works like this. For every day, take the worst valid hour. For every year, line up its 365 or 366 daily values and take the 8th-highest one (the rule allows a few bad days a year, about the worst 2%). That gives one number per year, five numbers for five years. Their mean is the design value. The model does this at every receptor, and for this standard the receptor with the largest design value is the binding one.