ITPA HDB5 · STD56228 shots18 tokamaks no synthetic data

Confinement Scaling Reversal

On the real ITPA database, a random forest beats the published 1998 scaling law by 41%. Hold out an entire tokamak instead of a few shots and the ranking inverts completely: the best cross-validated model becomes the worst of the three, on every machine tested.

What is being measured

A scaling law is not for the machines you have

Energy confinement time τE sets how big a tokamak has to be to reach ignition, and the field predicts it with a power law fitted across existing devices. The reference is IPB98(y,2), published in the 1999 ITER Physics Basis and used to size ITER itself. It is a product of eight engineering parameters raised to fitted exponents, so in log space it is exactly a linear model, and fitting one is ordinary least squares on a log design matrix.

The obvious modern move is to replace it with something more flexible. Trained and scored on the ITPA H-mode database under grouped cross-validation, that works: a random forest reaches 0.128 RMSLE against the published law's 0.199, a 41% improvement measured against real physics rather than against a mean predictor.

But cross-validation here holds out discharges. Every machine in the held-out fold also sits in the training fold, so it measures how much of JET is predictable from the rest of JET. That is interpolation. A scaling law exists to answer a different question, and the honest test is to escalate what gets held out:

01 Hold out a discharge. Grouped CV by shot. Same machines on both sides. interpolation
02 Hold out a machine. Train on 12 tokamaks, predict the 13th, rotate. transfer
03 Hold out a size range. Train below a major-radius cut, predict above it. extrapolation

Both arms use the same nine engineering features and the same models. Only the split changes. The IPB98 prior is dropped as a feature throughout, because its exponents were fitted on this database including whichever machine is held out.

Step 02 · interactive

Hold out a tokamak

Pick a machine. The left column is fixed: it is how the models rank under cross-validation, where every machine is in the training set. The right column is how they rank on that machine when it is removed from training entirely. Machines are ordered by how far they sit outside the training distribution.

Leave one tokamak out

13 machines · ordered by distance
Held out
Rows
Distance from training
Rows above training max

Ranked by cross-validation

held-out shot · identical every time

Ranked on the held-out machine

held-out machine · rearranges

ref IPB98(y,2) is ranked for reference but never marked as the leader: its exponents were fitted on this database with the held-out machine included, so it is not blind. The accent marks the leading model that is.

The result

The order under one split is the reverse of the order under the other

Averaged over all 13 machines, the ranking of the three models that actually fit something is exactly inverted between the two splits. The random forest is worse than the log-linear power law on 13 of 13 machines, and the 95% interval on that paired gap is [+0.157, +0.342], nowhere near zero.

ModelCV, by shotLeave one machine out DegradationCV rankMachine rankρ(distance)

ρ(distance) is the rank correlation between a model's error on a machine and how far that machine sits outside the training data. The forest's errors are explained by extrapolation distance (ρ = +0.85). The power law's are not (ρ = −0.06): its error is uncorrelated with how far away the machine is, which is the property that matters when the next machine is the furthest of all. IPB98 is listed for reference and is not blind, since its exponents were fitted on this database.

Three panels: model scores under both splits crossing over; per-machine error against Mahalanobis distance; median against worst case along a flexibility ladder.
Interpolation against extrapolation. Left, each model under both splits: the lines crossing is the result. Middle, per-machine error against distance from the training data, rising for the trees and flat for the power law. Right, the flexibility ladder, where what grows with flexibility is the tail of the error distribution rather than its centre.

Step 03 · the ITER direction

The jump ITER asks for, measured inside the database

Holding out JET still leaves twelve machines spanning much of its range, so it extrapolates in identity while interpolating in size. ITER's major radius is 6.2 m against 3.40 m for the largest row here, a factor of 1.82 beyond anything measured. That factor turns out to be available inside the database: train on the 14 smallest machines up to DIII‑D at 1.865 m and predict the 4 largest up to JT‑60U at 3.40 m, and the jump demanded is 1.823 against ITER's 1.824.

ModelHeld-out shotHeld-out machine Machine larger than any in trainingSkill

Skill places each model between predicting a single constant (0.00) and the analytic power law (1.00). The power law keeps 93% of that distance. The trees keep 41% and 31%: asked the question a scaling law exists to answer, they land closer to a constant than to the law they beat by 41% under cross-validation.

Size-ordered extrapolation: escalation across three splits, per-machine breakdown, and the sweep across every size cut.
Size-ordered extrapolation. The cut is chosen as the one whose size ratio is closest to ITER's, so it is a property of the data rather than a hand-picked split: add a larger machine and the matched rung moves on its own.

What fixes it, and what does not

A bounded correction, and an interval that collapses

The diagnosis is specific enough to build against. A tree ensemble predicts by averaging training targets, so every prediction it can make lies inside the training range: it structurally cannot reach a machine whose confinement times run above everything it has seen. A power law has no such bound. So keep the power law and let a heavily damped tree correction work on its log residuals, where the bounded quantity is centred on zero instead of growing with machine size.

Result 6 · the hybrid

Best blind model at the ITER cut

Power law plus a bounded correction: 26% below plain ridge and 4.6× below the random forest, better on all three held-out machines and by the widest margin on the most distant. The base law is biased by −0.218 in log units out there; the correction supplies 46% of it and never leaves its training range.

Result 7 · the interval

Coverage of a 90% interval

Split-conformal intervals hit nominal 90% under cross-validation for every model, then cover 3% of the rows across the ITER-matched cut for the random forest and none at all for the gradient booster. Widths move by under 1.5% between the two: the intervals do not get vague out of distribution, they stay the same size and miss.

The honest limit on the hybrid: scored at every rung of the size sweep rather than only the matched one, it wins at 5 of 8 well-powered cuts, not all, and no rule over those eight points separates the wins from the losses. What survives is narrow, and it is stated narrowly.

Step 04 · the cure

One line of physics, and it is not a model at all

Every model above learns its form from the data. None of them is told any physics, which is odd, because the field has known since Connor and Taylor (1977) that a scaling law is not free: requiring it to be expressible in dimensionless variables imposes linear equality constraints on the exponents. So a physics assumption is a matrix C, the fit becomes least squares subject to Cb = d, and the solver for exactly that was already in the repository.

The check that the derivation is right is external and hard to fake. IPB98(y,2) was published in 1999 and was never told about these surfaces, yet it lands on the Kadomtsev surface at a distance of and on the collisionless surface at , both inside the rounding of its own two-decimal exponents.

ModelIn sampleCV, by discharge Held-out machineITER-matched cut

is the best score any blind model on this page reaches at the ITER-matched cut. It beats the hybrid above, and it beats the analytic law that was fitted with those machines included. It has no hyperparameter and nothing to tune: the only difference from the unconstrained row is the constraint.

The free constraint

Kadomtsev wins at every size cut

It costs nothing in sample, because the data already satisfies it unaided, and it still beats the unconstrained fit at every cut in the sweep. A constraint the data satisfies on average still stops the fit wandering when the training set is small.

The one that costs something

Collisionless wins where it is resolvable

It costs 0.001 in sample and wins at every well-powered cut. Push one rung further, to a beta-independent law, and it degrades again: there is an optimum in the middle of the hierarchy, and nothing here predicted where. Cross-validation cannot select any of it.

Cost against benefit across the size sweep: in-sample penalty for each constraint, and the score at every size cut for the constrained and unconstrained fits.
Cost against benefit across the size sweep. The constraints are derived in code from the definitions of rho*, beta and nu* rather than copied out of a paper, so the surfaces move if the definitions do.

Step 05 · the interval, repaired

The collapse is repairable, and the repair stops exactly where the diagnosis says

If the interval failure really is about exchangeability rather than about the models, then calibrating on held-out machines instead of held-out discharges should repair it, and should repair it only as far as that unit reaches. Both halves land, and the half that fails is the stronger result.

Nominal 90% intervalSplit conformal, held-out machine Calibrated on machinesPlus distance scaling Best scheme at the ITER cut

On a held-out machine every model returns to within two points of nominal, tree ensembles included. Across the size cut none does, because every calibration machine is smaller than every test machine and no recalibration makes those two exchangeable. The constrained power law is the single exception: its intervals hold at the ITER-matched cut under every scheme, including the plain split conformal that fails there for everything else.

Coverage under three calibration schemes, on a held-out machine and across the ITER-matched size cut.
Coverage under three calibration schemes. A repair that does not work everywhere is worth more than one that appears to: it tells you which assumption was actually doing the work.

Step 06 · is it just this file?

It reproduces on rows this database does not contain

Everything above rests on one file, which is the honest ceiling on all of it. The same OSF project publishes the full database revision, 14153 rows against STD5's 6228, pinned by its own content hash. Matching on (tokamak, shot, time) shows STD5 is a quality selection out of it, and leaves two populations this repository had never analysed.

PopulationRowsRows shared with STD5Machines Best under CVBest on an unseen machineCV gain over the published law

The column ordering inverts in both arms. On the disjoint H-mode rows the best cross-validated model beats the published law by a margin within a point of the one this page opens with, computed on rows that headline never saw. So the reversal is not an artifact of the standard set's selection criteria, and not a property of ELMy H-mode or of IPB98(y,2) specifically. It is not an independent database either: both arms come from the same ITPA collection and the same devices, and the five-machine arm is too small to carry a claim alone.

Step 07 · the whole argument in one table

What it predicts for ITER, written down before the answer exists

Everything above is retrospective. This table is not. It records what each model says about three real machines, with intervals, a date and a digest over the rows, so that a later edit leaves a mark. The parameter sets are published design values, and each reproduces the confinement time its own source quotes.

Model

Inside the data, they agree

Spread on JT‑60SA, which is operating

JT‑60SA sits inside the database's size range, and there all five models land within a few percent of each other. Nothing about the disagreement below is visible from a machine you already have.

Outside it, they do not

Spread on ITER, 1.82× beyond the data

A tree ensemble averages training targets, so it cannot exceed the largest one, which here is s. The random forest that wins cross-validation by 41% is arithmetically incapable of returning ITER's predicted confinement time, and its 90% interval does not contain the physics answer or anything near it.

One thing nothing above anticipated: SPARC sits further from the training data than ITER does, against , despite being smaller than JT‑60U. Its 12.2 T field is far outside a database that tops out near 4 T. Size is not the only direction a next-step device leaves the data in.

Locked · content digest

Step 08 · is any of this about fusion?

The same audit on a scaling law from another science

Everything above rests on ITPA data, so the last question is whether any of it is about plasma. The way to find out is to run the identical audit on a scaling law from a different field. Mammalian basal metabolic rate against body mass is the same object and the older one: Kleiber found in 1932 that rate scales as mass to the 3/4, and that exponent is still the published baseline. Taxonomic order plays the part of tokamak, body mass the part of machine size. species records across orders spanning in median mass, pinned by content hash, and run through the reusable module rather than a copy of the fusion pipeline.

ModelCV, by speciesLeave-one-order-out Widest mass cutrho vs distance

The half that reproduces

Mass cuts where the power laws beat both trees

All of them. The random forest also loses to Kleiber on held-out orders, and error tracks extrapolation distance for the trees far more than for the laws. The extrapolation failure is not about tokamaks.

The half that does not

no

Ranking reversal here

The trees never win the easy split either, so there is no inflated margin to invert. With one predictor and a relationship close to a straight line in logs, a tree has nothing to exploit. The reversal needs enough feature dimensionality for the flexible model to win interpolation first, and nothing above could have shown that.

The constraint result travels in direction but not in strength: Kleiber's published exponent costs under cross-validation and wins the widest cut by , but at only of the cuts rather than the 8 of 8 the collisionless constraint manages. A constraint helps out of distribution; that it helps reliably is not something this second dataset supports.

Kleiber's law under the same three splits: no inversion, error against distance, and the mass-ordered sweep.
Kleiber's law under the same three splits. One replication, one predictor, eleven phylogenetically related groups. Enough to show the extrapolation failure generalises and that the reversal needs a condition this dataset lacks; not enough to put a rate on either.