Why this note exists
We publish models of how infrastructure gets allocated. We also sell infrastructure execution. Those two facts sit uncomfortably together, and the honest response is not to promise we are careful. It is to write down the specific ways a model like ours goes wrong, then turn each into something that fails loudly.
All three below are real. Each nearly reached a published page.
One: no identification condition
We compared two ways of pricing the same trade and found a difference. Good. But two formulas of different algebraic shape produce different numbers whether or not the effect exists at all. Finding a difference tells you nothing about what caused it.
So before trusting any of it, we pinned the case where the answer is known in advance. Switch reclaim off. Over identical jobs on identical units, the two regimes must charge exactly the same total, because with no risk to allocate there is nothing for the denomination to do.
They do, to floating-point precision. That one test is what makes every other number interpretable: any spread at positive hazard is now attributable to reclaim rather than to the shape of two formulas.
A comparison without a null case is not a finding. It is a pair of numbers.
This is the test we would look for first in someone else's simulation, and the one most likely to be missing. It is also the cheapest of the three to build, and the only one of the three we thought of unprompted.
Two: a premise that was only true where we looked
A second model rested on an assumption we stated plainly and verified numerically: completion-time variance rises as the checkpoint interval grows. Longer interval, bigger quantum at risk, more spread. Intuitive, and checked across every regime we studied. Sixty-three grid points, no violations.
It is false. Variance is U-shaped, and below roughly a sixth of the classical interval it starts rising again: once the interval is small relative to the cost of writing a checkpoint, attempt length stops shrinking and overhead accumulates instead.
We missed it because our grid began just above the minimum. The premise looked true on exactly the range we had chosen to examine, which is the least reassuring place for a premise to look true.
What changed The assumption is now scoped to a range rather than stated globally, and the test that guards it deliberately searches below the minimum and asserts that the naive version fails there. A test that only exercises the region where you expect to pass is not a test.
The results survived, because the point where they are applied sits well above the minimum. That was luck. Nothing in our method would have caught it if it had not been.
Three: we searched the literature last
The third is the expensive one. We took a classical scheduling result, swapped its objective for one matching how workloads are actually submitted, derived the correction, proved two small theorems, ran a simulation across three regimes, and wrote the paper.
Then we searched for prior art.
It exists. The real-time systems literature calls the objective Level of Confidence, the probability that a job using roll-back recovery meets its deadline, and has been maximising it over checkpoint count since at least 2011. It states the motivating claim in almost our words, and reports the non-monotonicity we thought we had found.
The search took under an hour. The work it invalidated took considerably longer. No clever lesson here, only an ordering one: the literature search is step zero, not step nine, and the feeling that an idea is too natural to be new is evidence that it is not.
We published anyway, labelled as a rediscovery, because it is relevant to a setting the original work did not consider and because almost nobody implements it. That is salvage, not a plan.
The rails we run on now
Each failure became something that fails loudly rather than something we promise to remember.
- An identification condition per comparison. Switch off the mechanism under study. If the two arms do not agree, the comparison cannot be interpreted and no other number in it means anything.
- Search outside the range you expect to pass. A premise verified only on the grid where it is used is not verified. The guard now asserts the naive version fails where it should.
- Prior art before code. An hour, at the start, every time.
- Determinism, or it is not evidence. Every model here reproduces exactly from a seed. A result you cannot regenerate is not a result about anything, including itself.
- Provenance travels inside the output. Every assumption is named in a value that ships with the numbers, so a figure cannot be quoted without the caveat that qualifies it.
- Published data cannot drift. The figures on these pages are generated and committed. A test re-runs each model using the parameters recorded in the committed file and fails if they disagree, so a stale number cannot outlive the model that justified it.
- Measurement and model are separate trees. Packages that measure may say so; packages that model may not, however plausible their inputs look. A modelled number that can reach an evidence path will eventually be cited as measured.
None of that makes a model correct. It makes the specific ways this kind of model goes wrong expensive to reach by accident, which is a smaller claim and the only one we can support.
Both studies we have published lost part of the argument they were built to make. We prefer that to the alternative, which is a model that agrees with us.