CI LIBRARY / STANDARDS
What a result from our simulations has to show before we report it.
A simulation can be made to support almost any conclusion. So a result is not evidence until it survives variation of the things we are uncertain about. This page explains why we hold ourselves to that rule and what a reported result has to include. The standard itself is a versioned document in the library repository; comment on it there.
Agent-based models are the right tool exactly where the averaging behind equilibrium models breaks down: networked, heterogeneous, supply-chain-shaped systems. That is also where they are easiest to misuse. A model with forty parameters can be tuned to any conclusion, and a reader has no way to tell a finding from a parameter setting.
Three research fields already have their own standards of evidence, and they do not agree. The standard below is our attempt to state, in one place and in the open, what a result from a simulation has to show before we report it. Scroll through the argument first; the requirements follow.
Adjust the agents, choose the random seed, choose how the outcome is measured, and cooperation appears or disappears on demand. That is not a flaw in one model. It is a property of simulation. The same rules, run three times with small changes nobody would notice, end in three different places.
An outcome from one model, one agent type, one seed, and one way of measuring is an artifact until it survives variation of the things we are uncertain about. It might be a finding. It might be a parameter setting. Nothing in the run itself says which, and the run you would report looks exactly like the ones you would not.
Mechanism design asks for a formal proof: show that the incentives are compatible. Agent-based modelling asks for generative sufficiency: grow the pattern from simple rules. Ecology asks for a phase portrait: the stable states and how deep they are. Each is right about something, and none alone is enough. Elinor Ostrom made the same point about real institutions and combined all three streams: no single one suffices.
Vary the agent model: rule-based agents, learning agents, language models, human subjects. Each relaxes a different assumption, and none is more realistic than the others in every respect. Vary the environment model too: a claim about a mechanism is strongest when it holds across structurally different models of the same domain. Report what survives both. Where results diverge, that is a finding, not a failure.
Ask how large a perturbation the system can absorb before it shifts regime, not only where it ends up. The answer has the shape of a basin of attraction: the set of initial states from which the desired outcome is still reached. Its size, as a function of the mechanism parameters, is the result. A single trajectory is an illustration. A phase boundary is a finding.
Every model states how far it has been validated: it runs; it replicates a known result; it is calibrated to detect the phenomenon it was built to detect; it has been validated across agent models; it has been replicated or used independently. The levels rank the evidence, not the realism of the agents. The model documentation travels with the code.
The methods break down in known places: while agents are still learning, when participants can change the rules or exit, and when a mechanism is enforced by construction. And stability says nothing about whether a state is desirable. A stable but harmful equilibrium is lock-in, not a success. Every stability claim is reported together with a separate welfare assessment.
These are not a gate. They are what a reader needs to re-run our reasoning and disagree with it precisely. The library provides tools for each requirement; the standard is written so that a model built elsewhere can meet the same requirements.
The canonical text is STANDARDS.md in the library repository. It is versioned by pull request. If a requirement is wrong, open an issue that names the result it would have blocked or let through.
"Under mechanism M the desired outcome is reached from a larger set of initial states than under the alternative" is a claim we make. "This will happen" is not.
What was intervened on, over which distribution of initial states, measured by which observable. The choice of reference frame moves numbers, sometimes by a lot. It is stated every time.
Two axes: the agent model (rule-based, learning, language model, human subjects; the kinds chosen are the ones that stress the assumption the claim depends on) and the environment model (structurally different models of the same domain, kept in a model register).
How large a perturbation the system absorbs before it shifts regime. The reported quantity is the basin fraction as a function of the mechanism parameters; phase boundaries are where it crosses one half. Headline results live on parameter sweeps; single trajectories are illustrations.
Comparative results use paired seeds (common random numbers) and a bootstrap confidence interval. A result is resolved when the interval excludes zero; otherwise it is reported as unresolved. No arbitrary sample sizes.
Predictions are recorded before a parameter sweep runs. A mismatch between the recorded prediction and the result is a finding and is reported as one. It is never tuned away.
Every parameter value is one of: anchored to a cited source or a replicated known result; chosen for legibility and labelled as such; or unanchored and swept. A value that is none of these does not appear.
What the model takes the world to be, its assumptions, what it leaves out, which known result it replicates, and which parameter the claim turns on. Stored next to the code, in the spirit of the ODD protocol for describing agent-based models, so a fork carries its documentation with it.
Every figure regenerates from committed experiment artifacts. Nothing is hand-drawn and no numbers are typed into figure code. A figure that cannot be regenerated is not published.
An open question stated plainly is preferred to a proxy metric or a confident sentence. Unknowns are reported in place, as part of the result.
Acceptance criterion, in words: a mechanism is robustly good for an outcome if, under every agent model and environment model tested, the outcome lies in a basin of attraction whose stability exceeds a threshold set relative to the alternative mechanisms in the same domain. This is a robustness claim across what was tested, not a guarantee about what was not, and not a claim of optimality.
Every model in the library is placed on the same five-level scale. The level is declared in the model documentation and enforced by its tests, so a reader knows how far to trust it. The scale ranks the evidence, not the agents: it says how thoroughly a model has been validated, not how realistic its agents are.
Exists, compiles, produces trajectories.
Reproduces at least one established result from the literature the model claims to belong to.
With controlled agents, the environment demonstrably produces the target phenomenon and the observables demonstrably detect it. The instrument is calibrated before it is pointed at anything new.
Environment fixed, a structurally different agent model substituted, and the level-2 signals measured for survival. The environment stays bit-exact; agent behaviour is pinned to a provider and version, with changes disclosed.
Reviewed, replicated, or built on by someone outside the group that built it.
Known limitations of the method: while agents are still learning the landscape is non-stationary, so stability is measured at checkpoints or after convergence. When participants can change the rules or exit, measured exit times are upper bounds. A mechanism enforced by construction has a trivially infinite basin; the relevant question there is verification of the constraint. And a stable but harmful equilibrium is lock-in, not a success: every stability claim is paired with a separate welfare assessment.
We are proposing it, not enforcing it on anyone else. It is a versioned document in the open, distilled from a paper in preparation and from practice already in the library. Apply it to a model built elsewhere; where it does not fit, that is exactly the comment we want.