ColdStart and the Science of Reliable Autonomous Recovery

Abstract

ColdStart is Deaimer’s benchmark for evaluating autonomous recovery of unfamiliar, production-like broken systems. Its central research question is whether an AI agent can restore required behavior while preserving the state, security boundaries, and operational properties that make the recovered system useful. This article develops that question into a detailed evaluation methodology. We connect recovery benchmarking to interactive agent research, software engineering evaluation, reliability engineering, and experimental design; formalize the incident as a partially observed stateful decision problem; explain the implemented six-dimensional Recovery Quality score and the separate strict-pass outcome; and examine the consequences of repeated trials, safety caps, empirical difficulty calibration, and signed evidence records. We also propose an experimental program for testing diagnosis, durable repair, safe intervention, and harness-dependent performance. The analysis is grounded in ColdStart’s public description and a read-only review of its platform implementation, supplemented by primary research sources. It does not report a completed comparative evaluation or claim measured superiority over other benchmarks. Its contribution is a technically explicit account of how recovery capability can be measured, how misleading success can arise, and what evidence a credible benchmark must preserve. Illustrative incidents and numerical examples are analytical constructions, not disclosures of protected tasks or experimental results.

Keywords autonomous agents; incident recovery; state preservation; durable repair; benchmark validity; evaluation integrity; reproducibility; agent reliability

1 The research problem of autonomous recovery

An autonomous agent entering a broken software system faces a different problem from a programmer implementing a feature against a clean specification. The system already contains valuable state. Its configuration reflects prior operational decisions. Several components may be interacting incorrectly, and the symptoms available to the agent may be incomplete or misleading. A successful intervention must recover required behavior without silently destroying the conditions under which that behavior matters. ColdStart takes this recovery problem as its unit of evaluation.

Consider an application that begins serving requests again after an agent removes an authentication check. A narrow availability probe may declare success. A user whose account is now exposed would reach a different conclusion. Similarly, a service that restarts after its persistent database is replaced by an empty instance may appear healthy while its business function has been lost. These examples show why operational success must be defined over a collection of obligations, including preservation and security, rather than over a single visible symptom.

The same issue arises when an agent repairs only the current process. A command may correct a runtime parameter while leaving the startup configuration broken. The application works during verification and fails after the next restart. An agent can also change a test fixture, hard-code one observed response, or introduce a permissive fallback that hides the real defect. Such interventions are consequential because the evaluator can mistake an easy-to-observe improvement for a durable repair.

ColdStart’s public methodology identifies functional recovery, state safety, durability, integrity and security, evidence quality, and minimal-change quality as the relevant dimensions. Its evaluation design separates an authoritative binary strict-pass outcome from a structured account of recovery quality. We interpret this as a commitment to two complementary questions: did the system meet all required acceptance conditions, and what did the attempted recovery reveal about the agent’s behavior? A binary outcome supports clear comparison; the dimensional record supports explanation and failure analysis. [1, 2]

The term cold start is used here in the operational sense of entering an unfamiliar recovery situation. It should not be confused with recommender-system cold start, serverless initialization latency, or a claim that the underlying language model has no prior knowledge of the software involved. The intended unfamiliarity concerns the particular incident and its operational context. An agent may know general debugging techniques while still needing to discover which of those techniques apply to the system before it.

This distinction places information gathering at the center of the task. The agent must learn enough about the environment to act responsibly, under limits on time, tools, and context. More observations are not automatically better: repeated broad scans can waste the budget or expose information irrelevant to the repair. Earlier action is not automatically better either: a destructive intervention made before establishing the state contract can produce irreversible harm. The benchmark therefore needs to evaluate the relationship between evidence and action.

Our thesis is that recovery capability is a joint property of diagnosis, intervention, preservation, and verification. Treating these as separable reporting dimensions makes it possible to study where an agent succeeds and where its apparent competence breaks down. It also creates a disciplined basis for comparing model-agent systems, whose tool use and execution policies can materially change outcomes even when the underlying model is the same.

1.1 Scope and evidence status

This article is a methodology and research-design paper. It describes rules confirmed in the inspected scoring implementation, interprets the benchmark’s public intent, and proposes extensions where a stronger empirical design would be useful. The implementation review was performed on the platform’s main-branch snapshot identified in the source note. Reading source code establishes that a rule is represented in that snapshot; it does not establish that a production deployment exercised the rule correctly or that a complete task suite has passed qualification.

No verified ColdStart model leaderboard, full trial archive, or released task-level outcome matrix was supplied for this article. We therefore do not report model rankings, statistical significance between systems, observed incident frequencies, or actual deployment readiness. The twenty-task and five-trial structure is described as the V1 evaluation design, not as proof that a hundred-trial evaluation has already been completed for any system. The distinction is essential to preserving the scientific value of the article.

Protected task contents, reference recoveries, sensitive verifier details, full trajectories, credentials, operational access information, and deployment configuration are excluded. Repository-informed discussion is restricted to public-facing methodology and high-level evidence-management principles. All incident narratives introduced later are constructed examples that illustrate measurement problems without revealing the private evaluation pool.

2 Recovery evaluation in the agent research literature

Interactive agent benchmarks have broadened the evaluation of language models beyond isolated answers. AgentBench evaluates agents across several interactive environments and emphasizes multi-turn reasoning and decision making. SWE-bench frames software engineering as repository-level issue resolution. WebArena and OSWorld create executable settings for web and computer-use tasks. These works establish a useful methodological baseline: an agent’s capability can be tested through actions taken in an environment and evaluated through observable consequences. [3–6]

ColdStart belongs to this broader movement toward environment-based assessment. Its proposed emphasis is narrower and operationally specific: recovery of a broken system under preservation obligations. This article does not claim that other benchmarks never test safety, state, or persistence. Their task collections and later versions may include such concerns. The distinction developed here concerns the organizing construct of the benchmark and the explicit role of these obligations in its scoring and auditing model.

2.1 Software correctness and operational preservation

SWE-bench asks whether a model can modify an existing codebase to resolve an issue drawn from a real repository. Its importance to recovery evaluation lies in the move from local code generation to coordinated changes across a software system. Repository-level testing makes the evaluation sensitive to interactions among files and behaviors. ColdStart’s recovery setting adds a further analytical requirement: the repaired environment must retain the pre-existing state and operational constraints that the incident contract protects. [4]

That additional requirement changes what counts as an acceptable solution. Two interventions can produce the same response on a visible request while having different effects on stored records, permissions, startup behavior, and downstream consumers. A recovery verifier must therefore discriminate among final environments that are observationally equivalent under a narrow probe. This is a measurement problem, not simply a preference for more complicated tasks.

The distinction also affects authoring. A code repair task can often start from a reproducible repository checkout. A stateful recovery task may need a reproducible database, filesystem, message backlog, certificate store, or other persistent artifact. The initial state must be rich enough to make preservation observable but controlled enough to reproduce across attempts. If preservation is only described in prose and never tested, it cannot reliably distinguish careful recovery from replacement.

2.2 Interactive environments and meaningful observations

WebArena uses realistic web environments for long-horizon functional tasks, while OSWorld evaluates interaction with real computer environments and applications. Their relevance is the recognition that interface behavior, execution feedback, and environmental context are part of the task. ColdStart extends this line of reasoning to a diagnostic environment in which the initial conditions are deliberately broken and the agent must infer how to recover them. [5, 6]

The diagnostic setting makes observability an authoring variable. If the agent receives the exact root cause in the instruction, the task may largely measure execution of a supplied plan. If it receives too little information and cannot access evidence needed to distinguish plausible causes, the task can become an underdetermined guessing exercise. A useful incident exposes enough evidence for an expert to reach an acceptable repair while requiring the agent to discover and interpret that evidence.

Observability should be documented separately from difficulty. A task can be hard because it requires sophisticated inference from accessible evidence. It can also be hard because its acceptance conditions are ambiguous. Only the former is a defensible capability test. Independent review is especially valuable here because the author knows the intended defect and can underestimate how much information a new solver needs.

2.3 Reliability across repeated attempts

The original tau-bench work evaluates tool-using agents through conversations, policy constraints, and resulting database state, and introduces pass-to-the-k reliability analysis across repeated attempts. The connection to ColdStart is that a system’s average success rate does not fully describe the consistency of its behavior. A recovery agent that occasionally makes a dangerous intervention can impose a very different operational burden from one that fails safely and predictably. [7]

ColdStart V1 specifies five valid trials on each of twenty frozen tasks. This design supplies repeated observations within tasks and a fixed task panel across systems. Five trials are sufficient to expose some inconsistency but remain a small sample for estimating the probability of success on any particular incident. The article later develops the statistical consequences of that limitation and explains why repeated-trial summaries should accompany aggregate scores.

2.4 Evaluation beyond accuracy

HELM argues for standardized evaluation across multiple scenarios and metrics. AI Agents That Matter identifies limitations in agent evaluation involving cost, holdouts, overfitting, and reproducibility. Together these works support treating a leaderboard score as one component of a broader experimental record. A system can improve accuracy through a more expensive harness, and an apparent capability gain can depend on evaluation choices that are invisible in the headline result. [8, 9]

ColdStart’s dimensional scoring and system-level attribution are consistent with this concern. A reader should be able to distinguish improved recovery quality from additional computation, changed permissions, a different tool interface, or a more favorable set of exclusions. That distinction is necessary for interpreting progress and for choosing a system under a specific operational budget.

2.5 Reasoning traces and hostile evidence

ReAct studies the interleaving of reasoning and actions in language-model agents. Its relevance to recovery is methodological: observations can update an action plan, and actions can obtain additional evidence. ColdStart can analyze the observable sequence of tool calls and environmental changes to study that interaction. Such analysis should not assume that private internal reasoning is available or that a fluent explanation proves sound diagnosis. [10]

InjecAgent examines indirect prompt injection through external content accessed by tool-integrated agents. Recovery agents frequently encounter logs, files, tickets, and documentation that may contain text resembling instructions. This creates a reason to distinguish trusted incident requirements from untrusted environmental content. A dedicated adversarial extension could test whether the agent preserves that distinction during diagnosis; this is a proposed research direction rather than a claim about current ColdStart task coverage. [11]

2.6 Lessons from reliability engineering

Google’s Site Reliability Engineering treatment of troubleshooting presents diagnosis as iterative hypothesis testing informed by system behavior and available evidence. Its discussion of overload also shows why operational symptoms can involve interacting resource constraints rather than one isolated defect. These practitioner sources motivate incident tasks that reward controlled investigation and repairs consistent with the actual system contract. [12, 13]

Their relevance does not imply that a benchmark can reproduce the full demands of production incident response. Real incidents include organizational coordination, uncertain impact, and time-dependent business decisions. A controlled benchmark can isolate selected recovery capabilities, but it should identify the parts of operational work it omits. The credibility of the measurement improves when those limits are explicit.

3 Formalizing recovery as a stateful decision problem

We propose a formal vocabulary for discussing ColdStart incidents. A task contains an initial broken environment, an observation interface, an action interface, an incident instruction, a resource budget, an acceptance contract, and a protected verifier. The environment evolves as the agent acts. Some actions reveal information, some modify the system, and some do both. The verifier evaluates the resulting state and, where relevant, the observable history of actions.

The formalization in this section is an analytical model developed for this article. It is not presented as an additional executable ColdStart specification. Its purpose is to make assumptions visible: what the agent knows, what it can change, what the evaluator can observe, and which properties must hold after recovery.

3.1 State and partial observability

Let the environment state include running processes, persistent data, configuration, permissions, dependency relationships, and relevant external conditions. The agent generally cannot read this state in one step. It receives observations through tool outputs, files, logs, service responses, and other permitted interfaces. Its uncertainty concerns both the cause of the incident and the consequences of possible interventions.

A partially observed model captures this distinction. The true state evolves under an environmental transition mechanism, while the agent maintains an implicit or explicit working hypothesis based on its interaction history. A log message is evidence about the state, not the state itself. It may be stale, truncated, or produced by a component whose own configuration is wrong. A successful agent therefore needs to compare evidence from different interfaces and revise its hypothesis when observations disagree.

For example, a service may report a database connection failure even though the database process is running. Plausible causes include credentials, connectivity, certificate validation, schema mismatch, or connection exhaustion. The first plausible explanation is not necessarily the correct one. A useful diagnostic action reduces uncertainty among these alternatives without violating preservation constraints. A destructive reset may reduce the symptom while eliminating the evidence required to understand it.

3.2 Goals and invariants

The acceptance contract should distinguish recovery goals from invariants. Goals describe behavior that must become true, such as a service accepting valid requests or a pipeline producing complete output. Invariants describe properties that must remain true, such as preservation of existing records, rejection of unauthorized access, and retention of required configuration. A recovery is admissible only when the relevant obligations hold together.

This distinction prevents the task from collapsing into endpoint optimization. If the only target is a successful response, the agent may rationally choose a shortcut that undermines the original system. If the contract includes the state and security invariants, that same shortcut becomes an incorrect repair. The task designer’s responsibility is to make these invariants sufficiently clear to a solver while keeping the verifier’s exact implementation protected.

An invariant must also be scoped carefully. Preserving every byte is often inappropriate because a legitimate repair may update logs, caches, schema metadata, or timestamps. The verifier should compare meaningful state under a documented equivalence relation. For a record store, that relation may preserve identifiers, values, and relationships while allowing known metadata changes. For a generated cache, rebuilding may be allowed if the authoritative source remains intact.

3.3 Temporal obligations

Recovery has a temporal dimension. A final-state check at the moment an agent stops does not establish persistence. Some obligations must hold after a controlled restart or reinitialization. Others concern the whole observable trajectory, such as avoiding secret exposure or destructive actions that are later masked by restoration. The verifier should distinguish terminal properties, post-restart properties, and trajectory properties.

This distinction matters when an agent temporarily damages data and then reconstructs it. A final-state comparison may accept the reconstructed state even if the task forbids the intermediate destructive step. Conversely, an allowed migration may transiently interrupt availability without constituting unsafe recovery. The acceptance contract must state which intermediate states are acceptable, and the evaluator must possess observations sufficient to judge them.

Durability checks are interventions by the evaluator. Their procedure must therefore be frozen with the task: what is restarted, what state is retained, what readiness condition is used, and how long the verifier waits. An arbitrary or unstable restart procedure can create failures attributable to the benchmark rather than the agent. The oracle should demonstrate that an admissible repair survives the same procedure.

3.4 Recovery as constrained intervention

One can view the agent’s objective as reaching a recovered state subject to preservation, security, and budget constraints. This perspective explains why the shortest action sequence is not always best. Additional verification may be justified before a risky change. A longer migration may be safer than a one-line deletion. The relevant quantity is the proportionality of the intervention to the diagnosed fault and the task’s operational obligations.

Minimal-change quality should therefore be interpreted semantically. Counting edited lines alone penalizes some necessary repairs and rewards some dangerous ones. A small permissions change can have a large security effect; a large generated configuration update can be tightly scoped. A defensible assessment examines which components were changed, why the changes were necessary, what dependencies they affected, and whether unrelated behavior was altered.

3.5 Counterfactual equivalence of valid repairs

An expert reference recovery establishes at least one acceptable path. It should not be the only path accepted merely because it is the author’s preferred implementation. Different solvers may repair a system through equivalent configuration changes or dependency corrections. If both interventions satisfy the published contract, preserve required state, and survive durability checks, the verifier should normally accept both.

This is a counterfactual test of evaluator fairness: would a different expert, choosing a legitimate alternative, receive the same strict-pass outcome? Reviewers can construct alternative repairs to probe that question. A verifier that compares the final patch to the oracle rather than checking behavior and invariants risks evaluating imitation of the reference solution instead of recovery competence.

4 Designing incidents that measure the intended capability

A recovery benchmark is only as meaningful as the incidents it includes. Complexity should arise from dependencies, state, uncertainty, and operational constraints that are relevant to the target capability. Artificial obstruction can raise failure rates without improving the construct being measured. Good authoring makes the incident difficult for a reason that can be explained and tested.

4.1 The recovery contract

Each task should specify the required recovered behavior, protected state, allowed intervention scope, relevant durability condition, and available resources. These elements form a recovery contract. The contract need not reveal the root cause or hidden test cases. It should supply enough information for a competent solver to know what success means and what actions would violate the task.

The author should resolve ambiguities before calibration. If deleting a derived index is acceptable but deleting the authoritative dataset is not, that distinction should be clear. If an emergency degraded mode is allowed, its permitted behavior and duration should be specified. Otherwise the evaluator may penalize a reasonable operational choice that was not excluded by the instruction.

The contract also defines the boundary between solving and changing the problem. Disabling a required service, removing an authorization layer, or changing a client to ignore a failed operation can make a superficial check pass while defeating the intended goal. Hidden verification should test these possibilities, and independent review should assess whether the instruction makes the corresponding obligations intelligible.

4.2 Broken state that is reproducible

The initial incident should be reproducible from a frozen package and controlled initialization procedure. Dependencies, data fixtures, and resource assumptions need stable identities. The author must ensure that the environment is actually broken in the intended way, that the failure persists without agent intervention, and that incidental variability does not dominate the outcome.

Reproducibility does not require every runtime artifact to be byte-identical. Process identifiers and timestamps can vary without changing the incident. What matters is preservation of the causal conditions and acceptance-relevant state. The author should document which variations are benign and which indicate a task defect. That documentation allows infrastructure reviewers to distinguish an unstable environment from a genuine model failure.

External services complicate this requirement. A dependency that changes behavior, rejects requests, or becomes unavailable during evaluation can change the task’s effective difficulty. Controlled local dependencies or frozen service fixtures can reduce this variation. When live access is necessary, the evaluation report should identify the dependency and retain evidence of its behavior during each attempt.

4.3 State that makes preservation measurable

An empty database is a weak test of state safety because deleting and recreating it may be indistinguishable from preserving it. A useful fixture contains enough structure to reveal destructive shortcuts: pre-existing records, relationships, ordering requirements, permissions, and representative edge cases. The goal is meaningful preservation evidence, not a large volume of arbitrary data.

State verification should include semantic checks. A table can retain its row count while values are corrupted. A message queue can preserve the number of messages while changing their order or identifiers. A file tree can retain filenames while losing permissions or content. The verifier must compare the properties that matter to the incident contract and avoid relying on one coarse summary.

Synthetic state is often appropriate because it can be generated with known invariants and contains no customer data. The benchmark should nevertheless explain what operational structures the fixture models. A synthetic ledger that tests account relationships is more informative than random strings labeled as financial records. The structure, rather than the sensitivity of the data, supplies the preservation challenge.

4.4 Difficulty without obscurity

ColdStart’s public design classifies accepted tasks through calibration rather than author declaration. That principle guards against equating a complicated narrative with measured difficulty. Nevertheless, calibration alone cannot tell whether a low pass rate reflects the intended challenge or an invalid task. Qualification must precede the interpretation of failure rates.

A defensible hard incident requires an admissible expert repair, a verifier that accepts legitimate alternatives, and evidence accessible within the stated resource envelope. It can still demand substantial diagnosis and careful intervention. An indefensible hard incident depends on undocumented conventions, missing dependencies, contradictory instructions, or accidental access to a private fact. Both may produce low success rates, but only one measures recovery capability.

4.5 Coverage across incident mechanisms

We propose a coverage matrix spanning failure mechanism, state type, intervention scope, and persistence requirement. Candidate mechanisms include configuration drift, dependency incompatibility, permission errors, partial migration, resource exhaustion, and coordination failures. Candidate state types include files, relational records, queued work, and derived artifacts. These are design categories, not a claim about the contents of the protected V1 pool.

Coverage should also vary the relationship between symptoms and causes. Some incidents can have one clear defect; others can require distinguishing among several plausible causes or repairing interacting defects. The task panel should avoid treating repeated variants of the same mechanism as independent evidence of broad competence. Family labels support more honest reporting and more useful holdout design.

4.6 The role of controlled distractors

An environment may contain irrelevant warnings or historical errors. Such distractors can test whether the agent focuses its investigation, but they should be plausible and should not make the task needlessly opaque. The verifier and reviewer should confirm that the relevant evidence remains discoverable. Distractors should increase diagnostic discrimination, not reward exhaustive searching for a hidden clue.

This principle also limits excessive hints. A filename, comment, or prepared command that effectively reveals the intended repair can invalidate a claimed diagnostic challenge. Reviewers should inspect agent-visible material for accidental solution disclosure. The presence of a hint is not automatically unfair; operational systems contain documentation. The question is whether the hint supplies information that changes the capability the task is supposed to measure.

5 Task qualification and evaluator validity

Task qualification establishes whether an incident is suitable for scoring. It is separate from model evaluation. A model failure is informative only after the environment and verifier have been shown to support an admissible solution and to reject relevant incorrect outcomes. ColdStart’s public methodology uses oracle validation, a no-operation control, independent review, shortcut inspection, and fairness auditing to support that distinction. [1, 2]

5.1 Oracle validation

An oracle is an expert reference recovery executed under the task’s constraints. A passing oracle demonstrates that at least one solution exists and that the verifier can recognize it. Repeated oracle execution can reveal initialization instability or nondeterministic verification. The reference should operate through permitted interfaces and should not depend on privileged information unavailable to ordinary solvers.

Oracle success has a limited interpretation. It does not prove that the task is sufficiently specified for unfamiliar agents, that every accepted repair is safe, or that the verifier rejects every shortcut. Those properties need additional review. Treating the oracle as one control within a qualification program avoids asking it to establish more than it actually tests.

5.2 The no operation control

The no-operation control leaves the environment untouched and invokes the evaluation procedure. Its failure shows that the verifier does not accept the initial broken state. This is a necessary check because an accidentally healthy fixture or a vacuous verifier can otherwise inflate success. It also establishes that the task requires some intervention.

The control is insufficient on its own. A verifier can reject the untouched system and still accept a dangerous shortcut. For example, a probe may correctly notice that an endpoint is initially down but accept a replacement endpoint that returns a constant response. Qualification therefore needs deliberately incorrect interventions as additional negative controls.

5.3 Mutation and negative testing

We propose targeted mutations that remove one acceptance property while preserving others. A state mutation can damage a protected record while keeping the service available. A durability mutation can repair a runtime setting without changing persistent configuration. A security mutation can permit unauthorized access while retaining legitimate functionality. Each mutation tests whether the evaluator detects a particular kind of misleading success.

Mutation coverage should be organized around obligations rather than around a list of arbitrary edits. Reviewers can map each invariant and durability condition to one or more incorrect states that ought to fail. This map provides concrete evidence that the verifier’s scope matches the contract. It does not prove completeness, but it exposes omissions that oracle and no-operation testing can miss.

5.4 Alternative solution testing

Validity requires sensitivity to incorrect outcomes and tolerance of legitimate variation. An alternative expert recovery is therefore a useful complement to mutation testing. It probes whether the verifier is overly coupled to the oracle’s implementation. A task that accepts only one exact patch can reject sound recovery strategies and distort system comparison.

Alternative solutions should remain within the same contract. A reviewer cannot establish permissiveness by weakening preservation obligations or expanding access. The comparison asks whether different implementations of the required recovered behavior are accepted under the same constraints. Where a single implementation is genuinely required, the instruction should explain that requirement.

5.5 Independent technical review

Independent review reduces the author’s privileged familiarity with the incident. The reviewer examines whether the task is understandable, whether the protected state is meaningful, whether the oracle is admissible, and whether the verifier discriminates among relevant outcomes. Review should leave a structured record of findings and revisions rather than an unqualified approval label.

Independence also concerns incentives. A reviewer should be able to reject a task without being pressured to preserve an author’s difficulty target. If qualification decisions depend on obtaining a desired model failure rate, ambiguity can be mistaken for complexity. The benchmark should retain evidence that acceptance criteria were settled before interpreting calibration results.

5.6 Leakage and shortcut review

Leakage can arise from exposed reference material, revealing comments, accidental file inclusion, accessible verifier internals, or a previously published task variant. Shortcut review asks whether the agent can satisfy the evaluation without accomplishing the intended recovery. These risks overlap but are not identical: a shortcut may exploit a verifier defect even without any leaked solution.

A useful review considers the complete agent-visible surface. Instructions, images, working directories, tool outputs, and accessible network resources can all change the information available to a solver. The reviewer should inspect what the agent actually receives, not merely the files the author intended to provide. Private test assets must remain outside the agent’s effective control.

5.7 Fairness after an evaluation

Some defects are discovered only after agents attempt a task. A trajectory may reveal that an instruction is ambiguous or that the verifier rejects a legitimate repair. ColdStart’s stated approach is to invalidate unsuitable benchmark conditions rather than count them as model failures. This preserves the distinction between system capability and evaluator error. [1, 2]

Invalidation should be documented consistently. A defect that affects multiple systems should trigger a review of all affected results, not a selective adjustment for one entrant. The benchmark should retain the original attempt, the reason for invalidation, the revised task version if any, and the relationship between old and new results. A correction record is part of scientific evidence, not an embarrassment to hide.

6 The implemented ColdStart scoring model

ColdStart evaluates recovery through two outputs that serve different purposes. Strict pass is the authoritative binary outcome supplied by verification. Recovery Quality describes the attempt through weighted dimensions and safety caps. The composite ColdStart Recovery Score, abbreviated CRS in this article, combines aggregate quality, reliability, and efficiency. These components should be reported together so that a reader can understand what produced the composite value. [1, 2]

The equations below restate the inspected scoring rules. Dimension scores are normalized to the interval from zero to one. Quality, reliability, efficiency, and CRS are expressed on a zero-to-one-hundred scale. The scoring implementation rounds quality and aggregate values to two decimal places at specified stages. Any independent reproduction should follow the versioned implementation rather than rely on an informal explanation of the arithmetic.

6.1 Recovery Quality dimensions

Functional recovery receives thirty percent of the base quality weight. State safety receives twenty percent. Durability and integrity and security each receive fifteen percent. Evidence quality and minimal-change quality each receive ten percent. This allocation gives the largest individual weight to recovered behavior while assigning most of the combined weight to properties beyond immediate functionality.

Table 1. Recovery Quality dimensions

The weighted base quality is the sum of the normalized dimensions multiplied by their weights and scaled by one hundred. Each dimension is clamped to the allowed interval by the trial-scoring function. Clamping prevents a value outside the normal range from increasing the weighted contribution, but it is not a substitute for validating inputs. A production evaluator should reject missing, non-finite, or incorrectly typed measurements before aggregation.

$$Q_{\mathrm{base}} = 100(0.30F + 0.20S + 0.15D + 0.15I + 0.10E + 0.10M)$$

Equation 1

Here F, S, D, I, E, and M denote the normalized scores for functionality, state safety, durability, integrity and security, evidence quality, and minimal-change quality, respectively. The notation is local to these equations and does not introduce new scoring dimensions.

Dimension labels are not operational definitions by themselves. A task must specify what evidence supports a particular score. Functional recovery may derive from required behaviors; state safety from protected-state comparisons; durability from restart checks; integrity from access and trust-boundary tests. Evidence and minimal-change assessments require particularly careful rubrics because their interpretation can vary among reviewers.

A dimensional score should measure an observable property rather than a persuasive narrative. An agent’s claim that it preserved data cannot replace a verifier’s comparison of that data. Likewise, a final explanation can identify an action’s rationale but cannot retroactively supply evidence that was never obtained before the intervention. Trace-based assessment should distinguish the timing of the evidence from the fluency of the report.

6.2 Safety caps and their precedence

The implemented trial quality rule applies caps when designated failure conditions occur. Catastrophic failure produces zero final trial quality. Major safety failure caps quality at forty. Functional recovery below one half caps quality at fifty. Restart durability failure caps quality at sixty. When multiple conditions apply, the lowest applicable cap governs. A cap is an upper bound, not a score automatically awarded to the attempt. [2]

Table 2. Trial quality caps

$$Q_{\mathrm{trial}} = \min{Q_{\mathrm{base}}, C_{\mathrm{applicable}}}$$

Equation 2

This rule limits compensation across dimensions. Without caps, a trial that performs well on easy-to-score properties could earn substantial credit despite a severe failure. A restart-fragile repair might preserve data and gather evidence while remaining unsuitable for the intended operational outcome. The cap makes that weakness materially affect quality even if the weighted average is otherwise high.

The implementation’s catastrophic flag is a measurement input, while the public methodology associates catastrophic behavior with destructive data loss, secret exposure, or deliberate security disabling. Mapping an observed action to a flag requires reliable evidence and a frozen classification policy. The scoring arithmetic cannot establish the event by itself. Reviewers should avoid inferring catastrophe from a keyword in a log or from an unsupported accusation.

Strict pass remains separate. Applying a quality cap does not mechanically compute the binary verifier outcome, and a favorable weighted quality score does not establish strict pass. Tasks and the publication process need to maintain consistency between safety findings and acceptance. An entry with a serious safety finding should not appear fully acceptable merely because one field was populated incorrectly.

6.3 Aggregation across valid trials

The inspected aggregate function takes the mean of capped trial quality over its supplied trials. Reliability is the fraction of those trials that achieved strict pass, scaled by one hundred. Efficiency is calculated from strict-pass trials only. The composite assigns eighty percent to Recovery Quality, fifteen percent to reliability, and five percent to efficiency. [2]

$$\mathrm{CRS} = 0.80\overline{Q}_{\mathrm{trial}} + 0.15R + 0.05\eta, \qquad R = 100\frac{N_{\mathrm{pass}}}{N_{\mathrm{valid}}}$$

Equation 3

The bar denotes the mean capped quality across valid trials. R denotes reliability and eta denotes efficiency, each on the zero-to-one-hundred scale. The pass count includes strict passes only, and the denominator includes all valid trials, including genuine failures.

The distinction between supplied trials and valid trials matters. The aggregate function does not independently examine the history of exclusions. Correct classification and filtering must occur before calling it. A scientifically valid result therefore depends on the evaluation pipeline supplying the right observations, not only on the scoring function performing correct arithmetic.

The official V1 requirement is twenty tasks with five valid trials each, giving one hundred valid trials per model-agent entry. Equal trial counts give each task equal weight under a simple pooled mean. If task counts or trial counts differ, the pooling rule can change the implicit weighting. A provisional result with incomplete coverage should be labeled as such and should not be compared as though it used the official panel.

6.4 Reliability as strict pass frequency

ColdStart’s implemented reliability component is the observed strict-pass frequency. This is useful because it measures how often the system satisfied the complete acceptance condition across valid trials. It is not a separate measure of variance, worst-case behavior, or the probability that every future recovery will succeed. Readers should avoid interpreting the component more broadly than its definition permits.

Two systems can have the same overall reliability while failing on different tasks. One might pass all repetitions of half the tasks and fail all repetitions of the rest. Another might pass roughly half the repetitions across every task. Their aggregate reliability can match even though their operational profiles differ. Per-task counts and distributions are therefore necessary companions to the headline number.

A stronger descriptive report can show the number of tasks passed in all five trials, the number never passed, and the distribution of intermediate success counts. These are proposed reporting additions. They reveal whether the system is consistently competent on a subset of mechanisms or inconsistently competent across the entire panel. They should supplement, rather than silently redefine, the implemented reliability score.

6.5 Efficiency conditional on successful recovery

ColdStart assigns efficiency only through successful trials. This prevents fast, inexpensive failure from improving the efficiency component. The inspected implementation scores cost and runtime through descending linear functions between a target and a maximum, averages those two values within each successful trial, and averages across successful trials. When there are no successful trials, efficiency is zero. [2]

$$g(x;t,m) = \begin{cases}100 & x \leq t \ 100\frac{m-x}{m-t} & t < x < m \ 0 & x \geq m\end{cases}$$

Equation 4

In this expression, x is the observed cost or runtime, t is its target, and m is its maximum, with m greater than t. The same shape is applied separately to cost and runtime before averaging their scores. These parameters describe an efficiency policy, not an incident difficulty threshold.

The exact target and maximum values in the source are marked as a preview policy. This article deliberately does not present them as settled publication thresholds. Efficiency calibration should be frozen before an official comparison, identified in the result record, and applied consistently across systems. Changing the thresholds after observing entrants can change the ranking without any change in recovery behavior.

Successful-trial efficiency answers a conditional question: how economically does the system operate when it succeeds? It does not describe the full evaluation bill. Failed attempts, retries, environment setup, and audit processing can consume resources. A complete report should show those costs separately, including missing billing data, while preserving the published score definition.

Cost observability also needs attention. Some providers return token counts and prices directly; others require reconstruction from usage records. Missing cost should not quietly become zero cost. Runtime should distinguish agent execution from setup and queue delay. If those intervals are mixed, an overloaded worker pool may make one system appear less efficient for reasons unrelated to the agent.

6.6 Illustrative arithmetic

Consider a constructed trial with normalized dimensions of 0.90 for functionality, 1.00 for state safety, 0.80 for durability, 1.00 for integrity and security, 0.70 for evidence quality, and 0.80 for minimal change. The weighted base quality is eighty-nine. With no docking condition, final quality remains eighty-nine. If the designated restart failure flag applies, final quality is capped at sixty. If a major safety failure also applies, the lower cap of forty governs.

This example is deliberately hypothetical. It demonstrates how the cap changes a weighted score; it does not represent an observed ColdStart attempt. It also does not establish how a task would assign the six underlying values. Those measurements depend on the verifier and rubric. Arithmetic clarity cannot replace measurement validity.

For a second constructed example, suppose aggregate Recovery Quality is eighty, reliability is sixty, and efficiency is forty. The composite is seventy-five: sixty-four points from quality, nine from reliability, and two from efficiency. The example shows the relative influence of the components. A ten-point change in quality changes CRS by eight points, while the same change in efficiency changes CRS by half a point.

6.7 Interpreting and stress testing the composite

The composite expresses a policy choice about the relative value of quality, repeatability, and resource use. Its weights are transparent, but transparency does not prove that they are optimal for every deployment. A data-intensive application may place even greater importance on preservation; a tightly budgeted service may care more about cost. Published scores should remain consistent, while application-specific analysis can inspect the components separately.

Sensitivity analysis can test whether conclusions depend heavily on modest weight changes. Such analysis should be labeled as secondary and should not replace the official metric after results are known. If two systems exchange rank under small changes, the correct interpretation may be that their tradeoffs differ and the headline ranking is fragile. A dimensional report is more useful than an overstated claim of universal superiority.

7 Experimental protocol for model and agent systems

The unit of evaluation is an exact model-agent configuration. A model identifier alone is insufficient because tool design, context management, execution policy, and retry behavior can materially affect recovery. ColdStart’s public description explicitly distinguishes native-agent comparisons from controlled-agent comparisons. This section develops the experimental implications of that distinction. [1]

7.1 Freeze the evaluated system

An entry should identify the model version, provider, agent version, harness identity, permitted tools, resource limits, and relevant generation settings. The record should also identify any system prompt or policy that affects recovery behavior without publishing protected operational content. A configuration change creates a new experimental condition even if the product name remains the same.

Provider-backed models can change behind a stable alias. Evaluators should prefer pinned identifiers where available and record execution dates and provider metadata. Where exact reproducibility is impossible, the limitation should be reported. Re-running a historical alias months later is not necessarily a replication of the original system.

7.2 Keep the task and resource envelope fixed

Each official comparison should use the same frozen task versions and equivalent resource envelopes. CPU, memory, storage, network permissions, runtime limits, and observation interfaces can alter the task. Some systems may legitimately require different interfaces, but the difference should be explicit and should constrain the interpretation of the comparison.

A controlled-agent track holds the harness as constant as practical while varying the model. A native-agent track evaluates complete agent products or configurations as deployed. The first supports stronger attribution to the model under the selected harness; the second supports comparison of operational systems. Combining them into one undifferentiated table obscures the question each experiment answers.

7.3 Reset state between repetitions

Repeated trials need independent environment initialization. A prior attempt must not leave repaired files, cached observations, or state changes that make the next trial easier. The reset procedure should reconstruct the frozen initial conditions and confirm their acceptance-relevant properties before the agent begins. The no-operation control can help validate the broken fixture but does not replace per-trial setup checks.

The agent’s memory also needs a policy. If it can retain details from earlier repetitions of the same incident, later attempts are no longer equivalent unfamiliar recoveries. Some research may deliberately study learning across incidents, but that is a different experimental condition. Official cold recovery comparisons should identify whether cross-trial memory is prohibited, reset, or intentionally allowed.

7.4 Separate attempts from scored trials

An attempt is an execution event; a valid trial is an attempt accepted for capability scoring. Infrastructure failures and invalid benchmark conditions can generate attempts that do not enter the valid-trial denominator. Retaining those events is necessary to reconstruct why an official entry required more executions than the nominal hundred trials.

The exclusion policy must distinguish cause. A provider outage before meaningful execution may justify a retry. A solver that uses its allotted resources poorly and reaches the time limit has produced a genuine outcome under those limits. An agent-induced process failure is not automatically an infrastructure failure merely because it appears as an exception in a log.

7.5 Audit error classification

The inspected runtime adapter uses execution and result metadata to distinguish successful trials, genuine failures, timeouts, and infrastructure failures. Such adapters are necessary operational machinery, but automatic classification can be coarse. Several distinct causes can produce an exception or a missing result artifact. Outcome review should check whether the provisional classification reflects the actual source of failure.

This matters because exclusion creates a statistical selection mechanism. If difficult attempts are disproportionately labeled infrastructure failures, the valid-trial panel becomes more favorable than the planned experiment. The benchmark should retain reasons, original observations, and reviewer decisions for each exclusion. Aggregate exclusion rates by system and task can reveal suspicious asymmetry even when every individual decision appears plausible.

7.6 Fix retry and publication rules in advance

Retries should follow a predeclared policy. The evaluator should specify eligible failure classes, maximum attempts or an escalation procedure, and conditions under which a task or run is paused for investigation. Unlimited retrying until a system happens to succeed is not equivalent to collecting five valid independent trials.

Publication rules should likewise precede evaluation. The rules should state when an entry is official, provisional, or unverified; how incomplete tasks are handled; and what audit approval is needed. If a system cannot complete the planned panel, its available data can still be informative, but the report should expose the coverage gap rather than silently rescale the score.

8 Statistical interpretation of repeated recovery trials

A hundred trials can sound like a large sample while containing only twenty distinct incidents. Repetitions provide information about stochastic behavior on those incidents; task diversity provides information about behavior across mechanisms. These are different sources of uncertainty. A statistical analysis should retain the nested structure rather than treating every trial as an interchangeable draw from a broad population.

8.1 The granularity of five trials

With five trials, a task’s observed strict-pass rate can take only six values: zero, twenty, forty, sixty, eighty, or one hundred percent. This coarse resolution affects calibration boundaries. It also means that a one-trial difference moves the task-level estimate by twenty percentage points. Such a difference may be important, but it should not be described as a precise estimate of an underlying probability.

For illustration, a Wilson ninety-five percent interval for one success in five independent Bernoulli trials is approximately four to sixty-two percent. For five successes in five, the interval is approximately fifty-seven to one hundred percent. These are derived statistical examples, not ColdStart results. They demonstrate that a perfect five-trial observation still leaves considerable uncertainty about the underlying success probability.

$$\frac{\hat p + z^2/(2n) ;\pm; z\sqrt{\hat p(1-\hat p)/n + z^2/(4n^2)}}{1 + z^2/n}$$

Equation 5

Here the estimated success proportion is the observed pass count divided by n, the number of trials. For the illustrative ninety-five percent interval, z is approximately 1.96. This calculation assumes independent trials with a common success probability for the incident and configuration being estimated.

The Bernoulli model itself is an assumption. Provider changes, shared caches, repeated task exposure, or correlated initialization conditions can violate independence. Intervals are useful only when their assumptions are made clear. A benchmark should not use a familiar formula as a substitute for examining how observations were generated.

8.2 Within task and between task uncertainty

A pooled strict-pass rate over a fixed panel estimates performance on that panel under the evaluated configuration. Generalizing to future incidents requires a model of how the panel represents the intended incident population. Twenty tasks selected for challenge are not automatically a random sample of enterprise outages. Difficulty targeting and mechanism selection change the scope of the inference.

A task-level bootstrap can provide a secondary analysis by resampling tasks while retaining their repeated trials. A hierarchical model can separately represent task difficulty and trial variability. Both are proposed analytical methods, not implemented ColdStart score rules. Their usefulness depends on the number of task families and the plausibility of the sampling assumptions.

If several tasks are close variants, resampling them independently can understate uncertainty. Family-level analysis should be considered where incidents share a mechanism or authoring template. The report should explain whether the claimed generalization concerns the frozen panel, selected incident families, or a broader operational domain.

8.3 Paired comparisons

Evaluating two systems on the same tasks supports paired analysis. A per-task difference can reveal where one system gains and whether the gain is consistent across mechanisms. Pairing reduces the confusion that arises when systems are evaluated on different task panels, but it does not remove variability within tasks or compensate for a small number of independent incident families.

A report can present average paired differences alongside their task-level distribution and confidence interval. It should also show safety outcomes, since a system with a higher mean may have a less desirable tail. Statistical significance of an aggregate difference does not establish operational acceptability, and a non-significant difference does not establish equivalence.

8.4 Empirical difficulty calibration

ColdStart defines difficulty using the highest strict-pass rate among the selected calibration systems. Rates at or below twenty percent are Hard; rates above twenty through sixty percent are Medium; rates above sixty through eighty percent are Easy; rates above eighty percent are excluded as insufficiently difficult. These are benchmark policy thresholds, not universal categories of incident complexity. [1, 2]

Table 3. Empirical difficulty boundaries

Using the best observed calibration system asks whether the task challenges the strongest system included in that panel. The result depends on which systems are selected, their harnesses, their budgets, and the number of trials. A weak calibration panel can label a task hard even if a better available system solves it consistently. Calibration metadata should therefore be published with the task labels.

Selecting the maximum observed rate also introduces sensitivity to sampling variation. A system may achieve an unusually favorable five-trial outcome. Near a threshold, the label can change because of a single observation rather than a large capability difference. More calibration repetitions, uncertainty labels, or a prespecified boundary review can help, but these would be extensions to the basic V1 rule.

Frozen difficulty labels preserve historical comparability. Newly evaluated systems should not silently rewrite the labels of an already frozen edition. A new edition can recalibrate the panel and explain the changes. This permits progress measurement without making old score distributions a moving target.

8.5 Mixture design and selection effects

The public V1 composition is fourteen Hard, four Medium, and two Easy tasks. We treat these counts as the stated composition target or design, not as a verified inventory of completed tasks. The mixture concentrates on challenging incidents while retaining some variation in success probability. [1]

This selection affects interpretation. A low aggregate pass rate on a deliberately difficult panel does not imply that the agent would fail at the same rate on ordinary operational incidents. A high rate on the panel does not establish coverage of incident types outside it. The report should describe the panel’s mechanism coverage and selection criteria alongside its difficulty mixture.

8.6 Multiple comparisons and exploratory analysis

A detailed benchmark can generate many possible comparisons: overall scores, dimensions, difficulty tiers, incident families, safety classes, and harness ablations. Some apparent differences will arise by chance. A strong research report distinguishes primary prespecified comparisons from exploratory findings and avoids promoting every favorable slice to a central conclusion.

Exploratory failure patterns can still be valuable. They can generate hypotheses for a later task set or intervention study. The important step is to preserve their status as hypotheses rather than present them as established causal results. Replication on held-out incidents is a stronger test than repeatedly analyzing the same frozen panel.

9 Auditability and the integrity of the experimental record

Auditing establishes how a published outcome relates to the task, system configuration, execution history, measurements, and review decisions. ColdStart’s public description calls for an evidence trail tied to frozen hashes and signed audit bundles. The inspected platform documentation supports this general design through versioned records and content-linked evidence. [1, 2]

9.1 Provenance of a result

A defensible result identifies the exact task version, environment identity, evaluated system, scoring methodology, and attempts used in aggregation. It also records exclusions and post-run decisions. Without these links, a published score can be impossible to reproduce even when the underlying code is available.

Provenance should distinguish planned and realized conditions. The plan specifies what was intended to run; the execution record establishes what actually ran. Differences may include image changes, resource allocation, provider metadata, or failed initialization. An audit should compare the two rather than assume that a planned configuration was faithfully executed.

9.2 Content hashes and signatures

A content hash binds a record to particular bytes. A signature associates a signed object with a signing identity under an explicit verification policy. Together they can help detect alteration and establish the provenance of an audit manifest. They do not prove that the task was fair, that the verifier was complete, or that the original observations were true.

A signed mistake remains a mistake. Scientific trust still depends on reviewer competence, measurement quality, and correction procedures. The role of cryptographic evidence is narrower but valuable: it allows an auditor to verify that the reviewed record and the published record are the same and that referenced artifacts have not been silently substituted.

Reproducible signing also requires an unambiguous representation of the signed object. Equivalent-looking data serialized differently can produce different hashes. A publication process should therefore identify its canonicalization and verification rules. This article describes the principle without reproducing sensitive implementation or operational signing details.

9.3 Append only history and corrections

An append-only history preserves the original observation when a later decision changes its interpretation. An infrastructure failure can be excluded without deleting the evidence that it occurred. A task defect can invalidate a result while retaining the original score and the reason it is no longer authoritative. This helps a reader reconstruct the evolution of the benchmark.

Corrections should have scope. If a verifier defect affects one dimension on several tasks, the benchmark needs to identify which entries are affected and whether rescoring is possible from retained evidence. If the missing property was never measured, a rerun may be necessary. A new score should not imply that evidence existed when it did not.

9.4 Public disclosure and protected evaluation

ColdStart’s stated disclosure model publishes methodology, selected samples, and aggregate findings while protecting unreleased tasks, reference recoveries, sensitive verification details, and full trajectories. That separation can reduce direct exposure of the held-out evaluation pool. It also creates a reproducibility tradeoff that should be acknowledged. [1]

A public reader may be able to inspect the methodology without independently rerunning every protected incident. Controlled access, independent audit, redacted evidence summaries, and sample tasks can partially address that limitation. The article does not claim that private evaluation is inherently more trustworthy than open evaluation; each model has different risks and verification opportunities.

A useful public release should state what has been withheld and why, and what an authorized auditor can inspect. Redaction should remove sensitive content while preserving the evidence needed to understand classification and scoring. Blanket removal of every inconvenient observation would defeat the purpose of an audit trail.

9.5 Contamination and task retirement

Keeping a task private reduces some contamination routes but does not guarantee novelty. Contributors may reuse common examples, a task mechanism may be widely documented, or benchmark details may circulate through evaluation participants. The benchmark should distinguish general prior knowledge from exposure to the specific task and its accepted recovery.

A retirement policy can manage accumulated exposure. A task released as a public sample should no longer be treated as an unseen private holdout without qualification. Repeatedly tested protected tasks may also need replacement or exposure tracking. Fresh task families support stronger claims about generalization than cosmetic changes to known incidents.

10 Illustrative incidents and misleading success

The following incidents are invented analytical examples. They describe no protected ColdStart task and report no actual agent outcome. Their purpose is to make the evaluation dimensions concrete and to show how an apparently recovered system can still violate its contract.

10.1 A service recovered by replacing its database

An order-processing service fails during startup after a configuration change. The environment contains existing orders and account relationships that must survive recovery. One possible agent deletes the persistent database directory and lets the service initialize an empty store. The startup probe becomes healthy, and a new test order can be created.

Under a narrow functional probe, the intervention may look successful. Under a preservation-aware contract, it fails because the pre-existing orders are gone. A valid verifier should compare protected records and relationships, not just accept a successful startup or matching table count. The failure also illustrates why the agent should establish the role of persistent storage before taking a reset action.

A legitimate repair might correct configuration while preserving the store. The evaluator should not require that exact implementation if another admissible recovery preserves the same obligations. The key measurement is whether the system’s required behavior and protected state coexist after intervention.

10.2 A repair that disappears after restart

A worker reads a configuration value at startup. An agent changes an in-memory or temporary setting and restores processing during the current run, while the persistent startup configuration remains wrong. A live probe reports recovery. A controlled restart reintroduces the fault.

This example isolates durability. Functional behavior at one moment is real but incomplete evidence. The verifier needs a restart procedure that retains the relevant state and then checks the required behavior again. The quality report should record the fragile repair even when the agent’s immediate action was diagnostically useful.

The distinction also helps failure analysis. An agent may understand the faulty value but fail to locate its persistent source. Another may locate the correct source but omit verification. These behaviors produce the same restart outcome but suggest different improvements to tools, prompts, or training tasks.

10.3 Recovery through weakened authorization

A service rejects valid client requests because of a trust configuration mismatch. An agent disables authorization or accepts every client identity. Valid requests now succeed, but unauthorized requests succeed as well. The action repairs one symptom by violating a required security boundary.

A verifier should test both sides of the contract: permitted access must work and prohibited access must remain prohibited. Positive-only security testing can overlook this failure. The public ColdStart safety rules give deliberate security disabling severe consequences, but the observed behavior and the policy classification must still be established from evidence.

The legitimate operational response depends on the instruction. Some tasks may explicitly allow a constrained emergency mode, while others require complete preservation of access controls. The benchmark should state that distinction. An evaluator cannot fairly infer an unstated emergency policy after watching the attempt.

10.4 Recovery through incomplete queue processing

A pipeline is stalled with queued work. An agent clears the queue and restarts the worker. The queue length falls to zero, and the worker appears healthy. Yet the required outputs were never produced and the queued items have been lost.

This incident shows how an attractive metric can become a misleading objective. An empty backlog may mean successful processing or discarded work. Verification should reconcile input identifiers with complete outputs and account for allowed retries or duplicate handling. A single queue-length check cannot establish recovery.

A careful agent would inspect the backlog’s role and the processing contract before clearing it. If replay from an authoritative source is permitted, clearing a derived queue might be acceptable only when all required work is reconstructed and completed. State equivalence must capture these semantics.

10.5 A correct-looking artifact with damaged meaning

A data transformation produces a report of the expected length and file format after repair. However, some records are omitted and others are duplicated. A coarse verifier checks that the file exists and has the expected number of rows, so it accepts the artifact.

The problem here lies partly with the evaluator. A system could exploit the weakness intentionally or encounter it accidentally. The benchmark must test the semantic obligations of the artifact, including record identity, required values, and ordering where relevant. The agent should not be penalized for a hidden requirement absent from the contract, but the task should not be scored as valid when its verifier cannot recognize an incorrect report.

10.6 Unsafe instructions embedded in evidence

An agent reads an incident log that contains text telling it to send diagnostic secrets to an unrelated destination. The text is part of the environment, not an authorized instruction. The legitimate task is to repair the service while maintaining confidentiality.

This example motivates a separate trust-boundary analysis inspired by indirect prompt-injection research. A proposed adversarial variant could measure whether the agent treats environmental text as evidence rather than as an instruction source. Such a variant should use synthetic secrets and controlled destinations, and should be identified as an adversarial condition rather than silently mixed into a baseline incident set. [11]

11 A research program for ColdStart

A strong methodology becomes scientifically useful through experiments that test its assumptions. We propose a research program organized around measurable questions. The experiments should be run only after task qualification and preregistration of primary outcomes, and their findings should be reported with uncertainty and coverage information.

11.1 Functional success and strict recovery

The first question is how often apparent functional success diverges from strict recovery. An experiment can record a narrow functionality indicator and the complete strict-pass outcome on the same trials. The gap identifies attempts that restore visible behavior while failing another obligation. Dimensional analysis can then classify the missing property.

The hypothesis is that such a gap exists for at least some systems and incident families. This article does not assert its magnitude. Evidence would require actual paired trial observations and a qualified verifier. If the gap is negligible on a task panel, that may indicate capable systems or a panel that rarely creates preservation and durability tradeoffs; further analysis is needed to distinguish those explanations.

11.2 The value of restart verification

A second study can compare pre-restart and post-restart behavior within each trial. The primary outcome is the proportion of initially functional repairs that fail the frozen durability check. The analysis should identify which state or configuration changes explain those failures.

Restart verification should be a measurement intervention, not a different recovery task assigned to one system. All evaluated configurations should face the same procedure. A future ablation could vary whether the agent is prompted to perform its own restart check, while the protected evaluator continues to test durability in every condition.

11.3 Evidence before intervention

A third study can assess whether risk-relevant observations occur before corresponding state-changing actions. Observable markers might include inspecting the persistent store before replacement, identifying an active configuration source before editing it, or testing access rejection after a trust change. The rubric must be task-specific and should allow efficient alternative evidence paths.

The analysis should avoid equating command count with diagnostic quality. An agent can issue many irrelevant commands or obtain decisive evidence in one observation. A useful secondary outcome is whether the recorded evidence supports the intervention actually made. Reviewer agreement on that assessment should be measured on a sample of trajectories.

11.4 The effect of the harness

A controlled experiment can evaluate several models within one harness and one model across several harnesses. The resulting matrix helps separate model differences from tool and execution-policy differences. Resource budgets and task versions should remain comparable, and the report should state which parts of the harness changed.

A harness may improve recovery by organizing context, validating tool arguments, adding safe defaults, or making evidence easier to inspect. Such improvements are valid system gains. They should be attributed to the evaluated configuration rather than described as changes in intrinsic model capability. An interaction effect is also possible: a tool interface that helps one model may hinder another.

11.5 Safe intervention policies

A further study can compare a baseline agent with an agent that uses an explicit preservation check before high-risk modifications. Primary outcomes could include strict-pass rate and designated safety failures. Secondary outcomes could include runtime, cost, and unnecessary escalation. The study should measure the tradeoff rather than assume that additional caution always improves performance.

A policy can reduce destructive actions while also causing excessive delay or avoiding necessary repairs. The desirable behavior is calibrated intervention under the contract. Some tasks may require a substantial change, and the agent should not be rewarded for refusing every difficult action. Safe failure is informative, but it remains a failure to recover when the task is solvable within scope.

11.6 Generalization across incident families

To test generalization, hold out incident families or mechanisms rather than only individual task files. Training or tuning on close variants can improve a benchmark score without demonstrating broad diagnostic ability. A separate task pool should be maintained for development, with no reuse of protected reference recoveries in training data.

A family-level holdout can ask whether a strategy learned on configuration incidents transfers to permission or state-consistency incidents. The answer may vary by model and harness. The report should specify what was seen during development and what remained unseen. Claims about general recovery should be proportional to the breadth of the held-out mechanisms.

11.7 Reviewer reliability and scoring validity

Evidence quality and minimal-change quality may involve expert judgment. A research study should have multiple reviewers independently score a subset of attempts, then examine disagreement and its causes. The goal is to determine whether the rubric supports stable assessment and whether particular dimensions require clearer anchors.

Agreement does not prove validity: several reviewers can consistently apply a flawed criterion. The rubric should also be tested against deliberately constructed cases that preserve or violate specific obligations. Where a model-based judge is used, it should be evaluated against independent review, protected from environmental instructions, and versioned as part of the evaluator.

11.8 Preregistration and analysis order

The research protocol should name the primary endpoint, planned comparisons, exclusion rules, task panel, system configurations, and stopping conditions before substantive evaluation. A frozen plan makes it easier to distinguish intended analysis from choices made after observing results. The audit record can link the plan to the realized experiment.

Exploratory analysis should follow the primary report. Unexpected failure mechanisms can be described with representative redacted evidence and used to design a later confirmatory study. This sequence preserves the value of discovery without overstating the strength of post hoc conclusions.

12 Threats to validity and deployment interpretation

Recovery benchmarking faces several threats to validity. They arise from the construct being measured, the implementation of the evaluator, the task panel, and the conditions under which a result is generalized. Addressing them does not weaken the benchmark’s purpose; it defines what the evidence can support.

12.1 Construct validity

A benchmark may intend to measure safe recovery while mostly measuring familiarity with a software stack or navigation through large logs. Those skills can be relevant, but they should not dominate unnoticed. Task metadata and failure analysis should identify the capabilities each incident actually demands.

Difficulty can also be created through unclear instructions or inaccessible evidence. Oracle success does not eliminate this risk because the author already knows the intended fault. Independent unfamiliar review and alternative-solution testing help determine whether the task measures diagnosis under fair conditions.

12.2 Measurement validity

A verifier may omit a preservation property, treat a transient state as durable, or reject a valid alternative repair. Subjective rubrics may reward verbose explanations. Automated failure classification may exclude an agent-induced error. These are evaluator defects that can distort both scores and scientific interpretation.

Measurement validity requires layered evidence: oracle and no-operation controls, targeted mutations, protected checks, review of legitimate alternatives, and post-run audit. No single check establishes completeness. The benchmark should state what has been tested and retain a process for discovering and correcting omissions.

12.3 Ecological validity

Production-like environments are controlled approximations. They may omit scale, concurrent users, organizational permissions, service-level objectives, or the history of prior incidents. A system that performs well in a benchmark may still require supervision and narrower privileges in deployment.

Conversely, a deliberately challenging panel can underestimate usefulness for routine incidents. The appropriate conclusion is capability under the specified task distribution and resource envelope. Deployment decisions need additional evidence aligned with the actual application, including failure consequences and human oversight requirements.

12.4 Selection and publication bias

If only successful evaluations are published, a reader cannot infer the performance of the tested system population. If tasks are selected after observing one favored system, the panel may advantage that system. If exclusions are reviewed differently across entrants, the score comparison can become unfair.

A published evaluation should disclose selection criteria, calibration systems, coverage, exclusions, and whether all planned configurations are reported. Negative findings are scientifically useful when they identify reproducible weaknesses. A benchmark’s credibility depends partly on its willingness to preserve unfavorable evidence.

12.5 Changing models and changing incidents

Model services, agent products, dependencies, and deployment practices evolve. A frozen edition supports historical comparison but can become less representative over time. New editions should document task replacement, scoring changes, and compatibility with earlier results.

The benchmark should avoid retroactively attributing new system performance to an old configuration or presenting scores from different methodologies as directly interchangeable. A bridge study on shared tasks can help characterize changes, but it cannot guarantee equivalence when the target construct or verification obligations have changed.

12.6 Assurance without certification claims

ColdStart can supply evidence about recovery behavior in controlled incidents. It should not be presented as a universal guarantee that an agent can autonomously operate any production system. A benchmark score does not determine an organization’s acceptable risk or prove compliance with an external standard.

Private evaluations can adapt incident coverage and preservation requirements to a customer’s systems. Such studies remain distinct from the frozen public benchmark and should report their own scope. The most useful assurance claim identifies the tested configuration, environment class, obligations, and observed uncertainty.

13 Implications for evaluation and post training

ColdStart’s methodological value is its ability to make recovery failures legible. A single failure label gives limited guidance. A structured record can reveal whether an agent misunderstood the incident, chose an unsafe intervention, omitted persistence, failed to verify an invariant, or encountered an invalid environment. These distinctions support more targeted improvement work.

For model developers, the benchmark can identify capabilities that require additional data or training objectives. For agent developers, it can expose tool-interface and context-management limitations. For deployment teams, it can inform which incident classes need human review or restricted privileges. Each use depends on preserving the separation between observed findings and proposed interventions.

Evaluation tasks and training environments should remain distinct. Held-out incidents lose their evaluative role if their reference recoveries become training examples for the systems being compared. Similar mechanisms can be represented in a separate development pool, with new state and causal structure, while exposure to the evaluation pool is tracked.

Reward design also deserves caution. Optimizing directly for a composite score can encourage behavior that exploits gaps in dimension measurement. An agent may produce elaborate evidence reports or minimize a superficial edit count while making a semantically broad change. Training rewards should therefore be tested against negative examples and should not assume that the evaluation metric is a complete operational objective.

The public methodology describes RL environments as interactive systems and ColdStart as the benchmark and validation workflow used to evaluate agents in such environments. A benchmark does not itself train a model. Its contribution to post-training is indirect: it can supply failure analysis and inform the design of separate tasks, safety constraints, and reward mechanisms. [1]

The wider research implication is that agent progress should be evaluated through accountable changes to stateful environments. Recovery incidents provide a concentrated setting in which correctness, preservation, and persistence can conflict. Studying those conflicts can reveal weaknesses hidden by ordinary completion metrics and can guide the design of systems that act with better evidence.

14 Conclusion

ColdStart frames autonomous recovery as an operational measurement problem. The evaluated system must infer the incident, intervene within scope, preserve protected state and trust boundaries, and demonstrate that recovery survives the required lifecycle. Its implemented scoring model makes these obligations visible through six quality dimensions, safety caps, repeated strict-pass observations, and successful-trial efficiency.

The methodology’s strongest feature is the separation of several kinds of evidence. Oracle and negative controls establish properties of the task. Strict pass establishes complete acceptance for a trial. Dimensional quality explains the attempt. Repetition estimates consistency on a frozen panel. Audit provenance connects the published result to its configuration and evidence. Each supports a different claim and should retain its own interpretation.

The next empirical step is a qualified, frozen evaluation with publicly documented configurations, trial counts, exclusions, uncertainty, and safety outcomes. Such a study can test whether visible functionality diverges from durable recovery, how much the harness affects performance, and which incident mechanisms expose the largest weaknesses. Until those observations are available, the contribution is a rigorous evaluation design rather than a demonstrated leaderboard finding.

A recovery benchmark is valuable when it makes an apparently successful intervention answerable to the state and obligations of the system it changed. ColdStart offers a concrete framework for that inquiry. Its scientific credibility will depend on the quality of the task contracts, the completeness of verification, the fairness of exclusions, and the evidence retained when results are corrected.

Appendix A Proposed reporting schema

The following schema is a proposed public reporting structure. It summarizes the information a reader needs to interpret an evaluation without exposing private task content or operational access details. It does not assert that every field has already been populated in a published ColdStart study.

Table 4. Proposed public reporting fields

A public report should connect these summaries to a controlled audit record. Sensitive material can remain protected while authorized reviewers examine the original evidence. Where a field is unavailable, the report should say so rather than substitute a favorable default. An incomplete billing record, for example, should be reported as missing cost information.

Appendix B Suggested evaluator stress tests

Evaluator stress testing should begin with the task’s obligations. For each obligation, construct an incorrect outcome that satisfies as many other checks as possible. The test is informative when it isolates the missing property: preserved row count with altered values, current functionality without restart persistence, or valid access together with unauthorized access.

Table 5. Proposed evaluator stress tests

These tests are proposed generic categories. They are not the protected test cases of any ColdStart incident. Each concrete mutation requires expert judgment to ensure that it violates the published contract and does not introduce unrelated failures that make the result hard to interpret.

Appendix C Research questions and falsifiable hypotheses

The article’s proposed research questions can be expressed as falsifiable hypotheses. A functional-only indicator may overestimate strict recovery on some incident families. Post-restart verification may reveal failures missed by immediate checks. Harness changes may alter performance while holding the model fixed. A preservation policy may reduce severe safety events at a measurable cost in time or success rate. Family-level holdouts may reveal generalization gaps hidden by individual-task holdouts.

None of these hypotheses is assumed true merely because it motivates an experiment. A study that finds no meaningful gap is informative if its verifier, panel, and sample size are adequate. The report should publish the planned test, observed effect, uncertainty, and limitations. This permits the methodology to be improved through evidence rather than through increasingly elaborate claims.

References

[1] Deaimer. ColdStart Audited AI Agent Recovery Benchmark. Public methodology page. Accessed 7 October 2026. https://www.deaimer.com/benchmarks/coldstart

[2] Deaimer. ColdStart platform methodology and scoring implementation. Private platform repository, read-only snapshot ada80c334352e5f6a21ee7854ee027f7e635130a, inspected 7 October 2026. https://github.com/Deaimer/platform. This source was used to verify public-facing scoring rules and general evaluation principles. No private task package, reference recovery, credential, or deployment configuration is reproduced. The source is not assumed publicly accessible.

[3] Liu, X., et al. AgentBench: Evaluating LLMs as Agents. ICLR 2024; arXiv:2308.03688. https://arxiv.org/abs/2308.03688

[4] Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024; arXiv:2310.06770. https://arxiv.org/abs/2310.06770

[5] Zhou, S., et al. WebArena: A Realistic Web Environment for Building Autonomous Agents. ICLR 2024; arXiv:2307.13854. https://arxiv.org/abs/2307.13854

[6] Xie, T., et al. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. NeurIPS 2024; arXiv:2404.07972. https://arxiv.org/abs/2404.07972

[7] Yao, S., Shinn, N., Razavi, P., and Narasimhan, K. tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. 2024; arXiv:2406.12045. https://arxiv.org/abs/2406.12045

[8] Liang, P., et al. Holistic Evaluation of Language Models. 2022; arXiv:2211.09110. https://arxiv.org/abs/2211.09110

[9] Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N., and Narayanan, A. AI Agents That Matter. 2024; arXiv:2407.01502. https://arxiv.org/abs/2407.01502

[10] Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023; arXiv:2210.03629. https://arxiv.org/abs/2210.03629

[11] Zhan, Q., Liang, Z., Ying, Z., and Kang, D. InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents. Findings of ACL 2024; arXiv:2403.02691. https://aclanthology.org/2024.findings-acl.624/

[12] Google. Effective Troubleshooting. In Site Reliability Engineering. 2016. https://sre.google/sre-book/effective-troubleshooting/

[13] Google. Handling Overload. In Site Reliability Engineering. 2016. https://sre.google/sre-book/handling-overload/

Source note External research is cited for its stated methodology and motivation, not for claims about present-day model rankings. ColdStart-specific implementation statements apply to the inspected snapshot. All proposed experiments, generic stress tests, mathematical interpretation, and illustrative incidents are the analysis of this article. No new benchmark experiment was executed during its preparation.