When Inference Compute Helps: Search, Verification, and the Cost of Better Answers

More computation after a model receives a question can buy a longer continuation from one model call, independent candidates, or a search controller that expands promising prefixes. It can also buy another pass over the same draft, an executable test, a learned score, a tool call, or enough time to notice that the process should stop. Those interventions spend a resource in different ways. They do not have the same failure modes, and they do not estimate the same capability.

That distinction is the core of the answer. At fixed weights, additional inference compute can improve the answer that a system accepts on some moderately difficult, well-specified tasks. The packet establishes that existential claim for some protocols. H4 makes a separate effect-modifier claim: the size and direction of the gain vary with independently measurable base support, task decomposability, verifier quality, and stopping policy. The effect belongs to the complete inference protocol, not to visible reasoning length. It commonly diminishes, can reverse, and is not established as a domain-general or generally cost-effective exchange.

This is H4, the domain-limited interaction account. It is more precise than saying either that test-time scaling works or that it is a mirage. H2 survives only in narrowed form: checkable domains often make search easier to exploit, but the evidence does not measure what fraction of all gains comes from cheap verification. H3 survives as a boundary: training and base-policy support matter, but the packet does not show that training creates a quantified majority of capability. H5 identifies real measurement artifacts without making every gain illusory. H6 identifies use-case-specific economic ceilings without proving a universal ceiling.

H1 is unsupported as an affirmative thesis and formally unresolved. The registered H1 asks whether adaptive inference compute moves the quality-compute frontier across materially different task families at fixed weights. Its frozen rejection wording requires a public, versioned comparison to cover at least mathematics, coding or software engineering, and a domain without a cheap deterministic verifier; multiply total inference work by four; measure whether accepted-answer error falls by at least 20 percent relative in every family; include uncertainty intervals excluding no improvement; and hold candidate count, prompt, tools, verifier or reducer, stopping rule, and evaluation harness constant. No study in the packet executes that design. The absence of qualifying positive evidence defeats a broad affirmative article thesis. The absence of the registered rejection experiment prevents a claim of formal falsification.

The frozen uncertainty clause is also asymmetric and must not be treated as a complete no-effect rule. A precise null estimate of 0 percent relative error reduction with a 95 percent interval from -2 percent to +2 percent includes no improvement, so it cannot satisfy the literal trigger. A real but subthreshold 10 percent reduction with an interval from 5 percent to 15 percent excludes no improvement and can satisfy that clause even though it misses the required 20 percent reduction. Commission 002's trigger has not been repaired or satisfied. A future prospective rule should instead test whether the upper interval bound excludes the required 20 percent reduction, with direction and multiplicity fixed before outcomes are seen.

The narrower frozen H2 falsifier is separate: “comparable gains” must survive in open-ended domains without privileged graders after candidate count and total work are matched. That wording is under-specified, so no future result should be classified under an improvised margin. A prospective H2 rule should define a gain-ratio or noninferiority margin; name more than one independently maintained, genuinely open-ended family; preregister blinded-human aggregation and inter-rater reliability thresholds; specify uncertainty; report acceptance, coverage, and correctness conditional on acceptance; and match generation, human evaluation, tool, retry, and deadline work. No packet study executes the frozen test, and evidence from exact-answer mathematics cannot be silently promoted into evidence about essays, strategy, factual synthesis, or scientific judgment.

The object is a protocol, not a token count

A useful inference experiment starts by naming the whole system.

The checkpoint fixes the learned parameters. That sounds sufficient for “fixed weights,” but it is only the first control. The controller decides whether to continue one trajectory, sample independent trajectories, revise a draft, branch over prefixes, call tools, or allocate different budgets by estimated difficulty. The tool policy controls access to code execution, retrieval, browsers, calculators, files, and external environments. The sample policy fixes decoding, seeds, candidate count, dependence between candidates, and maximum lengths.

The reducer or verifier decides what becomes the accepted answer. It might use plurality consensus, a learned reward model, a process reward model, unit tests, a formal checker, a human rubric, or no reduction at all. The stopping rule decides when marginal work is unlikely to help. The retry policy decides whether malformed outputs, refusals, tool failures, and transient infrastructure failures receive another attempt. The harness fixes prompts, context management, parser behavior, repository state, container image, tool interface, and grading code. Timeout and nonreturn handling decides whether an incomplete call counts as wrong, disappears from the denominator, or receives a retry. The resource boundary decides whether “compute” means visible output tokens, all billed tokens, accelerator time, wall time, energy, verifier calls, tool costs, or some combination.

A result that changes any of these is a system comparison. That can still be operationally valuable. It simply is not evidence that one checkpoint became more capable because it emitted a longer trace.

The output metric must also be named. Accepted-answer accuracy is correctness after the declared operational selector. It is the quantity a deployed system can deliver. Oracle pass@k coverage asks whether any candidate among k is correct, as if a perfect but unavailable selector identifies it. Consensus selects a plurality or majority answer and can fail when errors are correlated. Learned reranking uses another trained model or scoring head and therefore measures a generator-reranker pair. Tool-enabled performance measures the checkpoint plus its tool loop. All-call performance counts timeouts and nonreturns in the requested population. Returned-only performance conditions on successful returns and becomes biased when difficult calls are more likely to fail.

Visible reasoning length is only a resource measurement. Even when those tokens reveal part of the computation, more of them do not prove that more useful reasoning occurred. Hidden state, tool results, branch diversity, verification, and selection can dominate the observed answer. A long failed loop can consume more tokens than a short correct trajectory.

Why more can become worse

The optimistic mental model is N independent chances, each with success probability p, followed by a perfect selector. Oracle coverage is then:

1 - (1 - p)^N

At p = 0.05, sixteen independent candidates contain a correct answer with probability 1 - 0.95^16 = 55.99%; sixty-four contain one with probability 1 - 0.95^64 = 96.25%. At p = 0.001, sixty-four candidates reach only 1 - 0.999^64 = 6.20%. Real candidates are correlated, and real selectors are imperfect. The formula is therefore an optimistic mechanism illustration, not a benchmark result. It clarifies two boundaries: search needs nontrivial proposal probability, and coverage needs conversion into selection.

Sequential work has a different structure. Some problems require dependent state: an intermediate computation changes what should happen next. A longer trajectory may therefore be useful. But forcing a trajectory to continue after it has already found a defensible answer introduces new opportunities to revise a correct conclusion into an incorrect one, loop, lose context, or optimize a proxy. A controller that cannot stop is not simply a more patient controller.

Training creates the policy that inference executes

“Inference scaling” sometimes combines a stronger trained checkpoint with more test-time effort. That comparison cannot isolate the inference effect. Training can change the probability of correct proposals, teach when to backtrack, teach tool use, or make long trajectories coherent. Inference then executes the acquired policy.

Selection is often the bottleneck

An oracle curve can rise while operational accuracy stays flat. Suppose 100 samples contain at least one correct answer 90 percent of the time, but the reducer selects it only half the time when it exists. Accepted accuracy is then at most 45 percent, before accounting for false acceptance when no correct candidate exists. Generating candidate 101 helps only if it changes coverage or makes selection easier. If it adds another persuasive error, it can hurt.

Consensus is a specific reducer. It works when correct solutions converge and errors are sufficiently dispersed. Correlated mistakes can dominate a vote. Learned reranking is another reducer. It can recognize patterns unavailable to a simple vote, but it imports a second trained component, a calibration problem, and an optimization target. Rule-based verification is strongest when the rule matches the intended property. Unit tests, exact-string checks, and final-answer equivalence can all be incomplete.

Benchmark numbers are protocol records

A benchmark score is not self-describing. Denominator, sample count, reducer, tool access, training exposure, timeout policy, and model version determine what it estimates.

The economic denominator is accepted correct work

The economically relevant question is not whether score rises with effort. It is whether the system reaches a required quality threshold at an acceptable total cost and service level.

For a disclosed workload, define:

cost per accepted correct result
= total generation + verification + tool + retry + failed-candidate cost
  / number of accepted correct results

The numerator should include the cost of the entire requested workload, not only visible or successful answers: every candidate, hidden billed reasoning, verifier call, tool charge, failed request, retry, and in-boundary infrastructure cost belongs there. The denominator remains the number of accepted-correct outcomes. For example, if 100 requests cost $100 in total and yield 20 accepted-correct results, the cost is $100 / 20 = $5 per accepted correct result. The accepted-correct yield is separately 20 / 100 = 20%; replacing the denominator with 100 would instead report the different metric of $1 per request. Reports should also state request count, acceptance or coverage, all-call accuracy, correctness conditional on acceptance, and the treatment of abstentions, invalid outputs, timeouts, and nonreturns. The same quality threshold should carry p50 and p95 latency, throughput, and incomplete rate. A cheaper token can produce a more expensive result if the protocol uses enough tokens or retries. A costly attempt can be economical if it crosses a valuable acceptance threshold with sufficiently high reliability.

135.22 Wh / 500 = 0.27044 Wh/query
424.54 Wh / 500 = 0.84908 Wh/query
(1 - 0.27044 / 0.84908) × 100 = 68.15% lower
83.2 - 45.7 = 37.5 percentage points

Strongest case for a broader thesis

Empirical conclusion

The evidence supports five empirical distinctions.

First, some inference protocols improve accepted-answer accuracy at fixed weights. Candidate generation, process verification, adaptive allocation, and stopping are mechanisms with positive bounded evidence [S03][S04][S05][S09].

Second, the effect is non-monotonic. Forced-long trajectories can flip correct answers, and shorter or parallel allocations can dominate at matched resource levels in named setups [S06][S07][S08]. This makes visible reasoning length a resource quantity, not a quality metric.

Third, training and inference interact. Released R1 is a hybrid trained policy, not pure inference scaling, and version-specific benchmark values must stay separated [S01][S02]. Search is more useful when the checkpoint assigns enough probability to useful candidates.

Fourth, selection and measurement can dominate interpretation. Oracle pass@k is not accepted-answer accuracy. Consensus, learned reranking, tool-enabled scores, returned-only scores, and all-call scores describe different protocols. AIME’s 30-question denominator, ARC’s sample shift and nonreturns, and SWE-bench’s targeted audit show how easily a headline changes estimand [S12][S13][S14][S15].

Fifth, quality movement is not an economic result. ARC gives one severe diminishing-return case, while the A100/NVML energy study gives a bounded accuracy-energy win for a small reasoning configuration against a larger base [S13][S16]. Full-stack cost per accepted correct result remains unknown.

These findings make H4 the strongest synthesis: inference scaling can be real, yet its effect may be modified by base support, decomposability, verifier quality, and stopping. H2, H3, H5, and H6 explain constraints and mechanisms. None supplies a measured global causal share.

For a prospective H4 test, those modifiers must be fixed without using the main-study gain. Base support is the checkpoint's preregistered one-sample or small-fixed-budget probability of producing a valid candidate on a held-out calibration set. Decomposability is a blinded, pre-outcome task annotation under a published rubric for whether independently checkable intermediate subgoals can be specified. Verifier quality is held-out precision, recall, calibration, and adversarial false-acceptance performance measured before the treatment comparison. Stopping is a versioned policy fixed before the study, including its observable state, thresholds, maximum budget, timeout action, and abstention rule; policy classes can be randomized as a factor without relabeling a failed arm after results.

H4's modifier claim is prospectively falsifiable even if the existential claim survives. A preregistered factorial study should cross low and high inference work with predeclared levels of those four modifiers, at multiple budget levels, model checkpoints, and independently maintained mathematics, software-engineering, and open-ended families. It should reject H4 if accepted-answer error shows a stable common work effect across checkpoints and families while every preregistered work-by-modifier interaction is smaller than a materiality margin fixed before data collection, with simultaneous uncertainty bounds excluding that margin. Reviewers should apply that rule from the registered measurements, not invent a new condition after seeing gains.

Post-research stress tests, not preregistered tests

Seven mechanism dimensions emerged after the evidence review. They are post-research stress tests, not preregistered hypotheses or falsifiers:

1. Training-acquired policy versus inference execution. Freeze the checkpoint and distinguish what was learned before deployment from what the controller does after the prompt.
2. Parallel or prefix search versus a forced-long monologue. Match total work and compare independent candidates, branch search, and sequential continuation.
3. Sequential state for genuinely dependent problems. Identify tasks where later steps require earlier results rather than treating every extra token as interchangeable.
4. Verifier or reducer bottlenecks. Measure oracle coverage and operational selection separately, then stress the selector under optimization.
5. Tool-grounded work versus unaided token deliberation. Report the tool policy, calls, outputs, failure handling, and cost.
6. Adaptive allocation and causal stopping. Compare uniform budgets with policies that stop resolved items and concentrate work where marginal value remains positive.
7. Benchmark and protocol artifacts. Freeze versions, exposure, prompts, scaffolds, resource limits, parsers, timeouts, and nonreturn treatment.

Eight checks that could reject or narrow a claim were also developed after research. They too are post-research stress tests, not the locked H1 or H2 falsifiers:

1. Monotonicity: estimate slopes and count correct-to-incorrect flips across the intended operating range.
2. Efficiency: compare shorter, parallel, and adaptive protocols at matched quality, compute, and p95 latency.
3. Protocol: match sample count, verifier calls, prompt, tools, scaffold, timeout, and hardware.
4. Freshness: compare exposed public tasks with private post-cutoff or generated tasks.
5. Process validity: replay steps or apply formal, invariance, and counterfactual checks instead of trusting final-answer reward alone.
6. Open-ended transfer: use blinded humans, length normalization, judge swapping, and order randomization.
7. Reliability: report selected pass@1 and calibration beside oracle pass@k, and report the probability that all required attempts succeed.
8. Production: predefine cost-per-correct, incomplete rate, throughput, and tail-latency limits.

These checks can narrow a claim without satisfying the more demanding registered rejection triggers. A future experiment can preregister them for that experiment. They cannot be backdated into Commission 002.

Anarcho-capitalist prior

Libertarian prior

Accelerationist prior

What would change the conclusion

Confidence is medium in H4 as the organizing synthesis. Confidence is high that the packet’s named metrics cannot be merged: pass@k coverage, consensus, learned reranking, accepted-answer accuracy, tool-enabled results, all-call results, and returned-only results are operationally different. Confidence is high that visible reasoning length alone is not a quality measure. Confidence is medium that the strongest transferable gains will remain concentrated in moderately difficult, decomposable tasks with useful feedback. The open-ended and full-cost records are too incomplete for higher confidence.

Three results would materially raise confidence in H1. First, a public, versioned fixed-weight study should execute the literal locked fields: mathematics, software engineering, and a family without a cheap deterministic verifier; four times total inference work; accepted-answer error with uncertainty; and controls for candidate count, prompt, tools, verifier or reducer, stopping rule, and evaluation harness. The frozen trigger also names at least a 20 percent relative error reduction in every family and its asymmetric “intervals excluding no improvement” clause; as explained above, that clause remains frozen but is not a statistically complete rule. Retries, timeout handling, and resource-boundary disclosure are additional post-research prospective requirements, not locked Commission 002 fields. Second, the open-ended arm should use blinded independent humans rather than a privileged learned grader and should preserve gains that clear a prospectively defined margin after candidate count and total work are matched. Third, an independent replication should reproduce the movement on fresh post-cutoff tasks.

Three results would lower confidence in useful bounded inference scaling. A matched-work comparison could show that positive accepted-answer movement disappears after oracle coverage is separated from actual selection. A verifier audit could show that apparent gains are mostly proxy exploitation and false acceptance under optimization. Or production measurement could show that failures, nonreturns, p95 latency, verification, tools, and retries make cost per accepted correct result worse than a lower-effort protocol at the same quality threshold across the intended use cases.

The economic conclusion would change with an audited dataset that jointly reports all-call accepted accuracy, candidate and verifier work, p50 and p95 latency, throughput, incomplete rate, accelerator and facility energy, and fully allocated dollars for matched tasks. The policy conclusions would change only with both empirical evidence and explicit normative premises about property, consent, compensation, coercion, concentration, and tail risk. A higher benchmark score by itself supplies neither.

The practical rule is consequently simple but demanding: evaluate the complete inference protocol. Freeze the checkpoint. Name the controller, tools, samples, reducer, stopping rule, retries, harness, timeout treatment, and resource boundary. Measure the answer the system actually accepts. Count every call. Then ask whether the additional work improved quality, whether a shorter or parallel protocol did better, and whether the improvement was worth its full cost. Without those controls, the evaluation risks treating added activity as useful reasoning.