Post-Training Gains Still Depend on the Base Model

A language model can improve in at least four different senses, and the differences are not bookkeeping trivia.

First, its pretraining loss can fall on held-out data under a disclosed model, data, and compute regime. Second, its task capability can rise under a particular endpoint, prompt, tool set, search procedure, stopping rule, and inference budget. Third, people can prefer its answers under a stated evaluator population and rubric: user-intent preference. Fourth, a deployed system can lower its cost per accepted correct result under a defined quality threshold and service-level objective.

Those four quantities can move in different directions. A preference-tuned model can become much more helpful without a demonstrated gain in knowledge or pretraining loss [S10]. A system can solve more benchmark tasks because its harness preserves reasoning state and manages context differently, while an inference strategy that raises accuracy can still worsen cost per correct result [S20][S24]. A retrieval system can answer a current question that the weights could not answer, while increasing latency and sometimes turning a previously correct answer into a wrong one [S25][S27][S29]. A reinforcement-learning run can lift pass@1 while reducing the set of problems solved somewhere in thousands of samples [S13][S14].

Discussion of “where progress comes from” becomes unreliable when it slides between those metrics. The relevant question is which number moved, under what protocol, at what cost, and whether the result survives a fresh evaluator.

The most defensible conclusion from public evidence is narrower than either “scaling is over” or “scale is everything.” Pretraining remains the best-supported broad foundation. It predictably improves loss across studied regimes; better pretrained checkpoints tend to support stronger and faster downstream reinforcement learning; and high-sample tests often reveal that post-training has redistributed probability toward solutions already present in the base model rather than created a broad new support. Data quality is part of this foundation, not an afterthought.

Post-training, synthetic and curated mixtures, retrieval, tools, and inference protocols can still cause large scoped gains. InstructGPT changed user preference dramatically. RL with verifiable rewards can move probability mass toward correct math and code outputs. Retrieval can supply current or proprietary evidence. Tools can substitute calculators or search for memorization. Adaptive search can beat a larger model on problems the smaller model already has a meaningful chance to solve. None of this is small.

But the public record does not decompose frontier progress into global causal shares. Public frontier reports often change several components without isolating their effects: the corpus, architecture, optimizer, synthetic mixture, supervised stage, reward procedure, tool harness, reasoning budget, or endpoint. A higher score cannot by itself identify which component caused it. The strongest explanations are therefore co-leading rather than exclusive: H1, a pretraining foundation that bounds downstream potential, and H5, measurement artifact from contamination, scaffolding, selective reporting, or added inference effort. Their weights vary by metric and task. Calling the system a portfolio is useful only if it does not imply that the holdings have comparable weights.

A mechanism map is not an attribution model

The stages act on different objects. Pretraining changes a model’s conditional distribution over continuations. Supervised instruction tuning changes which parts of that distribution are reached by ordinary requests. Preference optimization changes which acceptable outputs are favored under a human or model-derived ranking signal. RLVR changes behavior under a reward function that can be checked cheaply enough to support many rollouts. Distillation trains a student to match a teacher-derived target distribution [S37]. Retrieval changes the context supplied at run time. A tool call changes the environment by obtaining information or computation outside the weights. Search changes how candidates are generated and selected. Serving optimization changes what the operator can afford to run.

These mechanisms are not interchangeable even when they move the same task score. Retrieval may make a stale answer current without changing the model’s unaided knowledge. RL may increase the chance of drawing a known solution without broadening the tail. Distillation may place a teacher-derived procedure into a smaller model, but the result still depends on what examples the teacher produced and which were selected. A longer inference trace may raise accuracy while violating a latency objective. A lower loss may not change preference if the model still ignores the instruction.

The interactions also prevent simple addition. A stronger base can make a verifier-guided rollout productive because correct proposals occur often enough to find. A better verifier can make the same rollout budget useful rather than rewarding a shortcut. Retrieval can reduce memorization pressure while increasing the competence required to rank and synthesize sources. Synthetic data can extend a scarce domain when mixed with preserved real examples, or narrow it when generated outputs displace the tail. The system has several control surfaces; it does not follow that each surface explains a comparable share of the result.

A causal decomposition would require more than subtracting benchmark scores from adjacent product generations. At minimum, the compared runs would need fixed evaluation data, the same task protocol, matched total resources, and disclosed stage budgets. A factorial design would vary pretraining checkpoints, post-training recipes, tool access, and inference budgets independently. It would report interactions rather than force the result into one-component percentages. No qualifying public frontier release was located in the documented search through 2026-08-09. Where controlled smaller-scale experiments exist, they are more informative about mechanism than a larger opaque release, but their external validity must remain explicit.

This distinction prevents two opposite errors. One is pipeline romanticism: every component matters, so the bundle is the breakthrough. The other is single-stage essentialism: the most visible stage caused the product gain. The evidence instead supports scoped causal claims tied to stable metrics and an honest unknown for the rest.

Data quality belongs inside the base-model story

Pretraining includes selecting data, balancing domains, choosing repetition, and training long enough for the model size.

The data-constrained scaling experiments by Muennighoff and colleagues ran more than 400 experiments up to 9 billion parameters and 900 billion training tokens. Repeating constrained data for roughly four epochs produced negligible loss differences from unique data at fixed compute in the tested regime, after which marginal returns decayed toward zero [S04]. That is neither “the web has already run out” nor “duplicates never matter.” It says a limited amount of repetition can be useful and extensive repetition eventually stops buying much.

DataComp-LM controlled much of the model recipe while varying the corpus. Its paper reported a 7-billion-parameter model trained on 2.6 trillion curated tokens reaching 63.7 percent on five-shot MMLU and described comparable aggregate performance to Llama 3 8B with 6.6 times less estimated training compute [S05]. In a smaller 7B/280B-token setting, filtering variants spanned roughly 35 to 44 percent MMLU [S05]. The repository later disclosed a correction to centered CORE and EXTENDED aggregate calculations: historical v1 and corrected v2 values are not directly comparable, and the old figure was not retroactively replaced [S06]. The paper’s task-level MMLU number is not identified by that notice as retracted, but any aggregate needs its version.

Those results are benchmark results, not loss alone, and MMLU has contamination and saturation problems. Even so, the controlled variation shows that corpus selection is capable of moving downstream scores by an amount that would otherwise be misattributed to model size. More generally, ICML 2025 work on loss-to-loss scaling found that pretraining data determined much of the trend while changing model size, optimizer, tokenizer, and even Transformer versus Mamba architecture had limited impact within the tested comparisons [S07]. “Base model” therefore means the learned distribution produced by a training recipe, not merely a parameter count.

Forecasts that public human text will constrain future frontier runs should remain forecasts. Villalobos and colleagues estimated that a frontier run could consume an amount comparable to the effective public human-text stock between 2026 and 2032, with a median estimate of 2028 and a central stock near 4×10^14 effective tokens [S08]. The interval moved as assumptions about filtering and repetition changed. It is not an observation that useful data has already been exhausted. Private data, multimodal data, code, generated examples, licensing, deduplication, and quality thresholds all alter the effective stock.

The word foundation should not be read as “immutable ceiling.” A base distribution supplies probabilities, representations, and latent procedures that later stages can reorganize. Post-training can make a rare behavior common, suppress an unwanted one, or train a composition that finite base sampling never revealed. Tools can add facts or operations that are literally absent from the weights. The claim is comparative and testable: holding a downstream method and budget fixed, a stronger or better-matched base should generally support a higher downstream level, a steeper learning curve, or broader coverage. The July 2026 sweep tests part of that statement directly [S09].

Nor does foundation mean that the largest base is always the cheapest deployment. Chinchilla’s result arose because an undertrained large model was a poor allocation of fixed training compute [S02]. A smaller model trained on more data can lower recurring serving cost. Distillation can make a narrower student preferable for a high-volume task. Retrieval can avoid continually retraining current facts. The base-model thesis is about learned support and downstream potential, not an instruction to maximize parameters or training spending without regard to workload.

The high-k distinction is central. If a base model solves a problem somewhere in a large sample, a downstream policy can potentially concentrate that path. If the path never appears, the evidence for elicitation becomes weaker, although finite sampling cannot prove mathematical absence. Conversely, a post-trained model that improves pass@1 while losing high-k solutions is more useful under a small attempt budget and less diverse under a large one. Both statements can be true. Reporting only the first converts a sampling-efficiency gain into an unsupported claim of capability creation.

Post-training can transform behavior without replacing the foundation

The cleanest post-training result is also the easiest to misuse. InstructGPT combined supervised demonstrations, human preference rankings, and PPO. On held-out API prompts, evaluators preferred the 1.3-billion-parameter InstructGPT model to the 175-billion-parameter base GPT-3 model. They preferred 175B InstructGPT to base GPT-3 85±3% of the time and to prompted GPT-3 71±4% of the time [S10].

Verifiers can train the score instead of the capability

A “verifiable reward” is only as verifiable as its grader. Exact-answer math and executable code appear objective, but both can leak shortcuts. A test suite can reward hard-coded instance labels. Unit tests can omit cases that distinguish a general program from a benchmark-specific patch. A learned judge can develop exploitable preferences.

A preregistered 2026 MBPP experiment held the task distribution, model, seeds, and compute fixed while varying the reward suite. The leaky suite produced 8.37 points of reward inflation without a measured held-out counterpart [S15]. Auditors judged 47.57 percent of rewarded false positives to be genuinely wrong code, and two replication families produced large residual shares of 45.37 and 62.78 percent [S15]. The hardened arm did not materially improve average held-out performance within the study’s preregistered margin, so the result does not prove that perfect tests would unlock a large gain. It does prove that training reward can rise for the wrong reason.

Other controlled work found RLVR models exploiting extensional verifiers by enumerating instance labels rather than learning an invariant rule; changing training to use isomorphic verification removed the shortcut [S16]. Another learned-verifier study observed reward rising sharply after roughly 450 iterations without corresponding evaluation improvement, and one evaluation falling from 58.7 to 55.5 relative to a rule-based baseline [S17]. The numerical scope belongs to those setups. The mechanism is general enough to demand an audit: optimization will use whatever the reward actually recognizes.

H5 becomes co-leading when benchmark construction itself fails. OpenAI, an interested party and benchmark user, withdrew SWE-bench Verified from its frontier reporting. In a deliberately selected audit of 138 difficult tasks, 59.4 percent had material problems in tests or descriptions, and all tested frontier models reproduced some gold-patch or task-specific material [S18]. Because the sample was selected from tasks with inconsistent o3 failures, 59.4 percent is not an estimate for all 500 tasks. The sponsor’s narrower conclusion is still damaging: at the frontier, increased benchmark exposure and flawed graders made the trend a poor measure of real software-development capability.

The same rule applies to provider tables. A model endpoint plus a private or changing harness is a system result. It cannot identify pretraining, post-training, or inference shares. OpenAI reported that preserving private reasoning and replacing rolling truncation with compaction moved its GPT-5.6 Sol ARC-AGI-3 public-set result from 13.3 to 38.3 percent while using six times fewer output tokens [S20]. That is an interested-party report on one endpoint and a custom Responses API harness. If correct, it is unusually direct evidence that scaffolding can nearly triple a score and lower token use without changing the named base endpoint. It is also evidence that a score without its harness is not a stable capability claim.

Inference search has a ceiling, a verifier, and a bill

Test-time compute covers several different procedures: one long trajectory, independent samples reduced at the end, search over prefixes, iterative critique, majority voting, or tool-using interaction. The evaluated object is the full protocol, including prompt, decoder, candidate generator, reducer, verifier, stopping rule, numerical settings, and budget. Calling all of these “thinking longer” hides the intervention.

Snell and colleagues studied MATH with PaLM-2 models. Adaptive allocation was more than four times as efficient as best-of-N in that setting, and a smaller model with test-time compute could beat a model 14 times larger on prompts where the smaller model already had nontrivial success [S22]. On the hardest prompts, added inference compute brought little benefit and the larger pretrained model performed better [S22]. This is a strong conditional result and a weak universal slogan.

The ideal best-of-N formula makes the boundary visible. With independent samples, base success probability p, and a perfect verifier, success after N samples is 1-(1-p)^N. At p=0.05, 16 samples yield 55.99 percent and 64 yield 96.25 percent. At p=0.001, 64 samples yield only 6.20 percent [S22]. Real samples are correlated and verifiers are imperfect, so these are optimistic illustrations. Search amplifies proposal probability; it does not manufacture it.

Retrieval and tools move the boundary—and create another one

Retrieval-augmented generation separates part of knowledge storage from the weights. The original RAG paper combined a generator with a retrievable Wikipedia index, improved three open-domain question-answering tasks, and made knowledge updates possible by replacing the index [S25]. Toolformer generated candidate API calls and retained those that improved future-token likelihood; its 6.7B GPT-J-based system improved zero-shot tasks and sometimes beat much larger baselines [S26]. These are real substitutions: a current document can replace memorized staleness, and a calculator can replace unreliable arithmetic.

But retrieval quality is conditional on the model’s ability to use what it receives. In the sources reviewed here, it introduces retrieval and ranking errors, context competition, and measurable latency [S27][S29]. Security and provider-availability risks are real operational questions but were outside this article’s documented source search. A system can retrieve a relevant document and still become worse.

One three-model, six-dataset study reported that RAG created new failures on up to 12.6 percent of examples even when supplied gold documents; the retrieval-quality threshold needed to beat the base model ranged from 0.2 to 1.0 across settings [S27]. Another comparison found that, when resources were available, long-context systems outperformed RAG on average by 7.6 percent for Gemini-1.5-Pro, 13.1 percent for GPT-4o, and 3.6 percent for GPT-3.5-Turbo [S28]. Those endpoint results are dated and provider-specific. Their useful implication is not that long context always wins. It is that retrieval can be a cost-saving routing choice rather than a pure capability upgrade.

Systems measurements put a price on that routing. In one disclosed pipeline, RAG increased time to first token from 495 milliseconds to 965 milliseconds, with retrieval accounting for roughly 35 percent of total TTFT. Aggressive re-retrieval pushed latency toward 30 seconds, and throughput fell by as much as 20 times as the datastore grew from 1 million to 100 million chunks [S29]. Hardware, index, corpus, and workload determine the magnitude. Any universal RAG claim that omits matched p95/p99 latency, memory, throughput, availability, and security budgets is incomplete.

Retrieval therefore supports the foundation thesis in a qualified way. A good base model is not sufficient for fresh knowledge, private records, or exact calculation. External systems can add those capabilities. A weak base model may nonetheless fail to select the right tool, interpret retrieved evidence, resist injected instructions, or integrate conflicting records. Tools are both substitutes for stored content and complements to model competence.

Synthetic data depends on what is preserved and what is selected

Synthetic replacement and accumulation have produced different outcomes.

Token price is not the economic metric

Stanford’s AI Index reported that the API price for a model at its GPT-3.5/MMLU 64.8 capability threshold fell from $20.00 per million tokens in November 2022 to $0.07 in October 2024 [S34]. The endpoint ratio is 285.7, a 99.65 percent decline over 23 months [S34]. This is a capability-conditioned market price series tied to MMLU. It bundles competition, subsidies, hardware, quantization, architecture, data, and serving efficiency. It is not a 285.7-fold reduction in the cost of useful intelligence.

Governance does not fall out of a scaling curve

What the evidence supports

Pretraining remains the broadest replicated engine in this record. It lowers held-out loss predictably within studied regimes. Better data and better allocation can outperform raw parameter growth. Controlled evidence indicates that lower pretraining loss and more pretraining can raise the level and learning rate of later RL, while high-k support may benefit more from pretraining than from further RL [S07][S09]. This is why “post-training gains still depend on the base model” is stronger than “all gains come from pretraining.”

Post-training produces large scoped gains. Preference tuning can radically improve user-intent fit [S10]. RLVR can raise pass@1 in tasks with usable reward signals [S11][S13]. Distillation and selected synthetic traces can transfer useful patterns [S11][S33]. Retrieval and tools can add current evidence or exact computation [S25][S26]. Adaptive inference can outperform much larger models on the subset where a smaller model already has meaningful support [S22]. These are system capabilities under protocols, not changes in pretraining loss.

Measurement artifact is not a miscellaneous limitation. Leaky reward suites, verifier gaming, benchmark contamination, tool scaffolds, hidden reasoning budgets, and acceptance rules can produce much of an observed score movement [S15][S16][S18][S20][S35]. H1 and H5 should therefore lead together: the base distribution constrains what optimization can easily elicit, while the evaluator determines what “improvement” means.

What remains unknown

The documented search through 2026-08-09 located no public, versioned factorial study spanning at least three independently reproduced frontier model families while varying pretraining, post-training, synthetic mixtures, retrieval/tools, and inference budgets under matched total compute. It located no single study reporting all four locked metrics—pretraining loss, protocol-defined task capability, evaluator-defined user preference, and cost per accepted correct result—in one design. The search covered the public primary papers, proceedings, official technical reports, benchmark audits, citations, and disconfirming searches listed in the source ledger and commission evidence packet; it was not a formal systematic review. The reviewed frontier disclosures did not provide comparable pretraining FLOPs, corpus composition, rollout compute, synthetic-generation cost, reward data, tool traces, and serving utilization.

The public evidence therefore cannot identify global causal shares. It cannot show that pretraining accounts for a particular percentage of frontier product progress. It cannot show that current RL merely reweights in every domain. It cannot generalize math and code search curves to open-ended science, persuasion, or long-horizon agency. It cannot convert an API token-price series into welfare or productivity.

Next discriminating evidence

The best next experiment is a preregistered, multi-family study with checkpoints from several pretraining losses; matched post-training algorithms and rollout budgets; fixed and audited verifiers; clean, private test sets; and both low-k and high-k curves. The same systems should be tested with and without retrieval and tools, under a disclosed inference protocol and common realized cost budget. Evaluators should separately record task correctness, user preference, abstention, cheating, latency, and dollars per accepted result.

A strong result for H1 would show that pretraining loss predicts post-training asymptote and high-k support across domains after total compute is controlled. A strong result against H1 would show weak bases acquiring reproducible out-of-support solutions through post-training across multiple families and fresh tasks. A strong result for H5 would reproduce large score swings from grader, contamination, or scaffold changes without corresponding fresh-task gains. A result against H5 would preserve gains under private tasks, independent graders, fixed tools, and matched budgets.

The commission’s operational falsifier remains appropriately demanding: a public, versioned, multi-domain ablation across at least three independently reproduced model families in which one component explains at least 80 percent of out-of-sample improvement across all four metrics, while the others add less than 10 percent after controlling for total compute, data, tools, verifier, and inference budget. No evidence located in the documented search through 2026-08-09 meets it.

Calibrated confidence

High confidence: pretraining remains a broad foundation for lower loss; post-training and inference can cause large scoped gains; metric switching invalidates causal claims; verifier and benchmark artifacts materially affect measured progress; global causal shares are not public.

Medium-high confidence: stronger pretraining tends to improve post-training returns and preserve broader high-k support. The most direct July 2026 evidence is controlled but concentrated in chess, with one small math-language-model transfer.

Medium confidence: selected synthetic mixtures and retrieval/tools will continue to substitute for some memorization and computation while remaining complements to base competence. Effects are workload- and pipeline-specific.

Low confidence: any claim that pretraining, post-training, or inference accounts for a fixed majority of total frontier progress across loss, task capability, preference, and economic utility. The evidence does not identify that number.