1B tokens, picked by Fisher geometry, beat 10B tokens of replay on both adaptation and forgetting

What changed, and when

On 1 October 2026, Zhenghao Zhao, Gaowen Liu, Zhiling Lan and Yan Yan posted Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Models (arXiv:2610.02593), from the University of Illinois Chicago and Cisco Research. The test case is continual pretraining on a medical corpus (PMC full-text articles and PubMed abstracts) starting from TinyLlama-1.1B, with a scale-up check on Llama-3.1-8B.

Continual pretraining has a standard failure mode: adapting to a new domain degrades performance on everything the base model already knew, a problem usually addressed by replaying some fraction of general-domain text alongside the new domain's data, or by regularizing weight updates directly (EWC). Both approaches spend part of the training budget on something other than the target domain. This paper's claim is that selecting which target-domain documents to train on, using a measure of how each document's gradient relates to the model's existing knowledge, removes the need for that tradeoff.

The evidence table

All methods below continue-pretrain TinyLlama-1.1B on the same medical candidate stream at a 4B-token budget, evaluated on medical perplexity (Med-PPL, lower is better), two QA benchmarks (PubMedQA, BioASQ), an adaptation-gain score (A, higher is better, combining the two QA deltas against base), and a forgetting measure (Φ, lower is better). Each number is a mean over three seeds.

Source: arXiv:2610.02593, TinyLlama-1.1B main results, 4B-token budget, mean of 3 seeds. The paper's own text states the budget as "4.4B" in one place, which conflicts with the 4B figure in the results-table caption; the table caption is used here.

The budget comparison that gives the paper its headline claim runs the Fisher-guided method and Replay across 1B, 2B, 4B and 10B tokens. Replay never clears zero adaptation gain in this range: −3.5 at 1B, −2.6 at 4B, −2.0 even at 10B, with forgetting between 0.8 and 1.2. The Fisher-guided method clears positive adaptation gain at every budget tested, reaching +3.0 at 1B (forgetting 0.6) and +5.5 at 4B (forgetting 0.4). The 1B-versus-10B comparison, +3.0/0.6 against −2.0/0.8, is what the paper frames as a 10x token-efficiency advantage: the smaller, selected set beats the larger, replayed one on both the metric it helps (adaptation) and the metric it is supposed to protect (forgetting). On a larger backbone, Llama-3.1-8B at 10B tokens, the same method reaches A = +4.4, Φ = 0.1 against Replay's A = −1.8, Φ = 0.6, though the authors caution A and Φ are not directly comparable across the two backbones.

How Fisher geometry replaces a replay mixture

The method estimates a diagonal Fisher information matrix from a reference set of general-domain documents, which approximates which parameter directions the base model's existing capability actually depends on. For each candidate target-domain document, it computes a LoRA gradient and splits it into two views against that Fisher estimate: an anchor view (F^½·g) that measures how much the document's gradient overlaps with the high-Fisher directions holding retained capability, and a frontier view (F^−½·g) that measures how much it uses the low-Fisher directions available for new content without disturbing what the model already knows.

A log-determinant objective over both views rewards a selected subset for covering new, non-redundant directions in each view, a monotone submodular objective, which makes greedy selection provably near-optimal. To make this tractable at corpus scale, selection runs in a single streaming pass (Sieve-Streaming) using LoRA gradients, a TRAK-style random projection to keep per-document vectors small, and Sherman-Morrison updates to maintain the running log-determinant without recomputing it from scratch. The practical effect is that the method never needs a replay mixture: instead of diluting the training stream with general-domain tokens to protect existing capability, it chooses target-domain documents whose gradients are already aligned with preserving it.

What it means for teams budgeting continual-pretraining data

Replay is not a safe default at low token budgets. The intuition that mixing in general-domain text protects a model from forgetting assumes the budget is large enough to absorb the dilution. At 4B tokens here, Replay's forgetting score (0.9) is actually worse than doing nothing special at all beyond the domain data (Random: 3.5 forgetting is worse still, but EWC without any replay reaches 1.4 forgetting with less adaptation loss than Replay). Replay loses adaptation ground at every budget tested, from 1B to 10B tokens.

The gain is not from picking "hard" or "easy" documents. Low-PPL and High-PPL selection, the two perplexity-based heuristics in the table, land close to Random on adaptation gain and noticeably worse than the Fisher-guided method on forgetting (High-PPL: 5.2 forgetting, the worst in the table, despite the second-best adaptation score). A document's difficulty for the current model, measured by perplexity alone, is a weak proxy for whether training on it will damage unrelated capability.

A 10x token-efficiency claim is specific enough to budget against. If a team is deciding how many tokens of a medical, legal or other specialized corpus to curate and label for continual pretraining, this paper's 1B-versus-10B comparison is a concrete number to test against their own domain before assuming a bigger replay-protected run is the safer choice.

Our judgement: this is one paper's benchmark on one small-to-mid backbone pair (TinyLlama-1.1B, Llama-3.1-8B) and one domain (medical). The authors' own text contains an unresolved discrepancy between a "4.4B" and "4B" token-budget figure for the main results table, which this article resolves in favor of the table caption; that inconsistency is a reason for caution about how carefully the budget axis was tracked elsewhere in the paper. The method has not yet been tested on a frontier-scale model or outside the medical domain.

Check it yourself

The paper reports its full budget sweep in one table. The number worth re-deriving is the 1B-versus-10B comparison, not the headline 4B row alone.

open https://arxiv.org/abs/2610.02593          # abstract, authors, submission date
open https://arxiv.org/html/2610.02593v1       # full text: budget-sweep table, Fisher-geometry method section, Llama-3.1-8B scale-up

python3 - <<'PY'
# recompute the 1B-vs-10B token-efficiency comparison from the paper's reported numbers
ours_1b = {"A": 3.0, "phi": 0.6}
replay_10b = {"A": -2.0, "phi": 0.8}
print("Ours at 1B tokens: A=%+.1f, Phi=%.1f" % (ours_1b["A"], ours_1b["phi"]))
print("Replay at 10B tokens: A=%+.1f, Phi=%.1f" % (replay_10b["A"], replay_10b["phi"]))
print("Ours wins on adaptation by %.1f points using 1/10th the tokens" % (ours_1b["A"] - replay_10b["A"]))
print("Ours also wins on forgetting: %.1f vs %.1f (lower is better)" % (ours_1b["phi"], replay_10b["phi"]))
PY
# Ours at 1B tokens: A=+3.0, Phi=0.6
# Replay at 10B tokens: A=-2.0, Phi=0.8
# Ours wins on adaptation by 5.0 points using 1/10th the tokens
# Ours also wins on forgetting: 0.6 vs 0.8 (lower is better)

Both numbers in this comparison are the paper's own reported means over three seeds, not independently re-measured. What a reader can check without rerunning the experiment is whether the paper's own 4B-token table is internally consistent with its "4.4B" mention elsewhere in the text, since that discrepancy is not resolved in the version fetched for this article.

What would prove this wrong

The benchmark is one domain (medical), one small backbone (TinyLlama-1.1B) with a single scale-up check (Llama-3.1-8B), and the Fisher reference set, the thing the whole method is calibrated against, is chosen by the same authors proposing the method. A different domain where general-domain and target-domain gradients overlap more heavily could shrink or erase the anchor-versus-frontier separation the method relies on.

A dated prediction: by 30 June 2027, an independent group will test Fisher-guided submodular selection on a non-medical domain (legal, code, or a non-English language corpus) and at a backbone size above 8B parameters, and report whether the 10x token-efficiency gap over Replay holds, narrows, or reverses. If it narrows substantially outside the medical domain the authors chose, the general claim ("selection beats replay") would need to be qualified by domain, not treated as domain-independent.

Sources

1. Zhao, Z., Liu, G., Lan, Z., Yan, Y. Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Models. arXiv:2610.02593, 1 October 2026, University of Illinois Chicago and Cisco Research. HTML version.
2. BLOMEGA. Active learning beats full-dataset training on the hardest anatomical structures.
3. BLOMEGA. LESS data selection, recounted.
4. BLOMEGA. Data annotation research: the latest.