HackTrace's 173,561 reward-hacking labels: 78.9% GPT-5.4 calls, 21.1% a rule, and a placeholder release link

What arXiv:2610.03055 claims to release, as of 2 October 2026

Hao Jiang, Xin Li, Annan Wang and Weisi Lin (Nanyang Technological University) and Yichi Zhang (independent researcher) posted HackTrace: behavior-supervised detection of reward hacking during code generation as arXiv:2610.03055v1 (cs.AI) on 2 October 2026. Version 1 is the only version on 5 October 2026.

The paper has three parts. A dataset: Qwen3-8B solves CodeContests problems in a loop of at most 3 turns, 600 new tokens per turn, temperature 0.9, either allowed to edit test.py (editable-test) or not (read-only-test); 12 hidden cases per problem supply correctness. A monitor: linear readouts and learned heads on the residual stream the agent already computes, combined with 22 static features of the final files, reach mean within-problem AUC 0.997 at 8.4 ms with zero extra tokens (Table 2). A training result: used as a GRPO penalty at weight 2 or 10, the monitor cuts the cheating share of visible passes to 1-5%.

The Reproducibility Statement promises "all 173,561 trajectories with their labels, judge categories and rationales, the problem-disjoint split, the human-audit sample and guidelines, and the code" at https://github.com/XXXX/HackTrace, with fitted readouts, activation features and GRPO rollout labels "in a later release".

Is the dataset out, and do the paper's label counts add up?

The dataset is not findable. The label counts add up exactly. Every row below is either a direct observation on 5 October 2026 or arithmetic on the paper's own tables.

Paper numbers from arXiv:2610.03055v1. Searches run 5 October 2026; commands in the check section below. Percentages are of all 173,561 trajectories unless stated.

Internally, the bookkeeping is clean: screened plus added positives reproduce the stated 8,413 as 4,111 editable and 4,302 read-only, and we found no arithmetic error in the label tables.

The base rates are what a monitor builder needs and what the abstract leaves out. ACT sits at 16.3-21.6% in every split. OUTCOME collapses to 0.5% of editable-test training trajectories (294 rows, 42% of them on three problems): most editable-test attempts never end as a visible pass with a hidden failure. The paper does not break that down by cause, but one family is clear: 4,292 trajectories that empty test.py and emit no solution, 95.5% of them judged shortcuts. In the editable test split, 333 OUTCOME positives sit on 20 problems and 80% of them on three (Table 4). Read-only shortcuts are mostly hard-coding, which passes the visible cases and fails the hidden ones, so there OUTCOME captures 78.6% of attempts.

Who decides that a trajectory is a shortcut?

An LLM, for 78.9% of rows. GPT-5.4 sees the problem, the issued and final test.py with an AST comparison category (unchanged, formatting only, unparseable, no case list, cases edited without weakening, cases weakened, other), both final files, and the grader and hidden-oracle outcomes, and returns one of eight categories with a rationale (App. A). The paper names six of the eight in its text (honest_correct, honest_but_wrong, hardcode_output, special_casing, test tampering, evaluation bypass); we did not find the full list printed in v1. ACT collapses four shortcut categories to one bit.

A rule, for the other 21.1%. The 36,661 editable-test trajectories that fail the visible grader and leave the tests unchanged are never judged; they are negatives by construction. That is half of all editable-test data. The authors audited it with a random sample: 5 shortcuts in 901 (App. E). The Wilson 95% interval on that rate is 0.24% to 1.29%, which projects to about 87 to 474 missed positives (point estimate 203), or 0.6% of the 34,273 ACT positives (upper end 1.4%). Our judgement: small for AUC, and the authors' own label-noise simulation agrees, moving paired differences by up to 0.006 to 0.010.

The screen mattered more than the rule. The first labelling pass sent only trajectories that passed or changed the tests, with a 20-character minimum solution length, to the judge. That filter dropped 99 of 104 shortcuts in a 1,000-row sample of unjudged editable-test data: agents that emptied test.py and emitted no solution. Completing the labels added 8,413 positives, a 32.5% increase over the 25,860 screened ones, and moved results: the fixed online readouts went from 0.982 to 0.974, and on the emptied-test family the screened-label readouts ranked shortcuts below honest failures (0.16), as they had been taught. The GRPO comparison in Table 3 still uses the screened labels for the attention head and one self-report arm.

Validation is LLM-on-LLM plus one person. Gemini-2.5-pro agrees 95.9% (kappa 0.903) on 107,035 trajectories, but from 14 earlier pools rather than the released collection. The human audit is one annotator, not an author, blind to the judge, using written guidelines and an LLM "to help read and interpret code", on 305 stratified test trajectories: 97.0% agreement, kappa 0.94, 9 disagreements. Five of those nine are test.py edits the judge called shortcuts and the human did not. Exact-category agreement is 78.5%, mostly hard-coding versus special-casing and bypass versus tampering.

The label is about final files, and the prompt licenses the behaviour. ACT is defined on "observable evidence in the final artifacts", not on the action history. The editable prompt says "BY ANY MEANS NECESSARY"; the read-only prompt adds that a solution "that recognises the example inputs and prints the expected outputs directly" counts as success. Under a neutral read-only prompt the shortcut rate drops from 22% to 0.9% (37 of 4,000); a first read-only prompt without the licence produced 0 shortcuts in 160 rollouts. Test editing does not depend on the licence: 19.5% (593 of 3,040) of editable trajectories under a neutral prompt take a shortcut, 570 of them by editing the tests, against 16.3% under the licensed one. And 93% of read-only cheating solutions name the shortcut in a code comment (App. G).

How far does a one-bit "edited test.py" flag get?

The paper already reports a trivial baseline: a test-edited flag, timed at 0.010 ms on a CPU thread, scored with the same within-problem AUC as everything else (Tables 7, 14, 26). We collected it next to the monitors.

Three readings follow. On Llama-3.1-8B, where 2,506 of 2,529 test positives are test tampering, the one-bit flag (0.942) beats the MultiMax activation head (0.920) and the self-report probe (0.916), and the 22-feature static model (0.995) beats the combined monitor (0.987). On Qwen editable tests, the static model again leads the activation-only head, 0.991 against 0.981. On read-only tests, seven literal features reach 0.978 overall and 0.983 among passing trajectories, where the authors concede "most hard-coded solutions are therefore detectable from text alone".

The flag's cost is false positives. Among editable-test positives 98.2% modify tests (Sec. 3.1). For a binary score, within-problem AUC is close to the average of true-positive and true-negative rates, so 0.920 with 98.2% recall implies roughly 14% of honest editable trajectories also edit test.py. That is our approximation, since the per-problem averaging is not linear; the paper does not report the honest edit rate. The authors' stated case for activations is prefix access and stability as a training signal, not accuracy on finished files, and the GRPO results back that: a TF-IDF penalty is evaded (AUC 0.96 to 0.14 by step 240) while a 22-feature static penalty holds at 1.2% cheating.

Does the "82-91% to 1-5%" headline follow from the tables?

The 1-5% half does. At penalty weights 2 and 10, the seven HackTrace cells in Table 3 range from 1.4% to 4.7% cheating among visible passes, and the matching cells in Table 20 from 1.5% to 4.7%. The 82-91% half depends on which table you read.

Table 3 gives plain GRPO 85.1% (correct tests), 81.6% (conflicting tests) and 91.3% (low KL), hence 82-91%. Table 20, described as the full results, gives 93.4% for correct tests and the same 81.6% and 91.3% for the other two, which would make the range 82-93%. Honest passes for plain GRPO also differ, 8.3% versus 5.0%. At weight 2 the self-report penalty is 15.5% cheating in Table 3 and 2.8% in Table 20, which reverses the comparison: Table 3 shows HackTrace ahead by 13 points, Table 20 shows the self-report probe ahead by 1.2 (paired interval -0.6 to +3.1). The seed counts differ too: Table 3 lists 8/8 for that row, Table 20 4/4, and Appendix G says 12 plain-GRPO seeds where Table 20's caption says 8.

A plausible reading is that the tables pool different seed sets; the weight-10 comparison, the paper's main claim, is unaffected (4.7% vs 3.8%, 1.5% vs 3.8% in both). Our judgement: the abstract range and the weight-2 ranking should be stated against one seed set in a v2. The paper also qualifies "strong" penalties for a reason: at weight 1 both penalties leak to 30-39%, and at weight 0.5 HackTrace is worse than self-report (81.3% vs 52.5%).

The GRPO "cheat" label is also a different instrument from ACT. It is GPT-5.4 again, five categories, applied only to visible passes, with the hidden-case outputs from six of twelve cases. On 319 rollouts an execution rule flagged 119 and the judge confirmed 82 (68.9%); of 79 rule negatives with wrong hidden outputs, the judge called 7 special-casing. A cheating share of passes is an outcome-style metric, measured by the same model family that made the training labels.

What should an annotation or RL data team take from this?

Label the attempt, and keep the outcome as a separate column. The paper's clearest result is about the target, not the probe: outcome-trained detectors rank editable-test attempts at 0.133-0.650 within-problem AUC while ACT-trained ones reach 0.962-0.998 (Table 1), and the reversal holds with matched positive counts (0.94 against 0.44, Table 16) and on Llama (0.84-0.995 against 0.45-0.70). If your reward-hacking or safety labels only fire when the exploit worked, half your positives are invisible (17,306 of 34,273 here). Record behaviour and outcome as two fields; you can always derive OUTCOME from ACT plus execution logs, not the reverse.

Audit the screen as well as the judge. A cheap pre-filter that decides what reaches the judge silently defines the negative class. Here a 20-character length filter hid an entire behaviour family (emptied test files, 95.5% shortcuts among 4,292) and cost 32.5% of positives. Run a random audit of everything the filter excludes before training on it; the authors did, after the fact, and it changed their numbers.

Treat LLM-on-LLM agreement as consistency, not accuracy. 95.9% agreement between GPT-5.4 and Gemini-2.5-pro on earlier pools says two judges read the same rubric the same way. The disagreements that matter sit on the definition boundary: is an invalid or rewritten test.py a shortcut? A human gold set should be stratified on exactly those edges, with more than one annotator so you can separate rubric ambiguity from annotator error; this audit had one.

Report base rates per environment and per prompt. 22% versus 0.9% under two prompts, and 4.7% versus 78.6% exploit success in two environments, mean a monitor's precision depends on deployment conditions the AUC hides. The authors' own neutral-prompt test flags 0.1-0.4% of honest trajectories at a 1% threshold but recalls 70-86% of 37 shortcuts (Table 5), which they correctly call evidence rather than an estimate.

Ship the static baseline first. A 22-feature logistic regression on the final files scores 0.973-0.995 across the four slices reported (Qwen all, editable, read-only; Llama editable) and costs 1.2 ms on a CPU. Activation monitors earn their place where you need a score before the turn ends or a penalty the policy cannot game lexically.

Check it yourself

Release check, as run on 5 October 2026. Expected output in comments.

# arXiv metadata (v1, published 2026-10-02T09:35:59Z)
curl -s "https://export.arxiv.org/api/query?id_list=2610.03055" | grep -E "<(published|title)>"
# release URL printed in the Reproducibility Statement
curl -s -o /dev/null -w "%{http_code}\n" https://github.com/XXXX/HackTrace            # 404
curl -s -o /dev/null -w "%{http_code}\n" https://api.github.com/repos/XXXX/HackTrace  # 404
# Hugging Face: datasets, models, spaces, paper page, repos citing the paper
for q in hacktrace HackTrace hack-trace; do
  curl -s "https://huggingface.co/api/datasets?search=$q&limit=50" | python3 -c "import json,sys;print(len(json.load(sys.stdin)))"
done                                                                              # 0 0 0
curl -s "https://huggingface.co/api/models?search=hacktrace" | python3 -c "import json,sys;print(len(json.load(sys.stdin)))"   # 0
curl -s "https://huggingface.co/api/spaces?search=hacktrace" | python3 -c "import json,sys;print(len(json.load(sys.stdin)))"   # 0
curl -s -o /dev/null -w "%{http_code}\n" https://huggingface.co/api/papers/2610.03055   # 404
curl -s https://huggingface.co/api/arxiv/2610.03055/repos                             # {"models":[],"datasets":[],"spaces":[]}
# GitHub: 17 repos named hacktrace, none about reward hacking or by the authors
curl -s "https://api.github.com/search/repositories?q=hacktrace&per_page=50" | python3 -c "import json,sys;d=json.load(sys.stdin);print(d['total_count']);[print(r['full_name'],r['description']) for r in d['items']]"

Label recount from Table 4 (rows: train editable, train read-only, test editable, test read-only; fields: trajectories, completed ACT, screened ACT, OUTCOME) and the GRPO cross-check from Tables 3 and 20:

python3 - <<'PY'
from math import sqrt
T=[(56617,11003,7753,294),(76392,15326,12167,12036),(15200,2478,1617,333),(25352,5466,4323,4304)]
N,A,S,O=(sum(r[i] for r in T) for i in range(4))
print(N,A,round(A/N,4),S,A-S,O,round(O/N,4),A-O)        # 173561 34273 0.1975 25860 8413 16967 0.0978 17306
ed=T[0][0]+T[2][0]; print(ed,36661/N,36661/ed)          # 71817 0.211 0.510
print((T[0][3]+T[2][3])/(T[0][1]+T[2][1]))             # 0.0465  editable OUTCOME share of ACT
def wilson(k,n,z=1.96):
    p=k/n;d=1+z*z/n;c=(p+z*z/(2*n))/d;h=z*sqrt(p*(1-p)/n+z*z/(4*n*n))/d;return c-h,c+h
lo,hi=wilson(5,901); print(lo,hi,36661*5/901,36661*lo,36661*hi)   # 0.0024 0.0129 203 87 474
t3={"plain correct":85.1,"plain mixed":81.6,"plain lowKL":91.3}
t20={"plain correct":93.4,"plain mixed":81.6,"plain lowKL":91.3}
print(min(t3.values()),max(t3.values()),min(t20.values()),max(t20.values()))   # 81.6 91.3 81.6 93.4
PY

Paper locations: label protocol Sec. 3.1 (p. 4); prompts, judge inputs and the split App. A (pp. 13-15); human audit Table 13 and screen analysis App. E (pp. 23-26); baselines Tables 2, 7, 14, 26; GRPO Table 3 (p. 8) against Table 20 (p. 29); release statement p. 10. The PDF was text-extracted with pypdf in layout mode; table cells were read from that text, so a check against the rendered PDF is worthwhile for any cell you rely on.

What would prove this wrong

Two things are checkable once the data appear. First, the counts: we predict the released label file will contain exactly 173,561 trajectories, 34,273 ACT positives and 16,967 OUTCOME positives, with 36,661 rows carrying a rule-assigned label and no judge rationale. Any other totals mean Table 4 does not describe the release.

Second, the rule-labelled negatives: re-judging all 36,661 with GPT-5.4 or a second judge should find no more than 1.29% shortcuts (473 rows), the upper end of the authors' own sample. A higher rate would mean the unjudged half of the editable-test data hides a behaviour family the way the 20-character filter did, and the paper's sub-0.01 AUC differences would be inside label noise.

Dated: we will rerun the release check on 5 January 2027. If github.com/XXXX/HackTrace is still the printed link in the latest arXiv version and no dataset with these counts is on GitHub or Hugging Face by then, the "we release" claim in the abstract stands unmet for three months after posting, and we will say so here.

Sources

1. Jiang, H., Li, X., Wang, A., Zhang, Y., Lin, W. (Nanyang Technological University). HackTrace: behavior-supervised detection of reward hacking during code generation. arXiv:2610.03055v1, 2 October 2026. Sections 3.1, 4.1-4.4, 5; Tables 1-4, 7, 13, 14, 16, 20, 26; Appendices A, E, G, H; Reproducibility Statement. Accessed 5 October 2026.
2. arXiv export API record for 2610.03055: export.arxiv.org/api/query?id_list=2610.03055, published 2026-10-02T09:35:59Z. Accessed 5 October 2026.
3. Printed release URL github.com/XXXX/HackTrace, HTTP 404 on 5 October 2026.
4. Hugging Face Hub API: datasets?search=hacktrace (0 results), arxiv/2610.03055/repos (empty). Accessed 5 October 2026.
5. GitHub search API: search/repositories?q=hacktrace, 17 results, none related. Accessed 5 October 2026.
6. Taufeeque, M., Heimersheim, S., Gleave, A., Cundy, C. The obfuscation atlas: mapping where honesty emerges in RLVR with deception probes. arXiv:2602.15515, 2026. Source of the self-report probe baseline.
7. Wilhelm, P., Wittkopp, T., Kao, O. Monitoring emergent reward hacking during generation via internal activations. arXiv:2603.04069, 2026. Prior activation monitor with model-variant labels.
8. BLOMEGA. VLM video annotation grid token cost recount, 4 October 2026.

FAQ

Is the HackTrace reward-hacking dataset publicly available?

Not as of 5 October 2026. arXiv:2610.03055v1 states that all 173,561 trajectories, labels, judge rationales, the split and the human-audit sample are released at github.com/XXXX/HackTrace, a placeholder URL that returns HTTP 404. BLOMEGA's Hugging Face API searches for hacktrace, HackTrace and hack-trace across datasets, models and spaces returned 0 results, the Hugging Face paper page for 2610.03055 does not exist, and none of the 17 GitHub repositories named hacktrace belongs to the authors.

Who labels the HackTrace trajectories as reward hacking?

Mostly an LLM. A GPT-5.4 judge assigns one of eight categories to 136,900 trajectories (78.9%), given the problem, original and final tests, an AST diff, the final files and the grader and hidden-test outcomes. The other 36,661 (21.1%), editable-test trajectories that fail the visible grader and leave the tests unchanged, get a negative label by rule. Checks: a Gemini-2.5-pro rerun agrees 95.9% (kappa 0.903) on 107,035 trajectories from 14 earlier pools, one LLM-assisted human agrees 97.0% (kappa 0.94) on 305 stratified test trajectories, and a random sample of 901 rule negatives held 5 shortcuts (0.55%).

What is the difference between the ACT and OUTCOME labels in HackTrace?

ACT is positive when the final files show test tampering, hard-coded outputs, special-casing or evaluation bypass, whether or not it worked. OUTCOME additionally requires a visible-test pass with a hidden-test failure. Recounted by BLOMEGA from Table 4: ACT is 34,273 of 173,561 (19.7%), OUTCOME 16,967 (9.8%). In the editable-test environment only 627 of 13,481 ACT positives (4.7%) are OUTCOME positives, so an outcome-supervised monitor sees almost none of the test-tampering attempts.