DRAFT Date: 2026-10-10 Author: Reuben Bowlby •

Experiment Receipt Discipline for Small-Model Autoresearch

This is an unreleased working revision. Any DOI and earlier PDF shown here identify a previous snapshot and do not certify this revision or imply peer review.

Download working source

Experiment Receipt Discipline for Small-Model Autoresearch

Status: DRAFT — source-audited living revision, 2026-10-10

Date seeded: 2026-07-06

Author: Reuben Bowlby (ORCID 0009-0002-5620-1103)

Repository: hummbl-io/hummbl-papers

Source snapshot: autoresearch-pipeline f39a3eec3a24a02c1e11d421d5752953e49d10db

This Markdown revision supersedes earlier draft interpretations. The existing PDF and deposit metadata remain historical release artifacts; they have not been regenerated or republished with these corrections.

Abstract

Experiment receipts can preserve the connection between a research proposal, its execution conditions, its measurements, and a keep or discard decision. This paper examines that design in one first-party autoresearch corpus and audits the numerical claims made from it. At the pinned source revision, the results table contains 243 run rows: 55 labelled keep, 186 discard, and two crash. Those counts are not counts of accepted patches. Two initial control receipts each contain five seeds; their observed validation bits-per-byte ranges are 0.180419 and 0.004988 under different configurations. The program records a decision gate based on those controls, but subsequently identifies the proposed QK-Norm patch as already present in the baseline. Five later candidate settings were discarded against a distinct BATCH 2x baseline. The audit also exposes changes in the definition of "noise floor," baseline mismatches, and counting errors in the earlier manuscript. We propose a receipt contract that makes units, measurement definitions, baseline lineage, and decision timing explicit. The case supports the inspectability of recorded decisions, not a causal claim that receipts improve model performance, eliminate missing results, or establish a scale threshold for architectural features.

1. Question and contribution

The research question is: what information must an agent-driven experiment loop preserve so that another reader can reconstruct and challenge its decisions? This differs from asking whether a particular architecture improves a language model. The contribution is a worked audit, a proposed receipt schema, and a replication plan with explicit availability requirements.

The motivating risks are selective reporting, comparisons against a different baseline, confusing variation across runs with an inferential threshold, and transferring a feature without testing the intended compute regime. Receipts make these risks inspectable when relevant information is recorded. A receipt can itself contain an error, an unsupported causal explanation, or an incomplete history; preserving it is not proof that its conclusions are correct.

Pineau et al. describe code submission, a reproducibility challenge, and a reporting checklist as complementary components of the NeurIPS reproducibility program. This paper adapts documentation principles to experiment authorization and result review; it does not characterize that program as merely retrospective. Pineau et al., 2021. Preregistration distinguishes a plan made before observing outcomes from later exploration. The present study is a retrospective source audit, not a preregistered evaluation of receipt efficacy. Center for Open Science.

2. Materials, access, and audit method

On 2026-10-10 we read Git objects from the pinned autoresearch-pipeline revision in an author-accessible local checkout. We counted results.tsv data rows using its tab-separated header, enumerated the nine Markdown receipt documents, and recomputed descriptive statistics from the control and Pack 2 tables. No models were trained, training logs independently inspected, or new performance experiments conducted in this audit.

The audit is reproducible at two different levels:

1. Arithmetic replay: source_audit.json supplies the selected numerical transcription, source paths, hashes, and recomputed statistics. A reader can check its arithmetic without the original repository.
2. Source verification: audit_corpus.py reads the pinned Git objects and reconstructs the JSON. This requires access to that source revision. The complete source repository, training code snapshots, dataset, and raw training logs are not bundled with this paper. Their accessibility to an unauthenticated external reader has not been established here.

Hashes identify the bytes examined; they neither authenticate an experiment's execution nor grant access to the original data. The nine receipts are wave-level documents, not nine independent experiments or a guarantee that every run has a complete individual receipt. The source program.md is a mutable planning document at a historical snapshot; its embedded dates alone do not prove that every rule preceded the corresponding execution.

2.1 Units of analysis

Thus "34 experiments, two keeps, 32 negative experiments" conflates no-ops with executed tests. Likewise, 55 keep-labelled rows do not imply 55 distinct retained patches. Controls and seed replicates must be joined to candidate identifiers before an experiment-level retention rate can be computed. The paper does not provide a complete row-to-candidate linkage audit.

3. Proposed receipt contract

The following seven fields are a proposed reporting contract, not a claim that all historical records satisfy it.

These fields distinguish execution status from scientific conclusion. A crash is an execution failure, a no-op was not a new treatment, and DISCARD is an operational choice that may reflect harm, insufficient evidence, overhead, or another stated criterion. None is automatically a statistically established negative effect.

An inherited control is admissible only with a linked baseline and an explicit justification that the conditions relevant to variability have not changed. Presence of a commit hash alone does not establish complete provenance. A completeness audit should classify each applicable field as direct, explicitly inherited, missing, or contradictory, retain the evidence span, and report disagreements between auditors. The earlier draft's 34/34 and 24/34 field-completeness percentages are withdrawn because no per-candidate coding matrix accompanied them.

4. The control gate and the QK-Norm no-op

The source program proposed a QK-Norm experiment with an anticipated 0.002–0.005 bpb improvement. This interval is a planning assumption in the source, not an effect estimate established by an external citation. The first control receipt records authorization to run the control alone and recommends withholding the candidate experiment. The tuned control later records conditional authorization. The same pinned program then reports that QK normalization and logit soft-capping were already implemented: both first-wave proposals became NO-OP entries, not accepted architecture patches.

4.1 Recomputed controls

All three control groups below contain seeds 42, 137, 271, 500, and 999. Statistics are recomputed from the six-decimal values in the receipts, so precision beyond the source measurements is not implied.

The initial pair of receipts documents a 300-second budget per run. Ten configuration parameters changed between the untuned and tuned controls, including depth, head dimension, learning rates, weight decay, and batch size. Consequently, the range ratio of about 36.17 is descriptive of these two recorded configurations; it does not identify a causal effect of batch size alone. The untuned control also differs from the nominal 26M tuned regime, so the entire corpus should not be described as one fixed-size model experiment.

The untuned range divided by the anticipated improvement is about 36–90. This provides a practical reason to investigate variability before running a small-effect comparison. It does not prove that any possible candidate outcome would be indistinguishable from noise, establish a sampling distribution, or demonstrate what would have happened without the gate. The recorded untuned minimum and maximum are 0.856008 and 1.036427, not the interval asserted in the earlier draft.

4.2 "Noise floor" needs a definition

The first two receipts use max(value) - min(value). The later BATCH 2x receipt calls 2 × sample SD = 0.000567 its noise floor. Comparing the tuned range to this later quantity as an "8.8× tighter range" mixes estimators. A consistent comparison gives a range ratio of approximately 6.42 (0.004988/0.000777), or an SD ratio of approximately 6.50. Neither quantity is a confidence interval or a minimum detectable effect.

The sample range is sensitive to N and can increase as more seeds are added. Additional runs can improve estimation; they do not necessarily lower measured variation. The conditional source rule—at least three candidate seeds, a two-sample test, and mean improvement above 0.003—is reported as a historical decision rule, not endorsed as an adequately powered universal test. Prospective studies should define the estimand, pairing, practical threshold, multiplicity policy, and uncertainty analysis before inspecting candidate outcomes.

5. Pack 2: baseline lineage changes the reading

Pack 2 contains five candidate settings with three reported seeds each. It compares against the later BATCH 2x baseline, whose matching seed values are 0.475223, 0.475353, and 0.475251. GELU is in Pack 1, not Pack 2. Using the earlier tuned mean of 0.480367 as Pack 2's baseline changes the apparent effect and is not a valid comparison.

Define the descriptive seed-paired difference as candidate_bpb - baseline_bpb; positive values mean worse validation performance. The table below averages those three differences and preserves the source's operational decisions.

These values differ slightly from the source summary deltas. The source values are consistent, to reported rounding, with subtracting the five-seed control mean rather than the three matching controls. We state the estimator instead of silently mixing the two. We have not revalidated the source's t-statistics or their selection procedure and do not reproduce them as confirmatory evidence.

The observed setting differences are compatible with several explanations: implementation choices, interactions with an already tuned baseline, altered steps under a fixed time budget, or model size. A one-size, short-budget comparison cannot isolate a scale interaction. No result here establishes that frontier features generally fail to transfer, identifies a threshold size, or warrants causal claims about attention sharpness or information loss. The value of preservation is that these alternative readings remain possible.

6. What the case does and does not show

The source contains discarded run records, control measurements, a documented authorization sequence, and no-op discoveries. Their retention enables this audit, including corrections to the paper's own earlier interpretation. This is evidence of inspectability for the included records.

It is not evidence that all attempted runs were recorded. Without an independent scheduler or launch ledger, the denominator of unrecorded attempts is unknown; a "0% silent dropping" rate cannot be estimated. Nor does a single recorded control gate demonstrate that all architecture proposals were gated. No contract-free control group is available to estimate the causal effect of receipt discipline on completeness, researcher behavior, review time, or model quality.

The case is also limited by first-party authorship, adaptive candidate selection, reuse of control runs, small seed counts, a mixture of configurations and budgets, and lack of publicly bundled training snapshots and logs. Documentary source verification is weaker than independent rerunning. The initial notebook entries include retrospective statements and known outcomes; their headings do not establish preregistration.

7. Evaluation and replication plan

Two evaluations should be kept separate:

• Receipt efficacy: assign comparable experiment tasks to ordinary logging or the proposed contract; fix the required run schedule independently; measure missing-receipt rate against that schedule, field completeness with explicit denominators, blinded decision-reconstruction accuracy, review time, and recording overhead. Randomize at a task/session level that limits prompt carryover; count crashes, skipped tasks, and verification failures. Predeclare a primary endpoint and sample-size rationale before collection.
• Model-performance replication: release the exact training revisions, dataset/evaluation hashes, config files, dependencies, and run commands; reproduce or explicitly depart from the original time budget and hardware; report all seeds, uncertainty, and deviations. A new QK-Norm removal/addition study would be a new ablation, not a replication of an original KEEP.

The replication protocol gives the source audit commands, known control differences, statistical definitions, and remaining release prerequisites. No future access or successful replication is promised. An independent, unsuccessful, or inconclusive replication would be informative if its conditions and missing data are documented.

References and source artifacts

• Pineau, J., Vincent-Lamarre, P., Sinha, K., Larivière, V., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Larochelle, H. (2021). Improving Reproducibility in Machine Learning Research. JMLR 22(164), 1–20.
• Center for Open Science. Preregistration. Accessed 2026-10-10.
• Bowlby, R. (2026). autoresearch-pipeline, source snapshot f39a3eec3a24a02c1e11d421d5752953e49d10db; first-party source, external availability not established by this audit.
• Source audit and selected numerical transcription; audit program; research notebook.