# Source Audit and Conditional Replication Protocol

**Revision:** 2026-10-10. This replaces an earlier protocol that incorrectly gave identical untuned/tuned configurations, three rather than five control seeds, and a QK-Norm KEEP prediction. No new training was conducted. This is a retrospective audit protocol plus a plan for future replication; it is not a registered report or a complete executable training package.

## 1. Reproduce the documentary audit

With Python 3 and Git, from this paper directory:

```bash
python audit_corpus.py /path/to/authorized/autoresearch-pipeline > reproduced_audit.json
```

The script reads commit `f39a3eec3a24a02c1e11d421d5752953e49d10db` directly. It does not check out a branch, execute training, fetch data, or modify the source repository. Compare the parsed JSON with [source_audit.json](source_audit.json); whitespace is immaterial. Failure to access the revision is an availability failure, not a scientific non-replication. The bundled JSON permits arithmetic replay alone if source access is unavailable.

The results table hash is `6ef53e29c12fb1099c3b4248d50aa20c3d84cbc5068daeaf027ad047c1448355`. Receipt hashes in the JSON identify the exact source bytes used. A matching hash establishes identity of the file, not whether training actually occurred.

## 2. Prerequisites before training replication

Do not describe a rerun as exact until the following are accessible and frozen:

| Requirement | Current status in this paper's package |
|---|---|
| Complete training code and patch diffs | Not bundled; receipts cite short `autoresearch-win-rtx` commits `d2ce712` and `a17a560` for initial controls |
| Full code commit resolution | Those short training commits were not resolvable in the locally inspected training checkout during this audit |
| Dataset, tokenizer, validation split and byte normalization | Need versioned identifiers/hashes and evaluation commands |
| Complete configs, dependencies, driver/runtime versions | Partial receipt descriptions available; machine-readable replay package still needed |
| Training commands, raw logs, seed handling and nondeterminism settings | Must be released or made available under an explicit access procedure |
| Hardware/power/thermal and time-budget semantics | Receipts identify Windows, RTX 4000 series, CUDA 12.8 and 300 seconds for initial controls; exact GPU and timing treatment need confirmation |

No minimum VRAM, disk requirement, or runtime portability claim has been benchmarked by this paper. A dry-run feasibility check on a replicator's GPU is appropriate after code and data are available. No purchase or paid compute is necessary for the documentary audit.

## 3. Known configuration differences

The tuned control receipt lists ten simultaneous changes. These are transcribed for reconciliation, not a substitute for the complete training configuration.

| Parameter | Experiment 0 | Experiment 0b |
|---|---:|---:|
| TOTAL_BATCH_SIZE | 2^19 | 2^14 |
| HEAD_DIM | 128 | 64 |
| DEPTH | 8 | 6 |
| EMBEDDING_LR | 0.6 | 0.32 |
| UNEMBEDDING_LR | 0.004 | 0.0064 |
| MATRIX_LR | 0.04 | 0.032 |
| SCALAR_LR | 0.5 | 0.4 |
| WEIGHT_DECAY | 0.2 | 0.0 |
| WARMDOWN_RATIO | 0.5 | 0.75 |
| FINAL_LR_FRAC | 0.0 | 0.02 |

Both receipts list seeds `[42, 137, 271, 500, 999]` and a **300-second** budget. Steps are outcomes (42–44 and 2,007–2,189), not a fixed 2,100-step input. At five minutes per run, ten initial control runs alone imply 50 minutes of training budget, excluding setup, evaluation and compilation. The later BATCH 2x control changes batch size to 2^15 and is a separate baseline. Do not apply later kept changes to reconstruct Experiment 0b.

QK-Norm and logit soft-capping were discovered already in the baseline. The source records them as no-ops. A new removal/reintroduction ablation needs a new plan and cannot confirm an original QK-Norm KEEP that did not occur in these wave summaries.

## 4. Estimands and measurement definitions

For recorded outcomes `y_1, ..., y_n`, calculate:

- Mean: sum of outcomes divided by N.
- Sample SD: square root of the sum of squared deviations divided by `N - 1`.
- Observed range: maximum minus minimum.
- CV: `100 × sample_SD / mean`, labelled as a percentage.

Report N, all seed outcomes, failures, and exclusions. Do not equate a range or twice the SD with statistical significance. Do not compare ranges from different N as if N had no effect. More runs do not necessarily reduce the observed range.

For candidate comparisons define `d_seed = candidate_bpb - matched_baseline_bpb`: negative means improvement. Use matching seeds only when the design, data ordering, and other coupling justify pairing; record the choice before outcomes are examined. If independent groups are used, state that estimand and estimator instead. Shared controls induce dependence across candidate comparisons.

Keep fixed-time and fixed-token experiments distinct. Fixed-time comparisons include implementation throughput effects, whereas fixing tokens or steps changes the experiment's question. Record both realized steps/tokens and elapsed time. Randomize or counterbalance run order within feasible hardware blocks; avoid competing jobs and log resource contention instead of assuming sequential execution eliminates it.

## 5. Future analysis plan

Before collecting new runs, archive a timestamped protocol specifying the primary comparison, practical effect size, confidence interval procedure, planned N, seed list, multiplicity adjustment or exploratory designation, stopping rule, failure handling, and allowed deviations. A power or precision analysis should use assumptions appropriate to the proposed design; three or ten seeds is not a universal adequacy threshold.

The historical values below are comparison targets, **not preregistered predictions of this retrospective audit**:

| Control | Historical N | Mean bpb | Sample SD | Range |
|---|---:|---:|---:|---:|
| Untuned | 5 | 0.921964 | 0.080456 | 0.180419 |
| Tuned | 5 | 0.480367 | 0.001843 | 0.004988 |
| BATCH 2x | 5 | 0.475312 | 0.000283 | 0.000777 |

A rerun need not equal rounded historical values. Report effect estimates, uncertainty, and compatibility with a prospectively chosen tolerance. A different five-seed range alone neither confirms nor falsifies the claimed utility of receipt discipline. Report an inconclusive result when uncertainty is too large. To attribute a variance change to one parameter, add an isolated controlled comparison; the ten-change historical comparison cannot identify that cause.

## 6. Replication receipt

Use the seven fields from [the manuscript](paper.md): hypothesis/plan, baseline, variability assessment, measured outcome, decision, configuration, and provenance. Include all planned run IDs and explicitly mark crashed, skipped, or unavailable runs. Preserve raw logs; if access restrictions prevent sharing, describe the restriction and downgrade the corresponding reproducibility claim.

State who performed the work, their relationship to the original project, funding or relevant interests, access provided by the authors, and any feedback received before analysis. Independence is a disclosure to assess, not a requirement that excludes informative affiliated replications. Submit reports through the repository's coordinated contribution process when ready; this protocol does not send or publish results automatically.