# BERT Bi-Encoder Depth Ablation: Seeds, Training Budget, and Zero-Shot Transfer

**Finding:** Across five controlled experiments on `nfcorpus`, `scifact`, `fiqa`:

1. At a moderate training budget (20K steps on 500K MS MARCO triples), BERT bi-encoder depth from **L3 to L12 is invariant within seed noise** (three seeds; L12 σ = 0.0069 vs L3/L6/L9 σ ≤ 0.001).
2. Extended training (80K steps, 4× longer) **degrades zero-shot BEIR transfer for every depth**, with the degradation concentrated on fiqa (finance): L12 loses 45% of fiqa NDCG@10, L3/L6 lose ~28%. This is DRAGON-consistent: single-source overfitting hurts, and more capacity overfits harder.
3. **L6 is the single most robust operating point** across both training budgets — tied-best at 20K and clearly best at 80K, while encoding ~20% faster than L12.

See [report.md](report.md) for the full analysis and [RESULTS.md](RESULTS.md) for every metric.

## The five experiments

| Experiment | Batch | Steps | Seed | Purpose |
| --- | --- | --- | --- | --- |
| A | 256 | 20K | 42 | Large-batch baseline |
| B | 32  | 20K | 42 | Standard batch-32 run |
| C | 32  | 20K | 43 | Seed variance |
| D | 32  | 20K | 44 | Seed variance |
| E | 32  | **80K** | 42 | Training saturation |

## Headline numbers

Mean NDCG@10 across the three BEIR benchmarks:

| variant | 20K seed avg (B,C,D) | ±2σ | 80K (E) | p50 latency (canonical) |
| --- | --- | --- | --- | --- |
| L12 | 0.2400 | ±0.0138 | 0.2143 | 16.2 ms |
| L9  | 0.2385 | ±0.0006 | 0.2109 | 15.8 ms |
| L6  | **0.2438** | ±0.0020 | **0.2254** | 13.0 ms |
| L3  | 0.2422 | ±0.0018 | 0.2203 | 12.7 ms |

At 20K the mean NDCG@10 spread across depths is 0.0053 — smaller than L12's single-seed variance. At 80K, L6 opens a 0.005 lead over L3 and 0.011 over L12.

## What this does *not* claim

- Not a claim about BERT bi-encoders in general. The claim is bounded to this recipe (`MultipleNegativesRankingLoss`, 500K MS MARCO triples, top-N layer truncation) and the three BEIR tasks evaluated.
- Not a claim that 80K training always hurts. It hurts *zero-shot BEIR transfer from MS MARCO*; it might help on in-domain MS MARCO dev (not measured here).
- Not a claim that other ablation strategies (bottom-drop, alternating-drop, structured pruning) would agree. Only top-N truncation was tested.
- Not a claim that the precise 80K degradation magnitudes would hold across seeds — the 80K run is single-seed. The effect size is large enough that seed noise is unlikely to flip the direction, but seed replication would tighten the bound.

See "Limitations" and "Open Questions" in [report.md](report.md) for the full list.

## Figures

**Headline: depth × training budget.** Seed-averaged NDCG@10 at 20K (±2σ band) with the 80K single-seed line overlaid. Every depth loses at 80K; L6 is the most robust operating point across both budgets.

![depth × training budget](results/depth_vs_budget.png)

**Where the 80K degradation lives.** Per-benchmark NDCG@10 change from 20K (seed-avg) to 80K. nfcorpus and scifact are roughly flat-to-positive across depths; fiqa crashes for every depth, worst for L12 (−45%).

![per-benchmark degradation](results/per_benchmark_degradation.png)

**Seed variance is depth-dependent.** Per-seed NDCG@10 at 20K for each depth. L12's σ is 7–23× larger than L3/L6/L9 — a fingerprint of undertraining at deep capacity, not a real depth signal.

![seed variance](results/seed_variance.png)

**Pareto: accuracy vs. latency.** Seed-averaged 20K points and single-seed 80K points on the canonical batch=32 latency ladder. L6 sits on the Pareto frontier at both budgets.

![pareto frontier](results/pareto_updated.png)

## A note on latency profiling

Of five latency profiling runs on the same H100 machine, only two (Experiments B and E) produced a physically coherent p50 ladder: `p50 ≈ 12.3 + 0.33 × n_layers`. These two runs reproduce each other to the decimal. The other three profiling runs (Experiments A, C, D) under-report shallow-model p50 by 2–5× for reasons we could not isolate (likely candidates: CUDA kernel autotuner state, per-process kernel cache warm-up, Python timing noise at sub-10-ms encodes). we report the canonical B/E ladder in all headline tables and flag the anomalous runs in [RESULTS.md](RESULTS.md). The retrieval finding is architecture-determined and latency-independent.

## Reproduce

```bash
# Experiment A — batch=256, seed=42, 20K steps
bash scripts/run_all.sh --config config.yaml

# Experiment B — batch=32, seed=42, 20K steps
bash scripts/run_b32.sh --config config_b32.yaml

# Experiment C — batch=32, seed=43
bash scripts/run_b32.sh --config config_b32_s43.yaml

# Experiment D — batch=32, seed=44
bash scripts/run_b32.sh --config config_b32_s44.yaml

# Experiment E — batch=32, 80K steps, seed=42
bash scripts/run_b32.sh --config config_b32_80k.yaml
```

Checkpoints are cached and skipped on re-run. All scripts share the same downloaded data.

Individual stages:

```bash
# Train one variant
python -m src.train --config config_b32.yaml --n-layers 6

# Evaluate one variant
python -m src.evaluate --config config_b32.yaml --n-layers 6 --benchmark scifact

# Profile latency
python -m src.profile_latency --config config_b32.yaml --n-layers 6

# Regenerate all figures (depth_vs_budget, per_benchmark_degradation,
# seed_variance, pareto_updated)
python make_plots.py
```

## Hardware and runtime

- GPU: H100 (Azure VM)
- Training: ~1 hr per variant at batch=256; ~4 hrs per variant at batch=32 / 20K; ~16 hrs per variant at batch=32 / 80K (all bf16, seq=128)
- Eval: ~10–15 min per variant across 3 benchmarks
- Latency profiling: ~5 min total (all variants)
- Total across all five experiments: ~75 hours