Depth, Seeds, and Training Budget in BERT Bi-Encoder Retrieval
A controlled ablation across three BEIR tasks.
Headline. At 20K training steps, BERT bi-encoder depth from L3 to L12 is invariant within seed noise. At 80K steps, every depth degrades on zero-shot BEIR transfer — catastrophically on fiqa (−45% NDCG@10 for L12). L6 is the single most robust operating point across both training budgets, while encoding ~20% faster than L12.
The experiments
Five controlled runs. Depth is the only within-experiment variable.
| Experiment | Batch | Steps | Seed | Purpose |
|---|---|---|---|---|
| A | 256 | 20K | 42 | Large-batch baseline |
| B | 32 | 20K | 42 | Standard batch-32 run |
| C | 32 | 20K | 43 | Seed variance |
| D | 32 | 20K | 44 | Seed variance |
| E | 32 | 80K | 42 | Training saturation (4× B) |
Each experiment trains four depths ({3, 6, 9, 12}) identically and evaluates on nfcorpus, scifact, fiqa with NDCG@10, MRR@10, Recall@10, Recall@100.
Headline figures
Figure 1. At 20K the depth curve is flat within the shaded band; at 80K every depth has dropped, and the drop is largest at L12.
Figure 2. nfcorpus and scifact move within noise; fiqa crashes for every depth — worst for L12 (−45%).
Figure 3. L12’s σ across three seeds is 0.0069, vs. σ ≤ 0.001 for L3/L6/L9. Deep models are undertrained at 20K; shallow models have already converged.
Figure 4. L6 sits on the Pareto frontier at both training budgets — tied-best at 20K, clearly best at 80K.
Read the writeup
- Full report — ~4-minute read. Hypothesis, findings, analysis, limitations, related work.
- Numerical results — every metric from every one of the five experiments.
- Repo README — one-scroll summary and reproduce instructions.
Contributions, calibrated
This is a small, well-controlled empirical study — not a new method, architecture, or training recipe. Specifically:
- A depth-dependent seed-variance signal. σ_L12 is 7–23× larger than σ_L3/L6/L9 at the same training budget. Usable as a cheap undertraining diagnostic for depth-ablation studies.
- A clean depth × training-budget interaction. More training compute makes every depth worse on zero-shot BEIR transfer; deeper models degrade more. DRAGON-consistent, but demonstrated with the depth axis attached.
- A pointwise robustness claim: L6. Top-2 at 20K, top-1 at 80K, ~20% faster than L12 to serve. Not “L6 is the best bi-encoder depth in general” — but “under this recipe, if you don’t know whether you will under- or over-train, L6 dominates L12 in expectation.”
See the report’s Related Work for how this relates to DRAGON, DPR, and Contriever.
Reproduce
git clone https://github.com/MohanKrishnaGR/bert-bi-encoder-depth-ablation.git
cd bert-bi-encoder-depth-ablation
# Experiment B (the canonical batch=32 run)
bash scripts/run_b32.sh --config config_b32.yaml
~75 GPU-hours total across all five experiments on a single H100.
What this does not claim
- Not a claim about BERT bi-encoders in general — the claim is bounded to this recipe (MNRL, 500K MS MARCO triples, top-N layer truncation) and these three BEIR tasks.
- Not a claim that 80K always hurts — it hurts zero-shot BEIR transfer from MS MARCO; in-domain MS MARCO dev was not measured.
- Not a claim about other depth-reduction strategies (bottom-drop, alternating-drop, structured pruning) — only top-N truncation was tested.
- Not a claim that the precise 80K degradation magnitudes hold across seeds — the 80K run is single-seed.
Full list in report.md → Limitations and Open Questions.