Depth, Seeds, and Training Budget in BERT Bi-Encoder Retrieval: An Ablation Across Three BEIR Tasks

At 20K steps the depth ordering (L3, L6, L9, L12) collapses within seed noise. At 80K steps all depths degrade on zero-shot transfer, with the degradation concentrated on fiqa (financial QA) and worst for L9/L12. L6 is the single most robust operating point across both training regimes.

Hypothesis

The working hypothesis — frozen before any training numbers existed — was that a BERT bi-encoder fine-tuned on MS MARCO would be robust to depth reduction down to ~6 layers but would show a sharp accuracy knee somewhere between L6 and L3, with L6 as the Pareto-optimal operating point. This hypothesis was partially refuted and partially confirmed: there is no knee between L6 and L3 at 20K steps, but L6 does emerge as the Pareto-optimal point once we widen the scope to include seed variance and training saturation.

Setup

The base model is bert-base-uncased (mean-pooled, via sentence-transformers), truncated so only the first N encoder blocks remain, for N ∈ {12, 9, 6, 3}. Training uses MultipleNegativesRankingLoss (in-batch negatives), AdamW at lr=2e-5, linear 10% warmup then linear decay, bf16 precision, gradient checkpointing, and 500,000 MS MARCO passage triples sampled deterministically from sentence-transformers/msmarco-hard-negatives. Every hyperparameter is identical across the four depth variants within each experiment — depth is the only within-experiment variable.

Five experiments were run. Each trains all four depths identically:

Experiment Batch Steps Seed Purpose
A 256 20K 42 Original large-batch run
B 32 20K 42 Standard batch-32 run
C 32 20K 43 Seed variance
D 32 20K 44 Seed variance
E 32 80K 42 Training saturation (4× B)

Evaluation uses BEIR dense retrieval (exact cosine search) on the full corpora of nfcorpus, scifact, and fiqa, reporting NDCG@10 (primary), MRR@10, Recall@10, and Recall@100. Latency is measured as per-query encode time (batch=1, seq=128, fp16, 50 warmup + 500 measured encodes, torch.cuda.synchronize() brackets) on an H100 GPU.

Full numerical tables for all five experiments are in RESULTS.md.

Results

Mean NDCG@10 across BEIR, all five runs

variant A: b256/20K B: b32/20K/s42 C: b32/20K/s43 D: b32/20K/s44 E: b32/80K/s42
L12 0.2551 0.2326 0.2410 0.2463 0.2143
L9 0.2655 0.2388 0.2382 0.2384 0.2109
L6 0.2709 0.2427 0.2441 0.2446 0.2254
L3 0.2708 0.2428 0.2427 0.2412 0.2203

Finding 1: At 20K steps, depth is invariant within seed noise

Aggregating the three b32/20K runs (seeds 42, 43, 44) gives the seed-averaged ladder:

variant mean σ (n=3) mean ± 2σ
L12 0.2400 0.0069 0.226 – 0.254
L9 0.2385 0.0003 0.238 – 0.239
L6 0.2438 0.0010 0.242 – 0.246
L3 0.2422 0.0009 0.240 – 0.244

Two observations matter:

L12’s seed variance (σ=0.0069) is 7–23× larger than the shallow models’. This is characteristic of undertraining: deep networks with more capacity swing more with data-order stochasticity when they haven’t converged. At 20K steps, L12 is still in the regime where seed choice materially perturbs final accuracy; L3/L6/L9 have essentially settled.

The “L6 beats L12” claim from Experiment B was a seed artifact. Per-seed gap of (L6 − L12): seed 42 = +0.0101, seed 43 = +0.0031, seed 44 = −0.0017. On seed 44, L12 beats L6. Averaged across seeds, L6 − L12 = +0.0038 — well inside L12’s ±0.007 seed band.

The corrected claim is: at 20K steps, L3, L6, L9, and L12 are statistically indistinguishable on mean BEIR NDCG@10. L9 is nominally the lowest but by a very small margin (L6 − L9 = +0.0053, ~3σ above L6’s noise but only ~0.8σ above L12’s). The earlier “L3 ≈ L6 > L9 > L12” ordering does not survive seed replication.

Finding 2: Extended training (80K steps) degrades every depth — and the degradation is concentrated on fiqa

The matched comparison is Experiment B vs Experiment E (both b32, seed=42, same data, only steps differs: 20K vs 80K):

variant 20K mean NDCG@10 80K mean NDCG@10 Δ % change
L12 0.2326 0.2143 −0.0183 −7.9%
L9 0.2388 0.2109 −0.0279 −11.7%
L6 0.2427 0.2254 −0.0173 −7.1%
L3 0.2428 0.2203 −0.0225 −9.3%

Every variant got worse with 4× more training. This falsifies the “L12 is just undertrained at 20K” reading of the earlier results. If capacity were the bottleneck, L12 should have improved at 80K. Instead it degraded.

Per-benchmark decomposition shows the degradation is not uniform:

variant nfcorpus (20K → 80K) scifact (20K → 80K) fiqa (20K → 80K)
L12 0.2195 → 0.2186 (−0.4%) 0.3644 → 0.3617 (−0.7%) 0.1139 → 0.0625 (−45.1%)
L9 0.2295 → 0.2111 (−8.0%) 0.3777 → 0.3530 (−6.5%) 0.1091 → 0.0685 (−37.2%)
L6 0.2140 → 0.2154 (+0.7%) 0.4048 → 0.3815 (−5.8%) 0.1092 → 0.0792 (−27.5%)
L3 0.2177 → 0.2185 (+0.4%) 0.3961 → 0.3593 (−9.3%) 0.1145 → 0.0830 (−27.5%)

The fiqa crash is striking: deeper models lose 37–45% of their fiqa NDCG, shallow models lose ~28%. nfcorpus is essentially flat for L6/L3 and mildly worse for L9/L12. scifact drops 5–9% across the board.

Finding 3: L6 is the most robust operating point across training budgets

Combining Findings 1 and 2: at 20K, L6 is tied with L12 and L3 (within seed noise). At 80K, L6 is the clear best (0.2254, next-best L3 at 0.2203). L6 is the only variant that is top-2 in every regime and top-1 at 80K.

Latency

Canonical ladder, from the b32/20K/seed=42 and b32/80K/seed=42 runs (which replicate to the decimal):

variant p50 (ms) p95 (ms) mean (ms)
L12 16.17 320.08 52.63
L9 15.78 316.71 51.43
L6 13.03 316.26 49.34
L3 12.65 315.58 48.59

The ladder fits p50 ≈ 12.3 ms + 0.33 × n_layers, consistent with a fixed embedding + pooling + Python-wrapper cost (~12.3 ms) plus a small per-layer attention cost (~0.33 ms) at seq_len=128 on H100.

Three of the five profiling runs (Experiments A, C, D) produced anomalous, faster, non-monotone ladders that do not match this fit. The root cause was not isolated; likely candidates are per-process CUDA kernel cache state, torch autotuner decisions taken differently after GPU warm-up, or Python timing overhead dominating at sub-10-ms encodes. Two independent runs (B and E) reproduce the canonical ladder exactly, so we treat it as the source of truth and flag the other three as unstable. The anomaly does not affect the retrieval finding because retrieval accuracy is architecture-determined, not latency-determined. See RESULTS.md for the full latency table across all five runs.

Figures

Depth × training budget: seed-averaged 20K with ±2σ band and single-seed 80K overlay

Figure 1. Mean NDCG@10 across {nfcorpus, scifact, fiqa} by encoder depth, for the seed-averaged 20K-step runs (B/C/D) with ±2σ shading and the single-seed 80K run (E) overlaid. At 20K the depth curve is flat within the shaded band; at 80K every depth has dropped, and the drop is largest at L12.

Per-benchmark NDCG@10 change from 20K seed-average to 80K

Figure 2. Change in NDCG@10 from the seed-averaged 20K baseline to the single-seed 80K run, broken out by benchmark. nfcorpus and scifact move within noise for every depth; fiqa crashes for every depth, and the crash deepens with depth (L3: −27.5%, L12: −45.1%). This is the failure mode behind the aggregate degradation in Finding 2.

Per-seed NDCG@10 at 20K shows L12's spread

Figure 3. Per-seed NDCG@10 at 20K for each depth. L12’s σ across three seeds is 0.0069, vs. σ ≤ 0.001 for L3/L6/L9 — a 7–23× gap. Deep models are unstable at 20K; shallow models have already converged. This is what drives the apparent “L3 matches L12” headline: the apparent match is an artifact of L12’s wide distribution intersecting L3’s tight one.

Pareto frontier: mean NDCG@10 vs. p50 latency

Figure 4. Pareto frontier on the canonical batch=32 latency ladder. Seed-averaged 20K points (with 2σ vertical bars) and single-seed 80K points are plotted against p50 latency; latency is x-inverted so the top-left region is the accuracy/latency sweet spot. L6 sits on the frontier at both training budgets — tied-best at 20K, best and well-separated at 80K — while running ~20% faster than L12.

Analysis

Why is depth invariant at 20K?

Two mechanisms, both likely active:

First, the bi-encoder task is fundamentally simpler than what BERT was designed for. At inference time, the model reads a query in isolation and compresses it into a single 768-dimensional vector. There is no cross-attention between query and passage, no span extraction, no entailment reasoning. This compression task may saturate at very shallow depth — the embedding layer plus three attention blocks appears sufficient to capture the lexical and shallow semantic signals that drive retrieval on these benchmarks.

Second, with MultipleNegativesRankingLoss, the gradient signal at each step is “push the query embedding toward its positive passage and away from all other passages in the batch.” This is a dense, well-defined direction in embedding space. Shallow models may respond to this signal more cleanly because the gradient travels through fewer layers and faces less vanishing or diffusion. At 20K steps, L3 is already close to saturation and L12 is not — consistent with L12’s seed variance being ~20× higher than L3’s.

Why does 80K training degrade all depths, and worse for deeper ones?

This is a textbook single-source-overfitting phenomenon, consistent with the DRAGON paper’s central finding (Lin et al., How to Train Your DRAGON) that retrieval quality is bottlenecked by training data diversity, not architecture choice within a reasonable range. Our 80K run sees 2.56M MS MARCO examples — 5.1 passes over the 500K triples, or ~8× the examples-per-parameter ratio of Contriever’s pre-training. MS MARCO is web-search Q&A; fiqa is financial QA. The 80K training pushes each depth to fit MS MARCO’s query/passage distribution more tightly, at the cost of zero-shot transfer to distributions MS MARCO under-represents (finance in particular). L12 has the most capacity to overfit, and it does — losing 45% of fiqa performance. L3 has the least capacity to overfit and preserves the most.

The narrow degradation pattern (fiqa » scifact > nfcorpus) is consistent with how far each target distribution sits from MS MARCO. nfcorpus (medical) and scifact (scientific claims) overlap with MS MARCO’s web-crawl content more than fiqa’s financial-discussion text does. Extended training shrinks the embedding space around MS MARCO’s patterns; domains close to MS MARCO are barely hurt, far domains are hurt a lot.

Why does L6 win at 80K specifically?

L6 appears to be the sweet spot on two axes simultaneously. It has enough capacity to learn the retrieval signal well at 20K (matching L12/L3 on mean NDCG@10) but not enough capacity to overfit MS MARCO aggressively at 80K (losing only 7% mean NDCG@10 vs L9’s 12%). L3 is capacity-limited on fiqa at 20K already (it matches L6 on mean but leans on nfcorpus to do so), and at 80K its smaller capacity doesn’t fully protect it either. L9 is the unambiguous loser across regimes: it trains slightly slower than L12 (more per-step compute), overfits almost as hard as L12 at 80K (−12% vs L12’s −8%), and doesn’t match L6’s robustness. Nothing in our results recommends picking L9.

Latency, and why we don’t lean on it

The canonical ladder (b32/20K/s42 and b32/80K/s42 runs) gives L12/L3 = 1.28×. That is small. The real serving-latency levers for a BERT bi-encoder at seq_len=128 on H100 are the embedding layer, the pooling, and the Python wrapper — not the attention stack. Under this deployment shape, depth reduction from 12 to 3 saves ~3.5 ms per query, which is useful for high-QPS retrieval but not transformative. The retrieval-accuracy argument for picking L6 over L12 is stronger than the latency argument, given the 80K-saturation effect.

The practical implication

Under this training recipe, on zero-shot BEIR transfer from MS MARCO:

  1. If you plan to train for ~20K steps (moderate budget): any depth from L3 to L12 works within seed noise. Pick the one that serves fastest, which is L3.
  2. If you plan to train longer (≥80K steps): pick L6. It is the single depth that is both competitive at moderate training and robust against single-source overfitting.
  3. Avoid L9. It is dominated by L6 on accuracy at every budget tested and is not meaningfully faster than L12.

Limitations

Three seeds, not many more. Seed variance σ for L12 is 0.0069, which is large relative to the inter-depth gaps. Five or more seeds would tighten the error bars further, particularly for the claim that L9 is the weakest depth.

80K single-seed. The saturation effect is observed for seed=42 only. A matched-seed 80K run at s43/s44 would confirm the degradation holds across seeds. The effect size is large enough (−7 to −12% mean NDCG) that it is very unlikely to be explained entirely by seed noise, but the claim would be tighter with replication.

Three benchmarks, not full BEIR. nfcorpus, scifact, and fiqa are three of 18 BEIR tasks, covering medical, scientific, and financial domains. The depth-invariance finding at 20K, and the fiqa-concentrated degradation at 80K, may or may not generalize to web retrieval, code search, argument retrieval, or other out-of-domain splits.

Depth-truncation strategy. Dropping the top N layers is the simplest ablation. Other strategies — dropping bottom layers, dropping alternating layers, structured pruning — could yield different results.

Latency profiling is unstable. Three of five profiling runs produced non-canonical, faster ladders. Root cause not isolated. The canonical ladder is reproduced by two independent runs, and the retrieval finding does not depend on it, but we cannot currently issue a single latency number with confidence beyond p50.

No in-domain MS MARCO evaluation. All reported numbers are zero-shot BEIR transfer. The training-saturation effect (80K < 20K on zero-shot) might look very different on MS MARCO dev, where overfitting to MS MARCO’s distribution would help.

Open Questions

  1. Does the 80K degradation replicate across seeds? Running 80K at s43/s44 would confirm the universal-degradation and fiqa-concentration findings. ~16 GPU-hours.

  2. Does L6 remain best at 40K steps, or only at 80K? A 40K run would characterize the decay curve: is L6’s robustness monotone in training length, or does it only emerge once overfitting kicks in?

  3. Does in-domain MS MARCO dev show the opposite effect? If 80K models beat 20K models on MS MARCO dev by the same magnitude they lose on fiqa, the framing becomes “specialization vs transfer” rather than “overtraining bad.”

  4. Middle-layer retention. Keeping alternating layers (e.g., 0, 2, 4, 6, 8, 10 for a 6-layer model) rather than the bottom N tests which layers carry the retrieval signal. If alternating-layer L6 matches top-drop L6, depth is truly irrelevant under this recipe.

  5. Longer sequences. Re-profiling at seq_len=256 and 512 would show whether the narrow latency gap between L3 and L12 widens when attention computation dominates over the ~12 ms fixed cost.

  6. Root-cause the latency profiling instability. Isolate whether the run-to-run bimodality comes from CUDA kernel autotuner state, per-process kernel cache, or Python-level timing noise at sub-10-ms encodes. The fix is operationally important for any depth-reduction work.

This finding is consistent with, and extends, three prior results:

What this work adds: a controlled, same-recipe, multi-seed, two-training-budget depth ablation that separates “capacity matters” from “training length matters” from “seed noise” on three BEIR tasks — an isolation that the existing literature performs only implicitly.