[misc]: add dated resume handoff for the BF16 MFU campaign

2026-07-28 02:50:37 -07:00
parent 9c86d28e11
commit c2becf42c2
1 changed files with 9 additions and 0 deletions
@@ -23,6 +23,15 @@ Historical 1x W&B runs `5h2cy7m9`, `zj6wqtpg`, `nzh2k20t`, and `gbjiw7og` report

 ## Resume here

+Dated handoff (2026-07-22 session, head `9c86d28e1`) — the queued gates, in priority order. All of them are blocked on the same prerequisite: a **healthy 2,062 MHz-class allocation** (check `nvidia-smi --query-gpu=clocks.max.sm` first; that day the gb-nvl-118 rack was the healthy bin while twelve rolls across racks 053/057/059 all drew 1,965 MHz trays, which are valid for ratio microgates but baseline-ineligible and too tuning-unstable for +-1% trainer gates).
+
+- **Env stack -> recipe.** A/X/B `run_current.sh` at 4x/B2 with `TORCHINDUCTOR_COORDINATE_DESCENT_TUNING=1 CUBLASLT_WORKSPACE_SIZE=76800` (accepted -1.083% / +0.432 pp on allocation `1631376`; neutral on a degraded 8x pair). If it reproduces, fold the two envs into `run_current.sh` and refresh the stopping-point rows (also 8x/B3 on a healthy pair).
+- **Pinned cuBLASLt capture verdict.** Gate [`harness/benchmark_fastvideo_train_lt_pinned.py`](harness/benchmark_fastvideo_train_lt_pinned.py) (traced custom-op variant at head; the graph-breaking variant that banked +0.234 pp is at `8bf6eff18`). The sweep margin is -8/-9.4% band, bit-exact; stable tuning on healthy hardware should decide between the variants and size the real capture (+1.9-2.2 pp potential). Production path if it wins: Inductor-level lowering or a `ReplicatedLinear` quant-method hook that keeps dbias fused.
+- **Two-tray launches need the preamble** recorded in the REPORT campaign section: `GLOO_SOCKET_IFNAME=enP5p9s0 NCCL_SOCKET_IFNAME=enP5p9s0`, a fresh `MASTER_PORT`, `MASTER_ADDR` resolved to a raw IP. Rack gb-nvl-057 pairs hang NCCL MNNVL/IMEX at first collective (`NCCL_MNNVL_ENABLE=0` is diagnosis-only); rack-059 pairs passed health at 458.9 GB/s.
+- **Kernel plan** in [`reports/bf16_kernel_research_plan.md`](reports/bf16_kernel_research_plan.md) carries per-item statuses: FA4 forward anomaly isolated standalone (tail-tile padding killed; remaining path is upstream CuTe forward schedule work), launch-gap graphs untouched. The feasibility budget fixes the target: 50% needs combined GEMM+FA4 throughput 58.9-64.7% vs 56.7% measured.
+
+The general procedure:
+
 1. Use a healthy GB200 allocation and record source SHA, GPU count/topology, clocks, power limit, throttle state, CUDA/PyTorch/FA4 versions, and W&B/job IDs. Run [`probes/pr1630_realloc1629519_native_health.py`](probes/pr1630_realloc1629519_native_health.py) and the topology/NCCL probes before accepting an 8x result.
 2. Run [`runners/run_current.sh`](runners/run_current.sh). For the final configurations set `LOCAL_BATCH_SIZE=2` on 4x or `LOCAL_BATCH_SIZE=3` on 8x. This timing harness intentionally uses a dummy tracker; its structured `BF16_*` records, not W&B, are the MFU source of truth. Use [`runners/run_observed.sh`](runners/run_observed.sh) for an observable 4x training/validation run with the same production contract.
 3. Attribute changes only with an A/candidate/B sandwich on one allocation. Use the slowest rank per step, discard 10 warmup steps, and report the 20-step median plus peak allocated/reserved memory. Never attribute a cross-allocation delta.