After a full checkpoint load each shard rank persists its local DTensor
chunks (post-rename, post-cast) to FASTVIDEO_WEIGHT_SHARD_CACHE (tmpfs);
identical (checkpoint, mesh, dtypes, name-mapping) launches rebuild via
DTensor.from_local in ~2-4 s per 33B model instead of ~10 min of cold-NFS
reads. DMD2's three roles share one entry; params absent from the cache
(gate_compress/proj_l) zero-init exactly like the full-load path. Any
validation failure or exception degrades to the normal full load.
FASTVIDEO_WEIGHT_SHARD_CACHE_PER_NODE=1 makes every HSDP replica write its
own node-local copy on multi-node runs (single-writer semantics would leave
non-zero replicas' nodes permanently cold). HITs refresh entry mtimes so
mtime-based tmpfs cleaners and the LRU GC spare active entries.
Also: distribute_tensor(src_data_rank=None) in the full-load path so every
rank feeds its own shard instead of scattering from rank 0.
5 CPU contract tests in fastvideo/tests/loader/test_shard_cache.py.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
