ReWorld: An Interactive World Model with Long-Horizon Memory
ReWorld resolves the structural tension between short-horizon control and unbounded memory by separating them during training and bounding them at inference via a pose-indexed landmark bank.
The model addresses the conflict where control mechanisms require short horizons while memory demands unbounded context. During training, ReWorld employs mixed per-head attention windows that confine most heads to the recent past while a small set of global heads attends over the entire history. Random head routing prevents either capability from binding to specific heads, and random chunk dropping ensures sparse histories remain in-distribution. At inference, the system operates under a fixed budget using a bounded KV cache backed by a pose-indexed landmark bank, retrieving only the landmarks nearest the current pose rather than maintaining full history.
Data alignment is achieved through a metric-scale-aligned engine that normalizes eight distinct sources, including Unreal-rendered fly-throughs, game roaming, and real-world footage, onto a single physical action scale. This ensures identical key presses move the camera the same distance across all domains. Palindrome trajectories provide the necessary revisit evidence for memory training. The architecture utilizes distribution-matching distillation confined to a LoRA adapter, compressing sampling to four steps. A single backbone supports both high-fidelity multi-step generation and real-time interactive streaming at 704x1280 resolution across photorealistic, game-style, and stylized environments.
Evaluation under a three-axis protocol covering action following, long-horizon recall, and video quality shows ReWorld outperforming six recent interactive world models. It achieves the best control fidelity with an 11.95° rotation error and superior camera-motion consistency. In minute-long out-and-back rollouts lasting 64 seconds with 384 latents, the fixed 12-chunk cache successfully regenerates the starting view. This performance persists at rollout lengths where sliding window approaches have evicted critical evidence and full-KV attention methods exhaust available memory.
For practitioners building interactive systems, the separation of training and inference memory strategies offers a path to long-horizon consistency without prohibitive compute costs. The use of a pose-indexed landmark bank allows for deterministic retrieval of historical context within a strict memory budget, enabling deployment on hardware that cannot support full-key-value caching for extended sequences.