Part III

LeWM architecture wall

What you should noticeTrace the boxes: pixels → ViT encoder → z → predictor(z, a) → SIGReg; planning searches actions in latent space.

LeWorldModel is an end-to-end JEPA from pixels: ViT encoder, action-conditioned predictor, SIGReg, CEM planning in latent space. No stop-gradient, EMA teacher, pretrained encoder, or reconstruction loss.

Trace the stack below, then return to the toys whenever a box feels abstract.

pixels / observation

ViT encoder → latent z (from [CLS] + projection)

predictor ẑ(t+1) = pred(z(t), a(t))

loss = prediction MSE + λ · SIGReg (anti-collapse)

planning: CEM over action sequences in latent space