WorldVLA

Cen (Alibaba) 2025 · Paper

One-line summary — WorldVLA unifies a vision-language-action model and a world model in one autoregressive transformer that predicts both robot action tokens and future image tokens, so learning environment physics and generating actions reinforce each other.

Problem

Robot learning pipelines have split into two camps that ignore each other’s strengths. Vision-language-action models (RT-2, OpenVLA) map observations and instructions to actions, but treat actions solely as outputs — never as inputs to be understood — so they have no mechanism for predicting the consequences of those actions. World models (GAIA-1, Cosmos) predict how the visual world evolves from observations and actions, but produce no actions themselves. WorldVLA asks whether one autoregressive model can do both: unify action and image understanding and generation in a single framework.

Method & architecture

WorldVLA defines an action (policy) model $\pi_\theta$ and a world model $f_\phi$ over image observations $o$, actions $a$, and a language instruction $l$:

$$a_t = \pi_\theta(a_t \mid o_{t-h:t},\, l), \qquad o_t = f_\phi(o_t \mid o_{t-h:t-1},\, a_{t-h:t-1})$$

and unifies both in one model $M_\psi$ with a policy head and a world head sharing all weights.

• One transformer, three tokenizers: the backbone is initialized from Chameleon, a unified image understanding/generation MLLM. The image tokenizer is a VQ-GAN (compression ratio 16, codebook 8192; 256 tokens per 256×256 image, 1024 per 512×512). The action tokenizer discretizes each of 7 action dimensions (3 relative positions, 3 relative angles, 1 gripper state) into 256 bins, so an action is 7 tokens. Text uses a BPE tokenizer whose 65,536-word vocabulary contains the 8192 image codes and 256 action codes — all modalities share one vocabulary and one autoregressive engine.
• Two interleaved data formats: action-model samples ask “what action should the robot take to {instruction}?” over $M$ observation images and supervise only the $K$ output action tokens ($\mathcal{L}_{action}$); world-model samples ask to “generate the next frame given the current image and action” and supervise only the generated image tokens ($\mathcal{L}_{world}$), repeated $N$ times. No instruction is needed for the world half, since the action fully determines the next state.
• Training objective: a mixed cross-entropy loss

$$\mathcal{L} = \mathcal{L}_{action} + \alpha\, \mathcal{L}_{world}$$

with $\alpha = 0.04$ balancing the few action tokens (7) against the many image tokens (256/1024).

• Action-chunk attention mask: naive causal attention makes later actions in a chunk depend on earlier predicted actions; since actions were absent from MLLM pretraining, generalization is weak and errors propagate. WorldVLA’s mask lets each action attend only to text and image tokens — not to prior actions — enabling parallel action generation; the world-model half keeps the standard causal mask.
• Setup: $M=2$ historical frames, chunk size $K=5$ ($K=10$ for LIBERO-Long), $N=1$.

Results

On the LIBERO benchmark (success rate over 50 rollouts per task), WorldVLA at 512×512 scores 87.6 / 96.2 / 83.4 / 60.0 on Spatial / Object / Goal / Long, average 81.8, beating the discrete-action OpenVLA baseline (76.5 average) without large-scale robot pretraining; the 256×256 variant averages 79.1. Ablations isolate the mutual enhancement: adding world-model data lifts the pure action model from 62.8 to 67.2 average SR; naive autoregressive action chunking collapses it to 54.0, and the proposed attention mask recovers 76.6 (78.1 with world data — the full model). The abstract-level summary: about +4% grasping SR over the same-backbone action model, a 10–50% SR drop from naive chunking, and a 4–23% improvement from the mask. In the other direction, the action data improves world modeling: over 50-frame rollouts FVD drops from 718.6 to 674.1 (about 10% better) and PSNR rises 23.98 to 24.30 vs a pure world model, which visibly fails at physics (objects vanishing, drawers refusing to open). Pretraining the action model with the world-model task alone also helps (62.8 to 66.8).

Why it matters for SLAM

WorldVLA marks the convergence of the two big threads of this level — world models (GAIA-1, Cosmos) and VLAs (RT-2, OpenVLA) — into a single architecture, closing the consistency gap between predicting the world and acting in it. Its ablations quantify something SLAM researchers intuit: models that must predict the geometric consequences of motion learn better representations for choosing motions, and vice versa. For SLAM, it sketches a plausible end-state for Spatial AI stacks: learned dynamics and action generation fused in one model, with explicit SLAM geometry remaining the complementary source of metric grounding that constrains and verifies the model’s imagined rollouts.

Related

• World model
• RT-2
• OpenVLA
• GAIA-1
• NVIDIA Cosmos