OpenVLA
Kim 2024 · Paper
One-line summary — OpenVLA is a 7B-parameter open-source vision-language-action model trained on 970k real-world robot demonstrations from Open X-Embodiment, matching or beating much larger closed VLAs while being efficiently fine-tunable on consumer GPUs.
Problem
Large policies pretrained on Internet-scale vision-language data plus diverse robot demonstrations promise a new way to teach robots: fine-tune a general vision-language-action (VLA) model instead of training each new behavior from scratch. But adoption was blocked on two fronts — existing VLAs (RT-2 and kin) were largely closed with limited visibility into architecture, training procedure, and data mixture, and prior work had not explored how to efficiently fine-tune VLAs for new tasks on commodity hardware, the key step for practical use. OpenVLA set out to remove both blockers.
Method & architecture
- Backbone: the Prismatic-7B VLM — (1) a ~600M-parameter dual vision encoder in which each image patch passes through both pretrained SigLIP (semantic, text-aligned) and DINOv2 (low-level spatial) and the feature vectors are concatenated channel-wise; (2) a 2-layer MLP projector into the LLM embedding space; (3) a Llama 2 7B language backbone. Prismatic itself was tuned on the LLaVA-1.5 data mixture (~1M samples).
- Actions as tokens: each dimension of an -dimensional continuous action is discretized separately into one of 256 bins, giving integers ; bin widths uniformly divide the interval between the 1st and 99th quantile of training actions (rather than RT-2’s min-max bounds, so outliers cannot destroy granularity). Since Llama’s tokenizer reserves only 100 special tokens, the 256 least-used tokens are overwritten as action tokens. Training minimizes standard next-token cross-entropy on the action tokens only.
- Training data: 970k trajectories curated from Open X-Embodiment (70+ datasets, 2M+ raw trajectories), filtered to single-arm end-effector-control manipulation with at least one third-person camera, re-weighted with Octo’s data-mixture heuristics (DROID was trialed at 10% weight but dropped for the final third of training due to persistently low action-token accuracy).
- Design decisions found empirically: fine-tuning the vision encoder is crucial for VLA performance (contrary to VLM practice); 224×224 px input matches 384×384 at 3x less compute; VLA training needs many epochs — the final run does 27 epochs until action-token accuracy exceeds 95%; fixed lr 2e-5.
- Infrastructure: trained on 64 A100s for 14 days (21,500 A100-hours, batch 2048); inference needs 15GB in bfloat16 and runs at ~6 Hz on an RTX 4090; a remote inference server streams actions to the robot.
- Efficient adaptation: LoRA learns low-rank updates on all linear layers (rank suffices), and quantized int4 serving halves memory — both studied as first-class contributions.
Results
- Out-of-the-box generalist control: across 29 tasks on WidowX (BridgeData V2, 170 rollouts/method) and the Google robot (60 rollouts/method), OpenVLA outperforms the closed 55B RT-2-X by 16.5% absolute success rate with 7x fewer parameters, beating it in all BridgeData V2 categories except semantic generalization (RT-2-X co-fine-tunes with web data; OpenVLA does not). RT-1-X and Octo trail far behind.
- Fine-tuning to new robots (Franka-Tabletop 5 Hz and Franka-DROID 15 Hz, 10-150 demos/task): fine-tuned OpenVLA achieves the highest aggregate performance and is the only method above 50% success on every tested task; it outperforms from-scratch Diffusion Policy by 20.4% on average, which retains an edge only on narrow single-instruction dexterous tasks.
- Parameter-efficient fine-tuning: LoRA r=32 reaches 68.2 ± 7.5% vs full fine-tuning’s 69.7 ± 7.2% while training just 1.4% of parameters (97.6M vs 7,188M) — 10-15 h on a single A100, an 8x compute reduction.
- Quantized inference: int4 scores 71.9 ± 4.7% vs bfloat16’s 71.3 ± 4.8% at 7.0GB vs 16.8GB VRAM; int8 drops to 58.1% because its 1.2 Hz inference breaks the 5 Hz controller dynamics.
- All model checkpoints, fine-tuning notebooks, and the PyTorch training codebase (FSDP, FlashAttention, HuggingFace integration) are released.
Why it matters for SLAM
OpenVLA is the reference open VLA and the base model for a large body of follow-on robot-learning work, showing that a 7B open model can outperform far larger closed systems. For the SLAM community it matters in two ways: it validates SigLIP/DINOv2-style foundation features as robot perception backbones, and it defines the action side of the emerging autonomy stack — SLAM supplies localization and metric maps while a VLA closes the loop from perception and language to physical action.