PaLM-E: An Embodied Multimodal Language Model
Problem
Large language models lack grounding in the real world: linking words to continuous sensor percepts is a challenge for robotics and real-world inference.
Why it matters
Real-world robotic manipulation planning and visual reasoning require language models that understand sensor input.
What the AI did
PaLM-E incorporates visual, continuous state-estimation, and textual inputs into a pre-trained large language model, trained end-to-end for robotic manipulation planning, visual question answering, and captioning.
Results
not stated in source (only qualitative claims: addresses varied embodied tasks across modalities and embodiments, with positive transfer)
Impact
Shows a single embodied multimodal model can perform diverse embodied reasoning tasks.
Record
| Venue | arXiv (Cornell University) |
|---|---|
| Published | 2023 |
| Work type | preprint |
| Authors | Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson et al. |
| Citations | 346 (OpenAlex, at import time) |
| Model or method | PaLM-E (embodied multimodal language model) |
| Organization | not stated in source |
| Primary source | https://doi.org/10.48550/arxiv.2303.03378 |
| Open access | green |
| Keywords | embodied language model, multimodal, robotics, grounding, PaLM-E |
Importance rating
significant — thousands to millions affected
Abstract claims a single embodied multimodal model addresses varied embodied reasoning tasks across modalities and embodiments with positive transfer; a genuinely new capability, but no scale or impact figures are stated.