Results

Below we share additional qualitative results of Video Language Planning.

Generated Video and Language Plans

Effect of Using Planning with VLMs and Video Models

Real Executions

Generalization

Limitations

Generated Video and Language Plans

Below are generated long-horizon language and video plans. Note that VLP is only given the first image and a single long-horizon language goal. All language subgoals and remaining frames in the video are generated.

Effect of Using Planning with VLMs and Video Models

We next analyze the effect of combining VLM and video models together through planning to synthesize long-horizon video plans. Below, we first compare with directly generating a long horizon plan using a single task-conditioned video model. We next compare a generated video plan using VLM and video model with no planning. Finally, we compare with a generated plan with extensive planning. Extensive planning is required for effective synthesis of long-horizon videos.

Real Executions

Synthesized long horizon plans can be directly converted to policy execution using a goal-conditioned policy. We illustrate executions of different long-horizon tasks below.

Generalization

VLP decouples action prediction into visual goal generation followed by goal-conditioned policy prediction. This abstraction enables VLP to different visual inputs -- the goal-conditioned policy simply to has to focus on the relevant visual parts of visual goal.

Both text-to-video models and VLMs in VLP can be trained using a wealth of internet knowledge. This also allows VLP to generalize to new tasks without any explicit training. Below we illustrate successful execution on two tasks which VLP has not been trained on.

Limitations

VLP has several limitations. First, the physics of objects may not be accurately preserved (as illustrated by several of the synthesized video plans above) and objects may suddenly disappear and appear. Furthermore, world knowledge may not be correctly transferred to new tasks. Below, we show a video plan where internet knowledge is inaccurately transferred.

Related Projects

Check out some of our related projects below!

	Compositional Foundation Models for Hierarchical Planning We present an approach to hierarchical planning by composing a set of foundation models, trained on different aspects of internet video. We leverage iterative energy optimization as a planner that takes outputs from a language, video, and action model and synthesizes actions that are consistent across models.
	Learning Universal Policies via Text-Guided Video Generation We cast the sequential decision making problem as a text-conditioned video generation problem, where, given a text-encoded specification of a desired goal, a planner synthesizes a set of future frames depicting its planned actions in the future, and the actions will be extracted from the generated video. Our policy-as-video formulation can represent environments with different state and action spaces in a unified space of images, enabling learning and generalization across a wide range of robotic manipulation tasks.
	Is Conditional Generative Modeling all you need for Decision Making? We illustrate how conditional generative modeling is a powerful paradigm for decision-making, enabling us utilize a reward conditional model to effectively perform offline RL. We further illustrate how conditional generative modeling enables us to compose multiple different constraints and skills together.
	Composing Pretrained Models through Iterative Consensus We present a method to combine different large pretrained models together by having individual models communicate with each other through iterative consensus. We illustrate how this combination of models can do zero-shot VQA, image generation, reasoning, and image generation.

Acknowledgements

Special thanks to Tomas Lozano-Perez for helpful feedback throughout the project and Kamyar Ghasemipour for help in setting up robot experiments on Language Table setting.