World Labs aims to build Simulative World Models. Meaning you can generate one, it simulates real-world behaviors, and you can interact with it.
What are they making?
They are building the RTFM model, a multimodal world modal. Here is the supposed architecture:
What data was used to train it?
The RTFM model is trained on large sets of video and image data from the internet. This may be images, and then extracting multiple viewpoints from it (front view, side view, birds view) and then extrapolating it in 3D, (possibly using depth maps, 3D scans, or game worlds)
Why did Fei Fei Li decide to build this?
To support creative world-building, simulation, robotics training, and maybe scientific and healthcare applications.
Her take is that LLMs today are limited, and just a building block. She wants to create spatial intelligence, an AI that can perceive, reason, and interact with the 3D world.
By building WorldLabs, they predict that their model will become the next frontier (which she is probably right 🐐).
The hardest part was definitely obtaining the colors from the live feed and storing them onto each generated particle. I had the following questions, and reached out to Claude for help.