SimWorld is a simulation platform for developing and evaluating agents driven by large language models (LLMs) and vision-language models (VLMs) in physical and social settings. It is built on Unreal Engine 5 and split into three layers: the Unreal Engine backend supplying environments, assets, and physics; an environment layer with procedural city generation, language-driven scene editing, traffic simulation, and Gym-like APIs; and an agent layer where an agent reasons over multimodal observations and history and executes actions through a local action planner.
Worlds are generated rather than hand-built. The procedural pipeline runs in three stages — road generation, building generation, and street-element generation — and scenes can also be created, modified, and composed at run time from natural-language commands such as generating buildings to fill an empty block or replacing buildings for a consistent style. The simulation covers physical laws, lighting, weather, and time of day.
Agents observe the world through RGB, depth, and segmentation sensors, a scene graph, and global and local map information, and act through an open-vocabulary action space: natural-language commands are decomposed by a built-in action planner into sequences of low-level primitives. The action space covers pedestrians, vehicles, and robots, with actions such as moving forward, sitting down, and picking up objects. A Gym-like reset / step loop drives humanoid agents spawned into the world.
The Python client requires Python 3.10 or later and connects to a separately downloaded Unreal Engine server package. Three levels of use are supported: a Base package with two city scenes and one empty map, an optional add-on of more than 100 pre-built maps distributed as individual .pak files, and custom environments, assets, and agent models packaged as your own .pak. Running the server needs Windows or Linux, a dedicated GPU with at least 6 GB of VRAM, 32 GB of RAM, and 50–200 GB of disk depending on the package.
Features
- Procedural city generation: a modular road, building, and street-element pipeline that produces open-ended urban scenes
- Language-based world editing: create, modify, and compose scenes at run time from natural-language instructions
- Physical and social dynamics: gravity and momentum, plus lighting, weather, and time of day
- Gym-like interface: reset and step APIs returning position, orientation, and camera observations with a reward signal
- Multimodal observation: RGB, depth, and segmentation images, scene graph, and global and local maps
- Open-vocabulary actions: natural-language commands decomposed into low-level primitives by a local action planner
- Agent types: humanoids, vehicles, and robots spawned and controlled through the communicator API
- Environment packages: a Base package with two city maps and one empty map, plus an optional pack of more than 100 maps
- Customization: import your own Unreal Engine environments, assets, and agent models as .pak files
- YAML configuration: global simulator settings and per-module behaviour for city generation, traffic, asset retrieval, and agent or model options
