NVIDIA Releases Cosmos3-H-Dreams Real-Time Surgical World Model

NVIDIA released Cosmos3-H-Dreams on October 20, 2026, a real-time, action-conditioned generative world model for robotic surgery fine-tuned on clinical cholecystectomy recordings from the CMR Surgical Versius system. The model checkpoint and companion code were published the same day.

In an NVIDIA-authored announcement on Hugging Face, the company said that, to its knowledge, Cosmos3-H-Dreams is the first real-time generative surgical simulator trained on clinical cholecystectomy recordings to support live kinematic control. Powered by Cosmos 3 and accelerated with FlashDreams, NVIDIA’s streaming-inference library, the model runs interactively on a single NVIDIA RTX PRO 6000 GPU, according to the announcement.

The model card lists release dates of October 20, 2026 on both GitHub and Hugging Face and states the model is ready for commercial or non-commercial use under the NVIDIA Open Model License. It names surgical robotics policy evaluation in simulation, interactive teleoperated world-model exploration, and synthetic data generation for robotic cholecystectomy as intended uses, aimed at surgical robotics researchers, healthcare AI developers, academic institutions, and surgical robotics companies.

Cosmos3-H-Dreams succeeds Cosmos-H-Dreams, a model built on Cosmos-Predict2.5 that targeted dVRK tabletop suturing. As a world foundation model, the earlier system learned visual dynamics directly from synchronized video and robot kinematics rather than explicitly modeling scene geometry, tissue deformation, and physical interactions. The new release moves that approach into clinical scenes, where tissue stretches under traction, reflections shift, structures become occluded, and smoke changes visibility, and where small generation errors can accumulate over long rollouts and gradually alter instrument geometry or anatomical structure.

The model builds on Cosmos3-H-Embodied-Nano, a 16-billion-parameter model trained on the Open-H-Embodiment dataset, and specializes a bidirectional teacher on curated cholecystectomy video and kinematics. Cosmos3-H-Dreams itself has approximately 15.17 billion parameters and uses a Mixture-of-Transformers architecture that pairs a Qwen3-VL-8B-Instruct reasoning tower with a latent-diffusion generation expert, operating on Wan2.2 VAE latents with action embedding layers for kinematic conditioning.

The action interface uses 28 dimensions per time step, with 14 dimensions per arm covering anchor-relative translation, 6D rotation, jaw position, electrosurgical energy activation, a thumbstick button, and thumbstick axes. The interface was extended from 20 dimensions for the Versius embodiment, with pretrained pose and jaw weights preserved and the new control inputs zero-initialized. Training uses 832 x 480 video with actions aligned at 10 Hz, a sampling rate the announcement notes is distinct from runtime generation throughput.

The bidirectional teacher was fine-tuned first on 24 future frames for 2,000 iterations and then on a 72-frame horizon for 4,400 iterations, corresponding to 2.4 and 7.2 seconds at 10 Hz. Distillation converted the 72-frame teacher, which used 35 sampling steps, into a block-causal student through causal teacher forcing over 1,000 iterations, followed by 300 iterations of Self-Forcing with distribution-matching distillation against the fixed teacher. The released student generates video autoregressively with a key/value cache and four denoising steps per latent frame; each latent frame consumes four future action vectors (0.4 seconds at 10 Hz) and produces four new video frames, enabling streaming rollout.

A second release checkpoint uses two denoising steps. NVIDIA states the two-step model is separately trained rather than the four-step model run with fewer sampling steps, and that fewer denoising steps yield faster inference at the expense of lower visual quality. Streaming runs through FlashDreams with an adapted FadeMem module that consolidates the growing key/value cache during long rollouts, complementing Self-Forcing’s handling of generated history.

Fine-tuning used proprietary CMR Surgical Versius clinical cholecystectomy recordings drawn from 4,792 episodes totaling roughly 17 million frames at 60 Hz and 1080p resolution, subsampled to 10 Hz. Surgeons teleoperated the robot during the procedures; action labels were derived automatically from robot telemetry, and instrument visibility was labeled manually.

Sample selection retained only windows in which exactly two instruments were visible and engaged for every native 60 Hz row of each history and future window, with a left Cadiere grasper and a right monopolar hook as the canonical pairing, and excluded windows with active camera commands above a fixed threshold. The training set comprised 64 recordings and about 526 unique minutes of video, with 183,528 window starts for the 24-frame horizon and roughly 436 minutes across 106,820 window starts for the 72-frame horizon used in distillation. Recordings