Giving AI Video Some Remembrance of Things Past

Back in March we looked at the difficulty generative AI video has in remembering things and places that it has seen, so that it can reproduce them a second time, and offer visual consistency.

The net effect of the problem is that you may walk through a door in an AI world model, turn round ten seconds later, and find that the door you came through is no longer there.

It’s not a trivial glitch to fix: since generative AI video is constantly inventing ‘the next thing’ it must contend with, it cannot really remember anything very far back, or anything that has expired from its narrow context window.

This means that if you explore a trained AI world model, it’s not like exploring a modeled CGI environment, where the textures and geometry are fixed, and saved as vector and bitmapped data; instead, it’s like walking through a dream of approximate probabilities – you may find the exact same thing twice, but that’s only because it was summoned up a second time from the latent space via context; not because the model actually remembers creating it or seeing it.

Thus, the average genAI video world model is similar to someone with anterograde amnesia, who has total recall prior to a cerebral event – such as a stroke or injury – but who cannot make new memories since the incident (a condition given publicity by Guy Pearce’s character suffering from it in Christopher Nolan’s Memento [2000], featured in the main article image).

In this way, video models retain a deep and profoundly-embedded understanding of many domains; but without the capability to remember anything new.

The HyDRA solution that we looked at in March was based on the use of a ‘memory bank’ of compressed visual tokens that would allow AI-generated characters to move offscreen and return in the same guise as when they were first seen.

There are a number of other methods that use this ‘breadcrumb’ approach; however, generating and leaving behind complex 3D geometry solves the problem at the expense of file size, as well as responsiveness, latency, and other potential bottlenecks.

These ‘3D-ground world models’ include the NVIDIA-led Gen3C, whose HyDRA approach uses the most typical method of resolving this particular problem – turning what it ‘sees’ into point-clouds, which are essentially stripped-down 3D geometry:

Click to play. The HyDRA approach uses a persistent 3D point-cloud cache to guide video generation, preserving scene consistency as the camera moves and previously seen areas reappear. Source

There are other similar approaches, such as Oxford University’s 2025 VMEM, and the Stanford-led SPMem collaboration from the same period:

Click to play. From the Stanford-led collaboration SPMem, which combines a persistent 3D map of static scenery with recent video frames for motion, and selected older frames for visual detail, helping the model remember places when revisiting them. Source

But all 3D geometry is heavy work, and hard on inference requirements. It’s not easy to walk round an environment and be constantly ‘building’ it as a CG artist might. To boot, any other clips or viewpoints or takes will have to be able to reference the created geometry later, which is a challenging proposition.

The latest entry to take up the ‘AI video memory’ is a new collaboration led by Tsinghua University, titled Persistent Planar Memory for Video World Models:

Click to play. From the project site for the Tsinghua-led PPMem initiative, an example of the system in action. PPMem replaces the volumetric 3D memory used by the prior project SPMem with a more compact collection of flat surfaces, updating and reusing them as the camera moves to keep previously seen environments consistent. Source

As indicated in the video segment above*, PPMem improves on the prior approaches by using a more compact representation of the 3D environment. Instead of storing millions of individual points to reconstruct previously-seen environments, the system identifies flat surfaces, such as walls, floors and furniture, and stores them as reusable geometric planes, called plane primitives; and these are continuously added to a persistent local memory as new parts of the environment are generated:

Two examples of PPMem generating video while building a persistent 3D memory. Each example begins with an input image on the left, with the camera’s planned route shown underneath. Reading the numbered frames from left to right across the first row, then the second, shows the camera exploring a shop or garden. The small colored insets reveal how the stored 3D surfaces accumulate as the journey progresses, preserving previously explored scenery. Source

When the camera returns to an earlier location, the stored planes are projected back into its field of view, guiding the generation of new frames, and reducing the amount of spatial information that must be retained and processed. This also helps to prevent familiar objects and surroundings from changing or disappearing between visits.

The use of plan