AI Model Transforms Live Video in Real Time

The image shows a scenic view of Central Park in New York City, featuring the iconic Gapstow Bridge with skyscrapers in the background. The stone bridge arches over a small pond, surrounded by lush green vegetation including tall reeds and trees in full foliage. The urban landscape behind shows several tall buildings including an Art Deco style skyscraper with an American flag on top. The scene captures the unique contrast between nature and urban architecture that Central Park is famous for. The left side of the image appears to have a pixelated or mosaic effect.

The Gapstow Bridge set against lush summer greens and the skyscrapers of midtown Manhattan.

Introduction to Mirage

Decart, a startup, has developed an AI model called Mirage that can transform live footage in real time. The model is both an impressive feat of engineering and a sign of how AI might soon shake up the livestreaming industry.

Dean Leitersdorf, the founder of Decart, demonstrates the model by feeding it a prompt and transforming into a bizarre, gold-tinged, subaquatic version of Julius Caesar in a poncho. The model tries to predict what each new frame should look like, resulting in surreal and mind-bending scenes.

Technical Details

Decart’s video-to-video model, Mirage, is a significant achievement in engineering. The model generates 20 frames per second at 768 × 432 resolution and a latency of 100 milliseconds per frame, making it suitable for decent-quality TikTok clips.

Manipulating live scenes in real time is computationally taxing, and Decart had to write low-level code to squeeze high-speed calculations out of Nvidia chips to achieve the feat. The company also developed a custom scheme for training and running the model to achieve greater coherence and devised a way for the model to correct errors at speed.

Potential Applications

Decart’s technology has big potential for gaming. The company demoed a game called Oasis that used a similar approach to Mirage to generate a playable Minecraft-like world on the fly. Users could move close to a texture and then zoom out again to produce new playable scenes inside the game.

The model also has the potential to become popular on platforms like TikTok or Instagram, allowing users to create weird and wonderful scenes. However, its unpredictability might prove controversial, and the company will need to address issues such as the model’s tendency to change a user’s race.