Bad Batch - An Experiment in Gamified Gaussian Splatting

To celebrate Easter 2024, Resn launched "Bad Batch," a psychedelic 3D Easter Egg Hunt made with Gaussian Splatting and high-end 3D game rendering to create an immersive gaming experience.

The Opportunity

While exploring Gaussian Splatting, we saw the potential to create fun interactive 3D environments. This inspired us to design an Easter Egg hunt that embraced the characteristic glitches and distortions of Gaussian Splatting.

What is Gaussian Splatting?

Gaussian Splatting is a technique that recreates 3D scenes from 2D photos, allowing real-time exploration in browsers. It represents a scene as a 3D point cloud of thousands of blurry elliptical particles (splats) with different sizes, orientations and colours. 

To generate the point cloud, hundreds of images captured from different angles are processed to train the data. The training process adjusts the particles to ensure the 3D scene remains faithful to the original from a wide range of angles, including lighting, shadows and surface reflections.

The Solution

Since Gaussian Splatting has the potential to quickly bring high-end 3D graphics and complex lighting into the browser with minimal computational power, we thought it was an ideal technology for a 3D game. And with Easter coming up, an egg hunt seemed like the perfect fit. 

Unreal Engine 5

We started by creating a 3D scene of a spherical treasure trove in Unreal Engine 5. The game engine allowed us to make iterative refinements to the scene as needed, which would have been much harder in a real-world setup.

We added realistic lighting, reflections, and shadows to the scene using ray tracing (supported in Unreal Engine for RTX graphics cards).

To ensure we had the necessary camera angles for the final scene, we developed a bespoke tool that captured and rendered 256 images of the model from different vantage points. 

Generating and Editing Splats

With a tight project timeline, we used Luma AI to generate high-quality Gaussian Splats from our 3D renders. Although other tools offer more fine-tuning, Luma AI provides excellent results with minimal configuration, enabling rapid content iteration.

To remove extraneous elements from the splats, we used PlayCanvas 's SuperSplat tool. This allowed us to trim out the background and unnecessary parts of the scene (although we kept a few of the misplaced chunks and holes in the mesh because we liked how it looked).

Rendering Splats in Three.js

Using Mark Kellogg 's Gaussian Splatting for Three.js, we loaded and rendered our custom-edited splats in the browser, allowing integration of our own view controls, effects, and interaction logic.

We added clickable eggs using spheres placed at specific locations. We developed a custom utility to help position the eggs and to add collision detection logic to ensure the eggs couldn't be clicked when obscured by other objects.

Custom Shaders

We used custom vertex shader instructions in the Gaussian Splat renderer to create a trippy, distorted experience. The shaders manipulated each splat particle's position, scale, colour, and opacity based on mathematical equations tied to their original 3D locations. These waves of rippling colour created a psychedelic pulsing and warping effect. Amplifying the waves caused severe distortions, occasionally ripping apart the scene.

When a user clicks on an egg, the equations create a ripple effect and brighten the colours into a glowing shockwave for a satisfying response.

Although users are essentially just rotating a sphere, we wanted to give the experience a more expansive feeling. To do this, we added a fish-eye lens distortion by transforming particles toward the edges of the screen. This made the space appear to curve around the viewer, enhancing the psychedelic atmosphere.

The Future of Gamified Gaussian Splatting

Gamified Gaussian Splatting holds enormous potential for creating immersive 3D experiences for online brand activations, promotional websites, product explorations, museums and art galleries, and experiential marketing for tourism and entertainment.

We're excited about this technology's creative potential and can’t wait to explore it further. Future possibilities include animated Gaussian Splats for dynamic scenes, generative splats using AI, and 3D models from text and image inputs.

Curious about how this technology could be used in your next project? Contact our team at newbusiness@resn.co.nz