Beyond Neural Networks: How Gaussian Splatting Challenged 3D Reconstruction Orthodoxy

The story of 3D reconstruction follows a familiar pattern: neural networks arrive, everything changes overnight, and the field never looks back. But sometimes, the most effective solutions emerge by questioning what everyone assumes comes next.

For decades, 3D reconstruction lived in the realm of geometric algorithms. Photogrammetry and Structure from Motion (SfM) dominated the landscape, relying on feature matching, camera calibration, and triangulation techniques refined since the 1900s. These methods worked reasonably well but hit fundamental walls: they struggled with textureless surfaces, reflective materials, and required extensive manual cleanup.

Then 2020 arrived with Neural Radiance Fields (NeRF), and everything changed.

The neural revolution and its costs

NeRF represented the kind of paradigm shift that computer vision has seen repeatedly: neural networks completely redefining what's possible. Instead of reconstructing explicit 3D geometry, NeRF learned to represent entire scenes as continuous functions that could synthesize photorealistic views from any angle. The results were stunning—transparent objects, complex lighting, and intricate details that had defeated classical methods for years.

The computer vision community did what it always does with breakthrough neural methods: everyone jumped in. Within months, researchers published dozens of NeRF variants, each pushing the boundaries further. The assumption was clear—this neural approach was the future, and progress would come through bigger networks, better architectures, and more sophisticated training.

But NeRF came with serious practical limitations that became harder to ignore over time. Training took 48+ hours per scene. Inference was painfully slow at roughly 0.1 frames per second. The learned representations were black boxes—impossible to edit or modify after training. Dynamic scenes remained largely unsolved. For all its visual quality, NeRF felt more like an impressive research demo than a practical tool.

The expected path forward

In any other neural success story, the roadmap would be predictable: build bigger networks, use more powerful hardware, develop more sophisticated training methods. We'd see NeRF-XL with billions of parameters, specialized neural architectures, distributed training systems, and increasingly complex optimization techniques.

Some researchers pursued exactly this path. FastNeRF, Instant-NGP, and others focused on acceleration through neural network innovations. The underlying assumption remained unchanged—neural networks were the future, and the solution was better neural networks.

A different kind of innovation

But in August 2023, researchers at INRIA published something unexpected. 3D Gaussian Splatting achieved real-time rendering speeds of 30-100+ frames per second while maintaining photorealistic quality—not through better neural networks, but by stepping away from neural networks entirely.

The key insight was counterintuitive: instead of improving the neural representation, they replaced it with something more direct. Gaussian Splatting represents scenes using millions of explicit 3D Gaussian distributions—mathematical primitives that store position, orientation, size, opacity, and color information directly.

This wasn't entirely novel in isolation. Gaussian functions have deep roots in computer graphics: Gaussian blur has been fundamental to image processing since the 1980s, Gaussian mixture models have long been used for point cloud representation, and volume rendering has employed Gaussian splatting techniques for decades. The innovation lay in combining these classical primitives with modern differentiable optimization—training millions of Gaussians through gradient descent, just like neural networks, but without the neural network.

The technical breakthrough

The elegance lies in the hybrid approach. Gaussian Splatting starts with traditional Structure from Motion to get a sparse point cloud, then converts each point into a 3D Gaussian primitive. During training, the system optimizes these Gaussians' parameters through backpropagation—familiar neural network concepts applied to explicit representations.

For rendering, the method leverages decades of GPU rasterization optimization rather than the ray-tracing approaches that made NeRF slow. The system projects 3D Gaussians onto the 2D image plane using efficient rasterization, sorts by depth, and performs α-blending—all techniques that GPUs handle exceptionally well.

Training takes 35-45 minutes instead of 48+ hours. The explicit representation means artists can directly edit, move, and manipulate scene elements—something impossible with neural implicit representations. Integration with traditional 3D graphics pipelines becomes natural rather than requiring specialized neural rendering infrastructure.

The trade-offs

This approach isn't without costs. Scene files are substantially larger—ranging from 0.6GB to 1.4GB compared to NeRF's compact neural networks of tens of megabytes. The method can struggle with aliasing artifacts and requires high-quality input images for optimal results.

Dynamic scenes remain challenging, though recent extensions like 4D Gaussian Splatting are addressing these limitations. The technique shows difficulties with highly reflective or transparent materials, areas where NeRF's volumetric approach naturally excelled.

Current limitations

Despite its advantages, Gaussian Splatting faces several key constraints:

1. Dependence on initial point cloud quality: The method requires a sparse point cloud from Structure from Motion (SfM) as its starting point. While this initial reconstruction doesn't need to be highly accurate or dense, poor camera calibration or insufficient feature matching can still impact the final results.

2. Dynamic scene challenges: The original approach struggles with moving objects or changing scenes over time. Recent extensions like 4D Gaussian Splatting (Wu et al., CVPR 2024) and temporal modeling techniques are addressing these limitations, but dynamic reconstruction remains more complex than static scenes.

3. Large file sizes: Scene representations range from 0.6GB to 1.4GB compared to NeRF's compact neural networks of tens of megabytes. Compression research by Papantonakis et al. and concurrent work on lightweight encodings are achieving 15-20x size reductions, but storage remains a consideration for deployment.

   
   

What this tells us about innovation

The success of Gaussian Splatting teaches us that the path of scientific and technological progress is always surprising and unpredictable. The enemy of advancement is fixation on concepts, even if they are the most innovative and successful concepts. There's tremendous value in revisiting clever ideas, even seemingly outdated ones—they can propel us forward in unexpected ways.

This pattern extends beyond computer graphics. Sometimes the most effective solutions emerge not from doubling down on the latest paradigm, but from stepping back and asking whether we're using the right fundamental approach. Gaussian Splatting reminds us that innovation often lies in the creative recombination of existing knowledge rather than always requiring entirely new concepts.

Free-form fly-through around the scene reconstructed using Gaussian Splatting

I'll leave you to ponder this against the backdrop of this beautiful 3D scene we built using Gaussian Splatting with very little effort—achieving a quality that could not have been accomplished with traditional SfM techniques. The video shows real-time rendering of an island in the Philippines captured from a drone footage, where one can see a free-form fly-through around the island, all rendered at real-time speeds that demonstrate the practical power of this approach.