SDF-Splats: Hybrid Volumetric Gap Filling for 3D Gaussian Splatting

Abstract

1. Introduction

1.1 The Gap-Filling Problem

3D Gaussian Splatting [Kerbl et al. 2023] has revolutionized real-time radiance field rendering through its explicit representation of scenes as anisotropic 3D Gaussians. However, this explicit nature creates inherent limitations:

• • Discrete Coverage: Splats only exist where training images provided evidence
• • View-Dependent Holes: Novel viewpoints expose gaps between captured splats
• • Under-Sampling Artifacts: Thin structures and distant geometry exhibit visible discontinuities
• • Transparency Issues: Alpha-blended regions can appear disconnected

Traditional solutions include increasing splat count (memory expensive), adjusting opacity scaling (causes over-blur), or post-processing inpainting (breaks real-time performance). None address the fundamental issue: Gaussian Splatting lacks a continuous implicit representation for unsampled regions.

1.2 Why Signed Distance Fields?

Signed Distance Fields (SDFs) represent geometry as an implicit function \( f(\mathbf{p}) \) that returns the shortest distance from point \( \mathbf{p} \) to the nearest surface. SDFs provide:

• • Continuous Representation: Defined everywhere in 3D space, not just at discrete points
• • Natural Interpolation: Smooth transitions between sampled regions
• • Efficient Raymarching: Distance-guided stepping enables fast intersection finding
• • Implicit Surface Normals: Gradient of the SDF \( \nabla f(\mathbf{p}) \) yields correct normals

The challenge: traditional SDF construction requires dense volumetric grids (memory intensive) or neural networks (too slow for real-time). My insight is that the Gaussian field itself implicitly encodes an SDF through its density distribution.

1.3 Contributions

I introduce a hybrid Gaussian-SDF system with:

• 1. On-Demand SDF Construction: Build sparse local SDFs only where gaps are detected
• 2. Gaussian Field Sampling: Treat overlapping Gaussians as a continuous density field
• 3. Adaptive Raymarching: GPU-accelerated sphere tracing through the SDF
• 4. Seamless Integration: Blend raymarched fill with explicit splats maintaining lighting consistency
• 5. Zero Preprocessing: Entire pipeline operates in real-time without offline training

2. Mathematical Foundation

2.1 Gaussian Field as Density Function

A 3D Gaussian Splat \( i \) is defined by position \( \boldsymbol{\mu}_i \), covariance matrix \( \boldsymbol{\Sigma}_i \), and opacity \( \alpha_i \). The contribution at point \( \mathbf{p} \) is:

The total density field is the weighted accumulation of all nearby Gaussians:

2.2 Constructing the SDF from Density

I convert density field \( \rho(\mathbf{p}) \) to a signed distance approximation. The key insight: high density regions represent surfaces. I define an iso-surface threshold \( \tau \) and construct:

Practical Approximation: For real-time performance, I use a simplified distance metric based on Gaussian kernel radius and accumulated weights rather than full inverse covariance computation.

2.3 Sphere Tracing Algorithm

Given ray origin \( \mathbf{o} \) and direction \( \mathbf{d} \), I march along the ray using the SDF to guide step sizes:

t = t_start
for i = 1 to MAX_STEPS do
    p = o + t·d
    dist = d_SDF(p)
    
    if |dist| < EPSILON then
        return p  // Hit surface
    end if
    
    if t > t_max then
        return ∅  // Miss
    end if
    
    t = t + |dist| · safety_factor
end for
return ∅  // Max iterations
                

The safety factor (typically 0.7-0.9) prevents over-stepping near surfaces. The key advantage: step size automatically adjusts—large steps in empty space, small steps near surfaces.

2.4 Gap Detection Strategy

I employ a screen-space gap detector before invoking expensive raymarching:

This dual criterion ensures I only raymarch in genuine gaps (low alpha) that represent scene boundaries (high gradient), avoiding wasted computation in empty background or well-covered regions.

3. GPU Implementation

3.1 Spatial Acceleration Structure

Naively sampling all splats for each raymarch step is O(N·M) where N = splat count, M = raymarch steps. For real-time performance, I employ:

This reduces sampling complexity to O(K·M) where K = average splats per cell (typically 10-50). The grid is rebuilt each frame on GPU using atomic operations for dynamic scene support.

3.2 Parallel Raymarching Kernel

The fragment shader dispatches one ray per detected gap pixel:

// Per-pixel fragment shader
float4 fragColor = splatColor(screenUV);

if (isGap(fragColor.a, screenUV)) {
    // Raymarch to fill gap
    float3 rayOrigin = cameraPos;
    float3 rayDir = normalize(worldPos - cameraPos);
    
    float t = startDist;
    float3 hitColor = float3(0,0,0);
    float hitAlpha = 0;
    
    [loop]
    for (int i = 0; i < maxSteps; i++) {
        float3 p = rayOrigin + t * rayDir;
        
        // Sample Gaussian field density
        float density = sampleGaussianField(p);
        float dist = densityToSDF(density, threshold);
        
        if (abs(dist) < epsilon) {
            // Hit surface - compute color
            float3 normal = computeSDFGradient(p);
            hitColor = sampleSplatColor(p, normal);
            hitAlpha = saturate(density / threshold);
            break;
        }
        
        t += abs(dist) * 0.8;  // Safety factor
        if (t > maxDist) break;
    }
    
    // Blend raymarch result with splats
    fragColor.rgb = fragColor.rgb + hitColor * hitAlpha * (1 - fragColor.a);
    fragColor.a = fragColor.a + hitAlpha * (1 - fragColor.a);
}

return fragColor;
                

3.3 Gaussian Field Sampling

The critical function sampleGaussianField(p) accumulates contributions from nearby splats:

3.4 Normal Estimation via SDF Gradient

Once a surface hit is found, I estimate the normal using central differences on the SDF:

This provides accurate normals for lighting calculation without storing explicit surface data. The epsilon is typically 0.001-0.01 depending on scene scale.

4. Performance Analysis

4.1 Computational Cost

The raymarch overhead depends on gap coverage percentage and raymarch parameters:

Worst case: 8 steps × 50 instructions + overhead ≈ 450 instructions per gap pixel

Typical case: 3-5 steps × 20 instructions + overhead ≈ 100-150 instructions per gap pixel

4.2 Memory Footprint

4.3 Real-World Measurements

Tested on NVIDIA RTX 3080, 1920×1080, 2M splats, 15% gap coverage:

Optimization Strategies: Adaptive step count based on depth complexity, early termination on alpha saturation, checkerboard raymarching with temporal reprojection, and level-of-detail based on distance from camera.

5. Results and Comparison

5.1 Qualitative Improvements

• • Gap Elimination: Visible holes in under-sampled regions filled seamlessly
• • Smooth Transitions: SDF interpolation creates natural boundaries between splats
• • Correct Depth Ordering: Raymarched geometry respects existing splat depth
• • Lighting Consistency: SDF normals enable proper shading on filled regions

5.2 Comparison with Alternatives

5.3 Limitations

• 1. Performance Overhead: 30-60% frame time increase depending on gap coverage
• 2. Extrapolation Bounds: Cannot hallucinate geometry beyond captured regions
• 3. Thin Structure Artifacts: Very thin geometry (<2 pixel width) may exhibit temporal flickering
• 4. Transparent Surfaces: Semi-transparent gaps challenging to fill correctly
• 5. Parameter Sensitivity: Requires tuning threshold values per scene

6. Conclusion and Future Work

6.1 Summary

I presented SDF-Splats, a hybrid explicit-implicit representation that addresses the fundamental gap-filling problem in 3D Gaussian Splatting. By treating the Gaussian field as a continuous density function and constructing on-demand SDFs for raymarching, I achieve seamless reconstruction without preprocessing or neural network inference.

My method demonstrates that combining the strengths of explicit splats (detail, speed) with implicit SDFs (continuity, gap-filling) produces superior results compared to either representation alone. The system maintains real-time performance suitable for interactive applications while significantly improving visual quality in under-sampled regions.

6.2 Future Directions

• 1. Temporal Coherence: Exploit frame-to-frame similarity to reduce raymarch cost
• 2. Adaptive Resolution: Variable raymarch quality based on view distance
• 3. Neural Priors: Lightweight network to predict likely geometry in gaps
• 4. Multi-Scale SDFs: Hierarchical representation for large scenes
• 5. Transparency Handling: Volumetric integration for semi-transparent gaps

References