FreGS: 3D Gaussian Splatting with Progressive Frequency Regularization
CVPR 2024

Abstract

3D Gaussian splatting has achieved very impressive performance in real-time novel view synthesis. However, it often suffers from over-reconstruction during Gaussian densification where high-variance image regions are covered by a few large Gaussians only, leading to blur and artifacts in the rendered images. We design a progressive frequency regularization (FreGS) technique to tackle the over-reconstruction issue within the frequency space. Specifically, FreGS performs coarse-to-fine Gaussian densification by exploiting low-to-high frequency components that can be easily extracted with low-pass and high-pass filters in the Fourier space. By minimizing the discrepancy between the frequency spectrum of the rendered image and the corresponding ground truth, it achieves high-quality Gaussian densification and alleviates the over-reconstruction of Gaussian splatting effectively. Experiments over multiple widely adopted benchmarks (e.g., Mip-NeRF360, Tanks-and-Temples and Deep Blending) show that FreGS achieves superior novel view synthesis and outperforms the state-of-the-art consistently.

FreGS

Overview of the proposed FreGS. 3D Gaussians are initialized by structure-from-motion. After splatting 3D Gaussians, we can obtain 2D Gaussians and then leverage standard α-blending for rendering. Frequency spectra Fˆ and F are generated by applying Fourier transform to rendered image Iˆ and ground truth I , respectively. Frequency regularization is achieved by regularizing differences of amplitude ∣F (u, v)∣ and phase ∠F (u, v) in Fourier space. A novel frequency annealing technique is designed to achieve progressive frequency regularization. With low-pass filter H_l and dynamic high-pass filter H_h, low-to-high frequency components are progressively leveraged to perform coarse-to-fine Gaussian densification. Note, the progressive frequency regularization is complementary to the pixel-wise loss between Iˆ and I. The red dashed line highlights the regularization process for Gaussian densification.

Visual Comparisons