Distractor-Free Novel View Synthesis via Exploiting Memorization Effect in Optimization

Overview

Neural Radiance Fields and 3D Gaussian Splatting assume a static scene, but real captures often contain moving people, transient objects, shadows, and changing illumination. MemE models residual distributions during optimization to separate scene purity from distractors. It works without external segmentation models or manual annotations and can be applied to both NeRF and 3DGS.

Method

MemE combines hierarchical residual measurement, purity-distractor mixture modeling, and purity attention generation. The generated attention mask prevents distractors from contributing to the reconstruction loss.

Qualitative results

MemE removes view-dependent distractors while retaining static-scene details on real and synthetic benchmarks.

Demo video

Results on PhotoTourism scenes, comparing the original 3D Gaussian Splatting reconstruction with MemE-3DGS. MemE reduces transient distractors while preserving the static scene.

Download video (MP4, 18 MB)

Selected checkpoints

Selected 250k-step models. The values are mean color-corrected evaluation metrics; higher PSNR/SSIM and lower LPIPS indicate better quality. Each checkpoint is linked to the exact benchmark and scene that produced these results.

Code

The implementation is based on MultiNeRF and uses JAX. MemE is implemented in internal/gmmsoftmask.py, with experiment settings in configs/.

git clone https://github.com/Yukun66/MemE.git
cd MemE
conda create -n meme python=3.9
conda activate meme
pip install -r requirements.txt
git clone https://github.com/rmbrualla/pycolmap.git internal/pycolmap

Training

bash scripts/train_meme.sh /path/to/scene /path/to/checkpoints

Evaluation

bash scripts/eval_meme.sh /path/to/scene /path/to/checkpoints

Citation

@inproceedings{wang2024meme,
  title     = {Distractor-Free Novel View Synthesis via Exploiting Memorization Effect in Optimization},
  author    = {Wang, Yukun and Li, Kunhong and Chen, Minglin and Wang, Longguang and Zhou, Shunbo and Xue, Kaiwen and Guo, Yulan},
  booktitle = {European Conference on Computer Vision},
  pages     = {477--493},
  year      = {2024}
}