FVGen: Accelerating Novel-View Synthesis with Adversarial Video Diffusion Distillation

Wenbin Teng1,2, Gonglin Chen1,2, Haiwei Chen1,2, Yajie Zhao1,2,
1Institute for Creative Technologies 2University of Southern California
ICCV 2025
Paper(Coming Soon) Code (Coming Soon)

Figure 1. We present FVGen, a method of fast novel-view synthesis for 3D scene reconstruction with sparse inputs. Recent method unleash the generation capabilities of video diffusion models conditioned on a prior 3D cue for dense view creation, but always suffers from a long sampling time. We propose a novel framework that can significantly reduce the sampling time and maintain the generation quality.

Abstract

Recent progress in 3D reconstruction has enabled realistic 3D models from dense image captures, yet challenges persist with sparse views, often leading to artifacts in unseen areas. Recent works leverage Video Diffusion Models (VDMs) to generate dense observations, filling the gaps when only sparse views are available for 3D reconstruction tasks. A significant limitation of these methods is their slow sampling speed when using VDMs. In this paper, we present FVGen, a novel framework that addresses this challenge by enabling fast novel view synthesis using VDMs in as few as four sampling steps. We propose a novel video diffusion model distillation method that distills a multi-step denoising teacher model into a few-step denoising student model using Generative Adversarial Networks (GANs) and softened reverse KL-divergence minimization. Extensive experiments on real-world datasets show that, compared to previous works, our framework generates the same number of novel views with similar (or even better) visual quality while reducing sampling time by more than 90%. FVGen significantly improves time efficiency for downstream reconstruction tasks, particularly when working with sparse input views (more than 2) where pre-trained VDMs need to be run multiple times to achieve better spatial coverage.

Results

Architecture

We initialize our student model by training a GAN objective. The student model $G_\theta$, initialized with the weights of teacher model, uses few-step denoising to generate fake videos to fool the discriminator $D$. The fixed teacher model, together with a trainable 3D CNN, serves as the discriminator to differentiate between fake and real videos.

Acknowledgements

Supported by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/ Interior Business Center (DOI/IBC) contract number 140D0423C0075. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DOI/IBC, or the U.S. Government.