NeRF Analogies: Example-Based Visual Attribute Transfer for NeRFs
CVPR 2024
Michael Fischer1,
Zhengqin Li2,
Thu Nguyen-Phuoc2,
Aljaž Božič2,
Zhao Dong2,
Carl Marshall2,
Tobias Ritschel1
1University College London,
2Meta Reality Labs Research,
-
Paper
-
Video (coming soon)
-
Code (coming soon)
We investigate transferring the appearance from a source Neural Radiance Field (NeRF) to a target 3D geometry in a semantically meaningful and multiview-consistent way by leveraging semantic correspondence from ViT features.
Abstract:A Neural Radiance Field (NeRF) encodes the specific relation of 3D geometry and appearance of a scene. We here ask the question whether we can transfer the appearance from a source NeRF onto a target 3D geometry in a semantically meaningful way, such that the resulting new NeRF retains the target geometry but has an appearance that is an analogy to the source NeRF. To this end, we generalize classic image analogies from 2D images to NeRFs. We leverage correspondence transfer along semantic affinity that is driven by semantic features from large, pre-trained 2D image models to achieve multi-view consistent appearance transfer. Our method allows exploring the mix-and- match product space of 3D geometry and appearance. We show that our method outperforms traditional stylization-based methods and that a large majority of users prefer our method over several typical baselines.
Results:
Citation
If you find our work useful and use parts or ideas of our paper or code, please cite:
@article{fischer2024nerf,
title={NeRF Analogies: Example-Based Visual Attribute Transfer for NeRFs},
author={Fischer, Michael and Li, Zhengqin and Nguyen-Phuoc, Thu and Bozic, Aljaz and Dong, Zhao and Marshall, Carl and Ritschel, Tobias},
journal={arXiv preprint arXiv:2402.08622},
year={2024}
}
This research was conducted during an internship at Meta Reality Labs Research. We extend our gratitude to the anonymous reviewers for their insightful feedback and to Meta Reality Labs for their continuous support. Additionally, we are thankful for the support provided by the Rabin Ezra Scholarship Trust.