This PhD will explore new encodings of the SDF-grid based on a combination of factored tensors [A,B] and efficient factored representations of the radiance function, rooted in analytical Gaussian lobe decompositions of the radiance function as for [C,D]. The former have been used to model opacity fields in the context of volumetric NeRF, but seldom applied to SDF-based rendering approaches. We believe the latter can be significantly simplified to the point of eliminating most neural components to focus on physically accurate, interpretable radiance parameterizations, in a way that can be combined with spatially efficient encodings. Recent work [D] shows a promising lead to obtain high precision reconstructions with more explainable and complete surface radiance models. But their approach does not easily scale to real-world complex human data, with a very high compute overhead. Most interestingly, simplified but expressive models of skin subscattering have been proposed on the basis of dipole (2-lobe) angular Gaussian parametrizations [30], allowing to explicitly model different Fitzpatrick skin indices [F]. We thus postulate that a natural and unified Gaussian-lobe parametrization of light interaction exists and would simultaneously lead to a sparse, lightweight, relightable and differentiable representation of the scene that could complement current surface estimation algorithms, and ultimately drastically improve their performance with natural scenes containing humans. Our intuition is also that such a proposition would easily lend itself to scalable implementations able to reach millimetric detail with enhanced reconstruction performance due to better radiance model expressivity. The scalability can be achieved by using recent factored models projecting coordinates of a 3D query point on lower dimensional spaces such as planes [A,G,H], which have been recently generalized to a more versatile framework [B] with multiscale capabilities and yet very simple implementations. This multiscale capability is particularly interesting to
encode hierarchical feature sets based on sparse Gaussian lobe sets that could be combined over various spatial levels in the hierarchy for improved expressivity. Various novel research contributions will be proposed and explored on the basis of such an encoding to optimize pipeline decoding, ray-batching, ray-marching, appearance and color decoding benefiting from this new targeted combination of models.
High Performance Computing (HPC) methods will be studied to overcome the computational complexity burden.
[A] Chen, Anpei, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. "Tensorf: Tensorial radiance fields." In European conference on computer vision, pp. 333-350. Cham: Springer Nature Switzerland, 2022.
[B] Chen, Anpei, Zexiang Xu, Xinyue Wei, Siyu Tang, Hao Su, and Andreas Geiger. "Dictionary
fields: Learning a neural basis decomposition." ACM Transactions on Graphics (TOG) 42, no. 4 (2023): 1-12.
[C] Wang, Jiaping, Peiran Ren, Minmin Gong, John Snyder, and Baining Guo. "All-frequency rendering of dynamic, spatially-varying reflectance. " In ACM SIGGRAPH Asia 2009 papers, pp.1-10. 2009.
[D] Fan, Yue, Ningjing Fan, Ivan Skorokhodov, Oleg Voynov, Savva Ignatyev, Evgeny Burnaev, Peter Wonka, and Yiqun Wang. "Factored-neus: Reconstructing surfaces, illumination, and materials of possibly glossy objects. " In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 21317-21327. 2025.
[E] Donner, Craig, and Henrik Wann Jensen. "Light diffusion in multi-layered translucent materials." ACM Transactions on Graphics (ToG) 24, no. 3 (2005): 1032-1039.
[F] Fitzpatrick, Thomas B. "The validity and practicality of sun-reactive skin types I through VI." Archives of dermatology 124, no. 6 (1988): 869-871.
[G] Fridovich-Keil, Sara, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa.
"K-planes: Explicit radiance fields in space, time, and appearance. " In Proceedings of the IEEE/CVF Conference on CVPR, pp. 12479-12488. 2023.
[H] Cao, Ang, and Justin Johnson. "Hexplane: A fast representation for dynamic scenes." In
Proceedings of the IEEE/CVF Conference on CVPR, pp. 130-141. 2023.
