Gaussians as Supervision for Joint Physical Parameter Estimation and Appearance Reconstruction of Elastic Objects

Abstract

Recovering the appearance and physical parameters of elastic objects from multi-view video is essential for many applications that require simulation of the real world. Past methods for this task have provided accurate results in recovering physical properties; however, their re- liance on Neural Radiance Fields (NeRFs) for novel view synthesis means they trade off visual quality for rendering speed. To address this issue, we present a novel framework for the joint appearance reconstruction and physical parameter estimation of elastic objects relying on 3D Gaussian splatting. Our key insight is that dynamic 3D Gaussian kernels extracted from multi-view video can be used to reconstruct the object’s geometry and supervise elastic parame- ter fitting through a differentiable physics engine. Novel views and object behaviours can then be constructed by forward simulating the extracted mesh and using it to drive the Gaussian kernels. We demon- strate that our method is competitive with the state of the art on phys- ical parameter estimation while being better at reconstructing object appearance. Additionally, our method can simulate novel views and object interactions at near real-time rates that outperform past ap- proaches.