Plateau-Reduced Differentiable Path Tracing

CVPR 2023

Michael Fischer, Tobias Ritschel
University College London


News:

Tomáš Iser kindly provided a pure NumPy implementation which uses our method to optimize arbitrary loss functions. Give it a try here!

TLDR:

We convolute the traditional rendering equation with a Gaussian smoothing kernel to reduce plateaus, i.e., regions of zero gradient, in inverse rendering. Our algorithm improves convergence and works on problems with intricate light transport (e.g., caustics) that previous algorithms do not converge on.

Abstract

Current differentiable renderers provide light transport gradients with respect to arbitrary scene parameters. However, the mere existence of these gradients does not guarantee useful update steps in an optimization. Instead, inverse rendering might not converge due to inherent plateaus, i.e., regions of zero gradient, in the objective function. We propose to alleviate this by convolving the high-dimensional rendering function, that maps scene parameters to images, with an additional kernel that blurs the parameter space. We describe two Monte Carlo estimators to compute plateau-reduced gradients efficiently, i.e., with low variance, and show that these translate into net-gains in optimization error and runtime performance. Our approach is a straightforward extension to both black-box and differentiable renderers and enables optimization of problems with intricate light transport, such as caustics or global illumination, that existing differentiable renderers do not converge on.

Interactive Examples

Below is an interactive 1D example which uses our method to differentiate through a discontinuous step function. The task here is to move the triangle center (parameterized by theta), such that it covers the black pixel at the bottom. The plateaus in the cost landscape come from the fact that the error between the pixel's desired and its current color does not take into account how "far away" the triangle is when it's not overlapping the pixel. We can smoothen these plateaus by our proposed convolution with a Gaussian kernel (displayed in plot in the right bottom corner, click 'Show Smooth' to see the convolved function). We then sample this convoluted space and use the samples to drive a gradient descent that moves the initial parameter (green) towards the region of zero cost, i.e., such that the triangle overlaps the pixel.

Antithetic?
Show Smooth?
_______________________________________

We also provide a simple 2D example of our method in Colab. Here, we optimize a square that, in the initial configuration, does not overlap its reference and hence creates a plateau in the loss landscape (the 2D counterpart to the example above). This example uses a simpler renderer and hence does not need all the scene config / rendering infrastructure used in the main repository.


Results

In the paper, we show a variety of results on different inverse rendering tasks. One such example is included here: we optimize the position of the red sphere (cf. the video below) that is initially occluded by the blue sphere. This leads to a jump discontinuity as soon as the red sphere moves in front of the blue one (cf. the sudden change in the image loss plot) with plateaus to the left and right. As in the interactive example above, our method (blue) successfully smoothes the space and differentiates through the plateaus, whereas a rigid optimization (orange) gets stuck on the plateau.

Another example of our method shows how a smoother space is easier to optimize: the rigid optimization (the orange curve in the plot) follows the slight slope leading to a wrong minimum and finally pushing the sphere out of the image, where a plateau is hit, and the optimizer cannot recover. Our formulation (the blue curve) allows the optimization to converge, as it is operating in a smoother space.

In the below example the task is to optimize the light's position such that a reference caustic is matched. As in the shadow example, the rigid optimization operates in non-smooth space, and pushes the light source far out, where a plateau (the grey image) is reached. Our method operates in smooth space and finds the correct position.


Citation
If you find our work useful and use parts or ideas of our paper or code, please cite:

@inproceedings{fischer2023plateau,
  title={Plateau-Reduced Differentiable Path Tracing},
  author={Fischer, Michael and Ritschel, Tobias},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={4285--4294},
  year={2023}
}

Acknowledgements
We thank the Chen Liu, Valentin Deschaintre and the anonymous reviewers for their constructive comments and insightful feedback. We further thank Meta Reality Labs for supporting and enabling this research.