A Signal-Processing Signal-Processing Framework Framework A for Inverse Inverse Rendering Rendering for Ravi Ramamoorthi Pat Hanrahan Stanford University Photorealistic Rendering Rendering Photorealistic Geometry Rendering Algorithm 70s, 80s: Splines 90s: Range Data Materials/Lighting (Texture Reflectance[BRDF] Lighting) Realistic input models required Arnold Renderer: Marcos Fajardo

80s,90s: Physically based Flowchart Flowchart Photographs Inverse Rendering Algorithm Geometric model BRDF Lighting Flowchart Flowchart Photographs Forward Rendering Algorithm

BRDF Rendering Geometric model Lighting Flowchart Flowchart Photographs Forward Rendering Algorithm BRDF Novel lighting Rendering Geometric model Assumptions Assumptions Known geometry

geometry Known Distant illumination illumination Distant Homogenous isotropic isotropic materials materials Homogenous Convex curved curved surfaces: surfaces: no no shadows, shadows, interreflection interreflection Convex Later, practical practical algorithms: algorithms: relax relax some some assumptions assumptions Later, Inverse Rendering Rendering Inverse

Known Lighting Miller and Hoffman 84 Known BRDF Unknown Marschner and Greenberg 97 Sato et al. 97 Unknown Dana et al. 99 Sato et al. 99 Debevec et al. 00 Nishino et al. 01 Marschner et al. 00

Textures are a third axis Inverse Problems: Problems: Difficulties Difficulties Inverse Surface roughness Ill-posed (ambiguous) Angular width of Light Source Contributions Contributions 1. Formalize reflection as convolution 2. Signal-processing framework 3. Analyze well-posedness of inverse problems 4. Practical algorithms Reflection as as Convolution Convolution (2D) (2D) Reflection L

i o L i 2 B( o ) 2 Reflected Light Field 2 L( i ) ( i , o ) d i Lighting BRDF B ( , o ) 2L( i ) ( i , o ) d i Reflection as as Convolution Convolution (2D) (2D) Reflection

L i o L i 2 B ( , o ) 2L( i ) ( i , o ) d i B L Spatial: integral Fourier analysis Bl , p 2 Ll l , p Frequency: product Spherical Harmonic Harmonic Analysis Analysis Spherical

2D: 2 B ( , o ) 2 L( i ) ( i , o ) d i Bl , p 2 Ll l , p 3D: 2 0 B ( , , o , o ) 2 0 L( R , [ i , i ]) ( i , i , o , o ) d i d i

Blm , pq l Llm lq , pq Insights: Signal Signal Processing Processing Insights: Signal processing framework for reflection Light is is the the signal signal Light BRDF is is the the filter filter BRDF Reflection on on aa curved curved surface surface is is convolution

convolution Reflection Insights: Signal Signal Processing Processing Insights: Signal processing framework for reflection Light is is the the signal signal Light BRDF is is the the filter filter BRDF Reflection on on aa curved curved surface surface is is convolution

convolution Reflection Filter is is Delta Delta function function :: Output Output == Signal Signal Filter Mirror BRDF BRDF :: Image Image == Lighting Lighting Mirror [Miller and and Hoffman Hoffman 84] 84] [Miller Image courtesy courtesy Paul Paul Debevec Debevec Image

Insights: Signal Signal Processing Processing Insights: Signal processing framework for reflection Light is is the the signal signal Light BRDF is is the the filter filter BRDF Reflection on on aa curved curved surface surface is is convolution convolution Reflection

Signal is is Delta Delta function function :: Output Output == Filter Filter Signal Point Light Light Source Source :: Images Images == BRDF BRDF Point [Marschner et et al. al. 00] 00] [Marschner Phong, Microfacet Microfacet Models Models Phong, Mirror Illumination estimation

ill-posed for rough surfaces Amplitude Roughness Frequency Inverse Lighting Lighting Inverse Given: B, B, find find L L Given: B Blm, pq L l Llm lq , pq 1 Blm, pq Llm l lq , pq Well-posed unless

unless denominator denominator vanishes vanishes Well-posed BRDF should should contain contain high high frequencies frequencies :: Sharp Sharp highlights highlights BRDF Diffuse reflectors reflectors low low pass pass filters: filters: Inverse Inverse lighting lighting illill Diffuse posed posed Inverse BRDF BRDF Inverse Given: B,L

B,L find find Given: lq , pq 1 Blm , pq l Llm Well-posed unless unless LLlmlm vanishes vanishes Well-posed Lighting should should have have sharp sharp features features (point (point sources, sources, edges) edges) Lighting BRDF estimation estimation ill-conditioned ill-conditioned for

for soft soft lighting lighting BRDF Directional Area source Source Same BRDF Factoring the the Light Light Field Field Factoring Given: B B find find L L and and Given: B L 4D 2D

3D More knowns knowns (4D) (4D) More than unknowns unknowns (2D/3D) (2D/3D) than Light Field Field can can be be factored factored Light Up to to global global scale scale factor factor Up Assumes reciprocity reciprocity of of BRDF

BRDF Assumes Can be be ill-conditioned ill-conditioned Can Analytic formula formula in in paper paper Analytic Practical Issues Issues Practical Incomplete sparse sparse data data (few (few photographs) photographs) Incomplete Difficult to to compute compute frequency frequency spectra spectra Difficult

Concavities: Self Self Shadowing Shadowing and and Interreflection Interreflection Concavities: Spatially varying varying BRDFs: BRDFs: Textures Textures Spatially Practical Issues Issues Practical Incomplete sparse sparse data data (few (few photographs) photographs) Incomplete Difficult to to compute compute frequency frequency spectra spectra

Difficult Concavities: Self Self Shadowing Shadowing and and Interreflection Interreflection Concavities: Spatially varying varying BRDFs: BRDFs: Textures Textures Spatially Issues can can be be addressed; addressed; can can derive derive practical practical algorithms algorithms Issues Dual spatial spatial (angular) (angular) and and frequency-space frequency-space representation

representation Dual Simple extensions extensions for for shadowing, shadowing, textures textures Simple Algorithm Validation Validation Algorithm Photograph True values Kd 0.91 Ks 0.09

1.85 0.13 Algorithm Validation Validation Algorithm Photograph Renderings Image RMS error 5% Known lighting Unknown lighting True values Kd 0.91 0.89 0.87 Ks 0.09

0.11 0.13 1.85 1.78 1.48 0.13 0.12 .14 Inverse BRDF: BRDF: Spheres Spheres Inverse Bronze Photographs Renderings

(Recovered BRDF) Delrin Paint Rough Steel Complex Geometry Geometry Complex 3 photographs of a sculpture Complex unknown illumination Geometry known Estimate microfacet BRDF and distant lighting Comparison Comparison Photograph Rendering New View, View, Lighting

Lighting New Photograph Rendering Textured Objects Objects Textured Photograph Rendering Summary Summary Reflection as convolution Signal-processing framework Formal study of inverse rendering Practical algorithms Implications and and Future Future Work Work Implications

Frequency space analysis of reflection Well-posedness of inverse problems Perception, human human vision vision Perception, Forward rendering rendering [Friday] [Friday] Forward Complex uncontrolled illumination Acknowledgements Acknowledgements Marc Levoy Levoy Marc Szymon Rusinkiewicz Rusinkiewicz Szymon Steve Marschner Marschner Steve John Parissenti, Parissenti, Jean Jean Gleason

Gleason John Scanned cat cat sculpture sculpture is is Serenity Serenity by by Sue Sue Dawes Dawes Scanned Hodgson-Reed Stanford Stanford Graduate Graduate Fellowship Fellowship Hodgson-Reed NSF ITR ITR grant grant #0085864: #0085864: Interacting Interacting with with the the Visual Visual World World NSF

Paper Website: Website: http://graphics.stanford.edu/papers/invrend http://graphics.stanford.edu/papers/invrend Paper The End End The The End End The Related Work Work Related Qualitative observation observation of of reflection reflection as as convolution: convolution: Qualitative Miller & & Hoffman Hoffman 84, 84, Greene

Greene 86, 86, Cabral Cabral et et al. al. Miller 87,99 87,99 Reflection as as frequency-space frequency-space operator: operator: DZmura DZmura 91 91 Reflection Lambertian reflection reflection is is convolution: convolution: Basri Basri Jacobs Jacobs 01 01 Lambertian Our Contributions Contributions Our

Explicitly derive derive frequency-space frequency-space convolution convolution formula formula Explicitly Formal Quantitative Quantitative Analysis Analysis in in General General 3D 3D Case Case Formal Spherical Harmonics Harmonics (3D) (3D) Spherical 0 Ylm ( , ) 1

y z xy yz 2 3z 1 zx -2 -1 0 1 x 2 . .

. 2 x y 2 2 Dual Representation Representation Dual Diffuse BRDF: BRDF: Filter Filter width width small small in in frequency frequency domain domain Diffuse Specular: Filter Filter width width small small in in spatial spatial (angular)

(angular) domain domain Specular: Practical Representation: Representation: Dual Dual angular, angular, frequency-space frequency-space Practical = B + Bd diffuse Frequency + Bs,slow Bs,fast slow specular fast specular (area sources) (directional) Angular Space

Inverse Lambertian Lambertian Inverse True Lighting Mirror Teflon Sum l=2 Sum l=4 Other Papers Papers Other Linked to from website for this paper http://graphics.stanford.edu/papers/invrend/ http://graphics.stanford.edu/papers/invrend/ Theory Flatland or or 2D

2D using using Fourier Fourier analysis analysis Flatland Lambertian: radiance radiance from from irradiance irradiance Lambertian: Application to other areas [SPIE 01] 01] [SPIE [JOSA 01] 01] [JOSA Forward Rendering Rendering (Friday) (Friday) [SIGGRAPH 01] 01]

Forward [SIGGRAPH Lighting variability variability object object recognition recognition [CVPR [CVPR 01] 01] Lighting