
Understanding and Editing Light in Images
Yixiong Yang successfully defended his PhD thesis on September 13, 2024, and he is now Doctor of Philosophy by the Universitat Autònoma de Barcelona.
What is the thesis about?
This thesis aims to develop a comprehensive solution to make image light effects both understandable and editable by leveraging intrinsic decomposition and relighting techniques. The challenges include a lack of suitable datasets and network architectures that generalize to more diverse scenarios. This thesis tackles these challenges through the exploration of three approaches.
Firstly, we propose a baseline framework for image editing that can modify the colors and lighting conditions of objects in a single image while ensuring global physical coherence. To achieve this, we introduce a synthetic dataset and a series of architectures based on intrinsic decomposition. Our framework successfully accomplishes the aforementioned editing tasks, demonstrating the effectiveness of intrinsic decomposition as a strategy. Additionally, we achieve satisfactory results on real images through fine-tuning, although these are confined to specific scenarios.
In a second approach we aim to overcome previous limitations. We further explore relighting from a single image from both the datasets and methodological perspectives. We propose two new datasets: a synthetic one with ground truth intrinsic components and a multi-light real one collected under laboratory conditions. To incorporate more physical consistency in the relighting pipeline, we establish a two-stage network based on intrinsic decomposition, providing outputs at intermediate steps and additional constraints. When the training set lacks ground truth for intrinsic components, we introduce an unsupervised module to enhance the training of the intrinsic outputs. In terms of relighting, our method outperforms state-of-the-art methods, as tested on both existing and newly developed datasets. Furthermore, we show that pre-training our and prior methods on our synthetic dataset can enhance their performance on other datasets. Nevertheless, we demonstrate the limitations of the single-image scheme that prevents perfect relighting.
To overcome these limitations, we move to use multi-view and multi-light training images for individual scenes. We propose MLI-NeRF, which integrates multiple light information in intrinsic-aware neural radiance fields. By leveraging scene information provided by different light source positions, we generate pseudo-label images for reflectance and shading to guide intrinsic image decomposition without the need for ground truth data. Our method introduces straightforward supervision for intrinsic component separation and ensures robustness across diverse scene types. We validate our approach on both synthetic and real-world datasets, outperforming existing state-of-the-art methods.
This thesis not only enhances the understanding of light effects in images but also provides robust tools for practical light editing applications. The contributions include new datasets, frameworks, and models that address key challenges in the field, paving the way for future research and applications.
Keywords: Intrinsic decomposition, Relighting, NeRF, Light effects, Computational Color, Reflectance, Shading, Deep Learning
