On Considering Semantics for Multi-image Processing

Upcoming PhD defence

Danna Xue will defend her PhD thesis on July 11, 2024 3 pm.

What is the thesis about?

In multi-image processing, leveraging semantic information is essential for content-aware operations and ensuring consistency across images. However, this presents challenges in obtaining high-precision semantic data quickly, tailoring semantic information to different tasks, and maintaining consistency across processing results. This thesis addresses these challenges through several proposed approaches:

Slimmable semantic segmentation: We introduce a flexible framework for training semantic segmentation models with knowledge distillation, enabling quick adaptation between accuracy and efficiency trade-offs. To further improve the accuracy of the compact models, boundary supervision is introduced to obtain better object boundary details.

Semantic integration in recoloring: We explore the integration of semantic features into palette-based image recoloring to enhance color consistency across multiple images. Moreover, we propose to introduce color naming features in color harmonization. We demonstrate that the integration of semantics improves image color consistency and harmony, producing better perceptual visual effects.

Temporal impact analysis: We investigate the impact of temporal information on multi-image restoration quality, highlighting the perception-distortion tradeoff and the importance of alignment. We demonstrate that the perception-distortion tradeoff still exists when introducing temporal information, and misalignment worsens both perception and distortion. Our analysis provides a reference for designing multi-frame restoration algorithms and potential shooting strategies.

Each approach contributes to overcoming the challenges of leveraging semantic information in multi-image processing, aiming to enhance both efficiency and effectiveness in various image processing applications.

Keywords: deep learning, semantic segmentation, image recoloring, image restoration, multi-image processing, perception-distortion tradeoff.