Deep Learning for Smartphone ISP

Last updated on Jun 22, 2021

Motivation

As the popularity of mobile photography is growing constantly, lots of efforts are being invested now into building complex hand-crafted camera ISP systems. Despite the advances in this field, the modern smartphone cameras are still unable to reach the level of the high-end DSLR cameras, partly because the conventional ISP algorithms are unable to accurately recover the information coming from the tiny mobile camera sensors or render the missing image details. To address this problem, we organize this challenge and present a large-scale dataset consisting of aligned smartphone raw / DSLR image pairs to train deep learning-based ISPs.

Challenges

The challenge aims to obtain an output image with the best trade-off between pixel fidelity (PSNR) and latency on mobile devices.

Our Approaches

The whole pipeline of Learned Smartphone ISP has two main steps (assume the input resolution is H x W):

deBayer pre-processing:
- Input: RAW data [H x W x 1]
- Output: deBayer RAW data [(H/2) x (W/2) x 4]
We propose an UNet-like architecture, PUNET, to take deBayer data and output RGB images:
- Input: deBayer RAW data [(H/2) x (W/2) x 4]
- Output: RGB image [H x W x 3]

A detailed description of the entire baseline training pipeline can be found in our GitHub: https://github.com/MediaTek-NeuroPilot/mai21-learned-smartphone-isp.

Dataset

(Will be released. Stay tuned.)

More information of the dataset can be found in our GitHub: https://github.com/MediaTek-NeuroPilot/mai21-learned-smartphone-isp.

Results

We evaluate the pre-trained PUNET on the validation data (resolution: 256x256), and obtain the following results:

PSNR: 23.03
Some visualized comparison with the ground truth:
Baseline results visualization.

Resources

Papers & Code

BibTex

@misc{mtk2021mai,
  title={Mobile AI Workshop: Learned Smartphone ISP Challenge},
  year={2021},
  url={https://github.com/MediaTek-NeuroPilot/mai21-learned-smartphone-isp}
}

@inproceedings{ignatov2021learned,
  title={Learned Smartphone ISP on Mobile NPUs with Deep Learning, Mobile AI 2021 Challenge: Report},
  author={Ignatov, Andrey and Chiang, Cheng-Ming and Kuo, Hsien-Kai and Sycheva, Anastasia and Timofte, Radu and Chen, Min-Hung and Lee, Man-Yu and Xu, Yu-Syuan and Tseng, Yu and Xu, Shusong and Guo, Jin and Chen, Chao-Hung and Hsyu, Ming-Chun and Tsai, Wen-Chia and Chen, Chao-Wei and Malivenko, Grigory and Kwon, Minsu and Lee, Myungje and Yoo, Jaeyoon and Kang, Changbeom and Wang, Shinjo and Shaolong, Zheng and Dejun, Hao and Fen, Xie and Zhuang, Feng and Ma, Yipeng and Peng, Jingyang and Wang, Tao and Song, Fenglong and Hsu, Chih-Chung and Chen, Kwan-Lin and Wu, Mei-Hsuang and Chudasama, Vishal and Prajapati, Kalpesh and Patel, Heena and Sarvaiya, Anjali and Upla, Kishor and Raja, Kiran and Ramachandra, Raghavendra and Busch, Christoph and de Stoutz, Etienne},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (Mobile AI)},
  year={2021},
  url={https://arxiv.org/abs/2105.07809}
}