Most image super-resolution (SR) methods are developed on synthetic low-resolution (LR) and high-resolution (HR) image pairs that are constructed by a fixed and predetermined operation, e.g., bicubic downsampling. As these approaches typically learn an inverse mapping of the specific function, existing SR methods usually produce blurry results when applied to real-world images whose exact formulation is different and unknown. Therefore, several methods attempt to synthesize much more diverse LR samples or learn a realistic downsampling model. However, due to restrictive assumptions on the downsampling process, they are still biased and less generalizable.

In this study, we propose a novel method to simulate an unknown downsampling process without imposing restrictive prior knowledge. We design an effective and generalizable low-frequency loss (LFL) in the adversarial training framework to imitate the distribution of target LR images without using any paired examples. We further propose an adaptive data loss (ADL) for the downsampler, which can be adaptively learned from the given dataset and updated in the training loops. Extensive experiments and analysis show that the LR images from our downsampling model can facilitate existing SR methods to perform more accurate reconstructions on various synthetic and real-world examples than the conventional approaches.

Constructing paired real-world images are very difficult due to a lack of scene diversity, misalighments, dynamic motions, and scalability issues. Therefore, we adopt configuration which utilizes unpaired dataset for real-world super-resolution.

Below figures show our 2-stage approach for unpaired SR. In the first stage, a downsampling model D learn to synthesize LR from HR. We note that proposed noble data loss ADL is used at this stage, which facilitates training for generating LR from HR. In the second stage, we train the SR model S, which can also be generalized to the target LR images LR by using pairs which are generated in first stage. Dotted lines in gray represent latent components that are not available in the entire learning process.

First stage
generating LR from HR

Second stage
traing SR model from generated pairs

Adaptive Data Loss(ADL)

Proposed ADL acts as not only preserving color contents, but also boosting the effect of adversarial loss. Note that ADL is updated along epochs, thus named as adaptive.

Data loss from a predetermined kernel
e.g. bicubic

Low-frequency Loss (proposed)

Adaptive Data Loss (proposed)

comparison to the other methods on real-world SR

Here we visualized SR results on real-world images. We compared with other methods which can be applied to real-world images.

Below images are SR results on RealSR(V3) datasets. As this dataset contains pixelwise well aligned HR, we note that there exists GT HR images for input LR.

input LR

RRDB
trained on bicubic pair

IKC

kernelGAN+ZSSR

ADL+RRDB

GT

Bibtex

@ARTICLE{9521710,
author = {Son, Sanghyun and Kim, Jaeha and Lai, Wei-Sheng and Yang, Ming-Hsuan and Lee, Kyoung Mu},
journal={IEEE Transactions on Pattern Analysis and Machine intelligence},
title = {Toward Real-World Super-Resolution via Adaptive Downsampling Models},
year={2021}
, volumne={}
, number={}
, pages={1-1}, doi = {10.1109/TPAMI.2021.3106790}
}