(1) Overview
Introduction
Single image super-resolution consists in generating a high-resolution image from a low-resolution image. In recent years, deep learning approaches have emerged as state of the art for single image super-resolution []. However, applying them to remote sensing images requires a few adjustments of the methodology applied on traditional true color images datasets. Remote sensing are a bit different in nature, because they usually have more than the red, green and blue channels, they often have, for instance, spectral bands in the near infrared domain. Those channels are generally encoded on 16 bits, against 8 bits for traditional images. Moreover, remote sensing images usually include metadata enabling to map each pixel on the earth surface, in computing their position in physical units. For this purpose, images generated with super-resolution methods must have an updated metadata, determined from the transformation applied to the low resolution image. Our software implement the well established single image super-resolution approach of Ledig et al. []. It provides an application to train the network, and we have improved the training step using recent advances in image generation [, ] that improve the stability of the training process. A second application enables to apply the trained model over low resolution images, to generate synthetic high resolution images. Up to our knowledge, while a growing number of software are available on GitHub for traditional images, no such open-source software is available for remote sensing image processing, integrating the specificity of geospatial data.
Implementation and architecture
Our software relies on the Geospatial Abstraction Library (GDAL) [] and the Orfeo ToolBox (OTB) [] to perform I/O operations on remote sensing images and geospatial vector layers. TensorFlow [] is used to perform the computations to train and apply the deep neural networks. We rely on an extension of the OTB that eases the use of TensorFlow on remote sensing images, OTBTF (Figure 1). We use OTB applications shipped in this extension, to (1) extract images patches in remote sensing images, and (2) apply a trained and serialized TensorFlow model, the SavedModel. Our software implements (i) the model training from the remote sensing images patches, using TensorFlow for the model build and the parameters optimization and GDAL for the patches images reading, and (ii) the high resolution image generation, from the trained model, using OTBTF. Figure 3 summarizes the involved software stack.
Cross platform software stack.
1. Model training This part consist in accessing patches of images using GDAL, and feeding the collected samples to a TensorFlow computational graph implementing a deep convolutional neural network. This neural network is optimized each step, where new images patches are gathered, and fed to the training process (Figure 2). The neural network consist in an implementation of Ledig et al. [] network, modified to suit remote sensing images: the number of channels in the images, which can be greater than 3 in remote sensing images, is detected and used to build the model accordingly. This way, we can train a model that is not restricted to true color images, and that can process a multispectral remote sensing image for instance. The loss function consist in a contribution of diverse functions that express the gap between the target image, i.e. the high-resolution image, and the image generated by the model. Our application enables the user to carefully set the weight of each function used to derive the final loss function, used to optimize the model parameters. The application also writes logs for the TensorFlow’s dashboard, called Tensorboard, enabling the visualization of losses values and preview images generated from the network. The user can then experiment a variety of settings, and monitor the losses dynamics and the visual results during the training process. During the training, the model parameters are regularly saved in the file-system. Hence the application can train a new model from scratch, or from a particular state saved in the file-system. At the end of the execution process, the model is exported in the SavedModel format, and ready to be used in the next step.
Step 1: model training. GDAL is employed to read the input images (on-the-fly or in-memory). TensorFlow performs the training of the network, and export the trained weights in the SavedModel format.
2. Model inference In this part, a second application is used to apply the previously trained model on low-resolution remote sensing image (Figure 3). The application employs OTBTF to read the SavedModel and apply it over the entire input image. Since the entire image cannot fit in the GPU memory or computer RAM, the output image is generated in a region-by-region fashion, using the so-called streaming mechanism implemented in the OTB. The application eases the user experience, since he or she just has to provide the input image path, the SavedModel path, and chose a file name for the generated high resolution image.
Step 2: inference using a trained model, to generate a synthetic high-resolution geospatial image from one entire low-resolution remote sensing image. Weights of the trained model is stored in the SavedModel format.
Quality control
Since the training process is stochastic, unit tests for the training cannot be set. The inference part however, is guaranteed through OTBTF and OTB, which include a number of unit and functional tests, that can be trigger after the build of the library. In particular, OTBTF checks that the inference of a deep convolutional network lead to the correct result. The software documentation explains how to install the software in a simple manner, using an existing docker image where all the dependencies are installed, and ready to use. The software documentation provides a detailed description of the application parameters. It also include a typical use case, detailing each steps that the user has to perform, and explaining how to properly use the software with well known remote sensing products. Finally, a pre-trained model is provided with the software (see Reuse section) that can be used directly to generate high-resolution images from Sentinel-2 images with Red, Green, Blue and Near-infrared channels.
(2) Availability
Operating system
Our software can run on any platform thank to the OTBTF docker images. While it theoretically can be installed on Windows, Linux, and Mac, due to the used libraries, we only have extensively used it on linux environments (CentOS 8, and Ubuntu 18.04).
Programming language
Python (minimum, 3.5)
Additional system requirements
For the training step, we advise to use a GPU suited for deep learning computations. While the inference step can be performed on CPU, we strongly recommend a GPU to speed-up the process.
Dependencies
- OTB >= 7.2.0
- GDAL >= 2.4.0
- Tensorflow >= 1.15
List of contributors
- Rémi Cresson
- Benjamin Commandre
- Vincent Delbar
- Nicolas Narçon
Software location
Archive (e.g. institutional repository, general repository) (required – please see instructions on journal website for depositing archive copy of software in a suitable repository)
Name: SR4RS
Persistent identifier: 10.5281/zenodo.4537578
Licence: MIT
Publisher: Rémi Cresson
Version published: r1.3
Date published: 26/01/22
Code repository GitHub (required)
Name: SR4RS
Persistent identifier: https://github.com/remicres/sr4rs
Licence: MIT
Date published: 12/02/2021
Emulation environment (if appropriate)
Language
English
(3) Reuse potential
The software could be reused by researchers in the field of deep learning applied to remote sensing images. In particular, the software provides a practical framework to implementation new methods for super resolution of remote sensing images. Along with the software is provided a pre-trained model corresponding to the default settings values of the training application of the tool. The pre-trained can be downloaded from the link available in the documentation (PRETRAINED_MODELS.md). The model was trained from 250 different Spot-6 and Spot-7 scenes covering the entire France Mainland, acquired during the year 2020, from march to october, and the Sentinel-2 images acquired close to the same day (Sentinel-2 products from the THEIA Land data center). Spot-6 and Spot-7 images were interpolated at 2.5 meters and radiometrically calibrated to match the Sentinel-2 radiometry. Around 150k patches were used to train the model. The users can either start the training from the weights of the pre-trained model, or simply use it directly on Sentinel-2 images (channels Red, Green, Blue and Near-infrared at 10 meters spacing). We provide representative illustrative images obtained with the provided model in Figure 4.
Results obtained all around the earth using the available pre-trained model. Left: original Sentinel-2 images (in RGB colors), right: synthetic high-resolution images (2.5 meters spacing) generated with the software.
It should also be noted that the model can be applied inside any pipeline composed of OTB applications, thank to the streaming support. Also, thank to the integration of OTB inside the QGIS software, any model obtained with our software can be used directly from QGIS with the TensorflowModelServe application, which is part of the OTBTF. QGIS provides a number of features to interact in graphical user interface with the OTB, i.e. input image choice, and display the result of in the cartographic canvas. QGIS also enable the chaining of various software for geospatial data processing with its so-called model builder (e.g. GDAL/OGR, GRASS, SAGA, etc.). We note that to enable the OTBTF applications in QGIS, the library must be compiled in the same environment, and with the QGIS descriptors.