Gustau Camps-Valls is Professor of Electrical Engineering and Lead Researcher in the Image Processing Laboratory (IPL) at the Universitat de València. His interests include development of statistical learning, mainly kernel machines and neural networks, for Earth sciences, from remote sensing to geoscience data analysis. Models efficiency and accuracy but also interpretability, consistency and causal discovery are driving his agenda on AI for Earth and climate.

Devis Tuia, PhD, is Associate Professor at the Ecole Polytechnique Fédérale de Lausanne (EPFL). He leads the Environmental Computational Science and Earth Observation laboratory, which focuses on the processing of Earth observation data with computational methods to advance Environmental science.

Xiao Xiang Zhu is Professor of Data Science in Earth Observation and Director of the Munich AI Future Lab AI4EO at the Technical University of Munich and heads the Department EO Data Science at the German Aerospace Center. Her lab develops innovative machine learning methods and big data analytics solutions to extract large scale geo-information from big Earth observation data, aiming at tackling societal grand challenges, e.g. Urbanization, UN's SDGs and Climate Change.

Markus Reichstein is Director of the Biogeochemical Integration Department at the Max-Planck- Institute for Biogeochemistry and Professor for Global Geoecology at the University of Jena. His main research interests include the response and feedback of ecosystems (vegetation and soils) to climatic variability with an Earth system perspective, considering coupled carbon, water and nutrient cycles. He has been tackling these topics with applied statistical learning for more than 15 years.

Foreword xvii

Acknowledgments xix

List of Contributors xxi

List of Acronyms xxvii

1 Introduction 1
Gustau Camps-Valls, Xiao Xiang Zhu, Devis Tuia, and Markus Reichstein

1.1 A Taxonomy of Deep Learning Approaches 2

1.2 Deep Learning in Remote Sensing 3

1.3 Deep Learning in Geosciences and Climate 7

1.4 Book Structure and Roadmap 9

Part I Deep Learning to Extract Information from Remote Sensing Images 13

2 Learning Unsupervised Feature Representations of Remote Sensing Data with Sparse Convolutional Networks 15
Jose E. Adsuara, Manuel Campos-Taberner, Javier García-Haro, Carlo Gatta, Adriana Romero, and Gustau Camps-Valls

2.1 Introduction 15

2.2 Sparse Unsupervised Convolutional Networks 17

2.2.1 Sparsity as the Guiding Criterion 17

2.2.2 The EPLS Algorithm 18

2.2.3 Remarks 18

2.3 Applications 19

2.3.1 Hyperspectral Image Classification 19

2.3.2 Multisensor Image Fusion 21

2.4 Conclusions 22

3 Generative Adversarial Networks in the Geosciences 24
Gonzalo Mateo-García, Valero Laparra, Christian Requena-Mesa, and Luis Gómez-Chova

3.1 Introduction 24

3.2 Generative Adversarial Networks 25

3.2.1 Unsupervised GANs 25

3.2.2 Conditional GANs 26

3.2.3 Cycle-consistent GANs 27

3.3 GANs in Remote Sensing and Geosciences 28

3.3.1 GANs in Earth Observation 28

3.3.2 Conditional GANs in Earth Observation 30

3.3.3 CycleGANs in Earth Observation 30

3.4 Applications of GANs in Earth Observation 31

3.4.1 Domain Adaptation Across Satellites 31

3.4.2 Learning to Emulate Earth Systems from Observations 33

3.5 Conclusions and Perspectives 36

4 Deep Self-taught Learning in Remote Sensing 37
Ribana Roscher

4.1 Introduction 37

4.2 Sparse Representation 38

4.2.1 Dictionary Learning 39

4.2.2 Self-taught Learning 40

4.3 Deep Self-taught Learning 40

4.3.1 Application Example 43

4.3.2 Relation to Deep Neural Networks 44

4.4 Conclusion 45

5 Deep Learning-based Semantic Segmentation in Remote Sensing 46
Devis Tuia, Diego Marcos, Konrad Schindler, and Bertrand Le Saux

5.1 Introduction 46

5.2 Literature Review 47

5.3 Basics on Deep Semantic Segmentation: Computer Vision Models 49

5.3.1 Architectures for Image Data 49

5.3.2 Architectures for Point-clouds 52

5.4 Selected Examples 55

5.4.1 Encoding Invariances to Train Smaller Models: The example of Rotation 55

5.4.2 Processing 3D Point Clouds as a Bundle of Images: SnapNet 59

5.4.3 Lake Ice Detection from Earth and from Space 62

5.5 Concluding Remarks 66

6 Object Detection in Remote Sensing 67
Jian Ding, Jinwang Wang, Wen Yang, and Gui-Song Xia

6.1 Introduction 67

6.1.1 Problem Description 67

6.1.2 Problem Settings of Object Detection 69

6.1.3 Object Representation in Remote Sensing 69

6.1.4 Evaluation Metrics 69

6.1.4.1 Precision-recall Curve 70

6.1.4.2 Average Precision and Mean Average Precision 71

6.1.5 Applications 71

6.2 Preliminaries on Object Detection with Deep Models 72

6.2.1 Two-stage Algorithms 72

6.2.1.1 R-CNNs 72

6.2.1.2 R-FCN 73

6.2.2 One-stage Algorithms 73

6.2.2.1 YOLO 73

6.2.2.2 SSD 73

6.3 Object Detection in Optical RS Images 75

6.3.1 RelatedWorks 75

6.3.1.1 Scale Variance 75

6.3.1.2 Orientation Variance 75

6.3.1.3 Oriented Object Detection 75

6.3.1.4 Detecting in Large-size Images 76

6.3.2 Datasets and Benchmark 77

6.3.2.1 DOTA 77

6.3.2.2 VisDrone 77

6.3.2.3 DIOR 77

6.3.2.4 xView 77

6.3.3 Two Representative Object Detectors in Optical RS Images 78

6.3.3.1 Mask OBB 78

6.3.3.2 RoI Transformer 82

6.4 Object Detection in SAR Images 86

6.4.1 Challenges of Detection in SAR Images 86

6.4.2 RelatedWorks 86

6.4.3 Datasets and Benchmarks 88

6.5 Conclusion 89

7 Deep Domain adaptation in Earth Observation 90
Benjamin Kellenberger, Onur Tasar, Bharath Bhushan Damodaran, Nicolas Courty, and Devis Tuia

7.1 Introduction 90

7.2 Families of Methodologies 91

7.3 Selected Examples 93

7.3.1 Adapting the Inner Representation 93

7.3.2 Adapting the Inputs Distribution 97

7.3.3 Using (few, well chosen) Labels from the Target Domain 100

7.4 Concluding remarks 104

8 Recurrent Neural Networks and the Temporal Component 105
Marco Körner and Marc Rußwurm

8.1 Recurrent Neural Networks 106

8.1.1 Training RNNs 107

8.1.1.1 Exploding and Vanishing Gradients 107

8.1.1.2 Circumventing Exploding and Vanishing Gradients 109

8.2 Gated Variants of RNNs 111

8.2.1 Long Short-term Memory Networks 111

8.2.1.1 The Cell State ct and the Hidden State ht 112

8.2.1.2 The Forget Gate ft 112

8.2.1.3 The Modulation Gate vt and the Input Gate it 112

8.2.1.4 The Output Gate ot 112

8.2.1.5 Training LSTM Networks 113

8.2.2 Other Gated Variants 113

8.3 Representative Capabilities of Recurrent Networks 114

8.3.1 Recurrent Neural Network Topologies 114

8.3.2 Experiments 115

8.4 Application in Earth Sciences 117

8.5 Conclusion 118

9 Deep Learning for Image Matching and Co-registration 120
Maria Vakalopoulou, Stergios Christodoulidis, Mihir Sahasrabudhe, and Nikos Paragios

9.1 Introduction 120

9.2 Literature Review 123

9.2.1 Classical Approaches 123

9.2.2 Deep Learning Techniques for Image Matching 124

9.2.3 Deep Learning Techniques for Image Registration 125

9.3 Image Registration with Deep Learning 126

9.3.1 2D Linear and Deformable Transformer 126

9.3.2 Network Architectures 127

9.3.3 Optimization Strategy 128

9.3.4 Dataset and Implementation Details 129

9.3.5 Experimental Results 129

9.4 Conclusion and Future Research 134

9.4.1 Challenges and Opportunities 134

9.4.1.1 Dataset with Annotations 134

9.4.1.2 Dimensionality of Data 135

9.4.1.3 Multitemporal Datasets 135

9.4.1.4 Robustness to Changed Areas 135

10 Multisource Remote Sensing Image Fusion 136
Wei He, Danfeng Hong, Giuseppe Scarpa, Tatsumi Uezato, and Naoto Yokoya

10.1 Introduction 136

10.2 Pansharpening 137

10.2.1 Survey of Pansharpening Methods Employing Deep Learning 137

10.2.2 Experimental Results 140

10.2.2.1 Experimental Design 140

10.2.2.2 Visual and Quantitative Comparison in Pansharpening 140

10.3 Multiband Image Fusion 143

10.3.1 Supervised Deep Learning-based Approaches 143

10.3.2 Unsupervised Deep Learning-based Approaches 145

10.3.3 Experimental Results 146

10.3.3.1 Comparison Methods and Evaluation Measures 146

10.3.3.2 Dataset and Experimental Setting 146

10.3.3.3 Quantitative Comparison and Visual Results 147

10.4 Conclusion and Outlook 148

11 Deep Learning for Image Search and Retrieval in Large Remote Sensing Archives 150
Gencer Sumbul, Jian Kang, and Begüm Demir

11.1 Introduction 150

11.2 Deep Learning for RS CBIR 152

11.3 Scalable RS CBIR Based on Deep Hashing 156

11.4 Discussion and Conclusion 160

Part II Making a Difference in the Geosciences With Deep Learning 161

12 Deep Learning for Detecting Extreme Weather Patterns 163
Mayur Mudigonda, Prabhat, Karthik Kashinath, Evan Racah, Ankur Mahesh, Yunjie Liu, Christopher Beckham, Jim Biard, Thorsten Kurth, Sookyung Kim, Samira Kahou, Tegan Maharaj, Burlen Loring, Christopher Pal, Travis O'Brien, Ken Kunkel, Michael F. Wehner, and William D. Collins

12.1 Scientific Motivation 163

12.2 Tropical Cyclone and Atmospheric River Classification 166

12.2.1 Methods 166

12.2.2 Network Architecture 167

12.2.3 Results 169

12.3 Detection of Fronts 170

12.3.1 Analytical Approach 170

12.3.2 Dataset 171

12.3.3 Results 172

12.3.4 Limitations 174

12.4 Semi-supervised Classification and Localization of Extreme Events 175

12.4.1 Applications of Semi-supervised Learning in Climate Modeling 175

12.4.1.1 Supervised Architecture 176

12.4.1.2 Semi-supervised Architecture 176

12.4.2 Results 176

12.4.2.1 Frame-wise Reconstruction 176

12.4.2.2 Results and Discussion 178

12.5 Detecting Atmospheric Rivers and Tropical Cyclones Through Segmentation Methods 179

12.5.1 Modeling Approach 179

12.5.1.1 Segmentation Architecture 180

12.5.1.2 Climate Dataset and Labels 181

12.5.2 Architecture Innovations:Weighted Loss and Modified Network 181

12.5.3 Results 183

12.6 Challenges and Implications for the Future 184

12.7 Conclusions 185

13 Spatio-temporal Autoencoders in Weather and Climate Research 186
Xavier-Andoni Tibau, Christian Reimers, Christian Requena-Mesa, and Jakob Runge

13.1 Introduction 186

13.2 Autoencoders 187

13.2.1 A Brief History of Autoencoders 188

13.2.2 Archetypes of Autoencoders 189

13.2.3 Variational Autoencoders (VAE) 191

13.2.4 Comparison Between Autoencoders and Classical Methods 192

13.3 Applications 193

13.3.1 Use of the Latent Space 193

13.3.1.1 Reduction of Dimensionality for the Understanding of the System Dynamics and its Interactions 195

13.3.1.2 Dimensionality Reduction for Feature Extraction and Prediction 199

13.3.2 Use of the Decoder 199

13.3.2.1 As a Random Sample Generator 201

13.3.2.2 Anomaly Detection 201

13.3.2.3 Use of a Denoising Autoencoder (DAE) Decoder 202

13.4 Conclusions and Outlook 203

14 Deep Learning to Improve Weather Predictions 204
Peter D. Dueben, Peter Bauer, and Samantha Adams

14.1 NumericalWeather Prediction 204

14.2 How Will Machine Learning EnhanceWeather Predictions? 207

14.3 Machine Learning Across theWorkflow ofWeather Prediction 208

14.4 Challenges for the Application of ML inWeather Forecasts 213

14.5 TheWay Forward 216

15 Deep Learning and the Weather Forecasting Problem: Precipitation Nowcasting 218
Zhihan Gao, Xingjian Shi, Hao Wang, Dit-Yan Yeung, Wang-chun Woo, and Wai-Kin Wong

15.1 Introduction 218

15.2 Formulation 220

15.3 Learning Strategies 221

15.4 Models 223

15.4.1 FNN-based Odels 223

15.4.2 RNN-based Models 225

15.4.3 Encoder-forecaster Structure 226

15.4.4 Convolutional LSTM 226

15.4.5 ConvLSTM with Star-shaped Bridge 227

15.4.6 Predictive RNN 228

15.4.7 Memory in Memory Network 229

15.4.8 Trajectory GRU 231

15.5 Benchmark 233

15.5.1 HKO-7 Dataset 234

15.5.2 Evaluation Methodology 234

15.5.3 Evaluated Algorithms 235

15.5.4 Evaluation Results 236

15.6 Discussion 236

Appendix 238

Acknowledgement 239

16 Deep Learning for High-dimensional Parameter...