General context :
The aim is to enhance and complement the use of remote sensing data archives via the combination of such data with textual and/or metadata information associated to the image scene or to the particular geographical area covered by the satellite data. To this purpose, research activities will be conducted on vision-language models tailored for the remote sensing field with a particular emphasis on the available CNES data archive.
PhD topic : Text/Metadata supervision for learning representations of satellite images.
Within satellite imagery, metadata such as ground sample distance, off-nadir viewing angle, time and location hold significant semantic information that improves scene understanding. Recent literature [Bourcier24] demonstrated that jointly combining remote sensing image data and their corresponding metadata allows to learn effectively representation capable to improve performances on downstream recognition tasks. In this project we propose to make a step further investigating how metadata can be used to learn large foundation models in the framework of the exploitation of large image archives of optical data. This task could leverage Pleiades (image and metadata) datasets.
In the frame of self-supervised learning, temporal contrastive learning is a simple way to construct a very high number of positive pairs. Another idea is to also leverage geographic metadata [Ayush21] in order to construct contrastive pairs, hence driving improved feature learning that will account for regional characteristics while foundation models are trained using data with a global coverage.
Large foundation models benefit from learning across multiple modalities and multiple tasks. Metadata may be used to address such situations by providing auxiliary labels (e.g., season, time of day, sensor name). The model can take metadata as an additional input (e.g., time + location embedding). Metadata can be directly combined with visual features in a cross-modal architecture (e.g., using vision transformers with tabular inputs).
Finally, models may generalize better across domains (sensor, location, season) by encoding domain-relevant metadata.
[Ayush21] Kumar Ayush, Burak Uzkent, Chenlin Meng, Kumar Tanmay, Marshall Burke, David Lobell, and Stefano Ermon, “Geography-Aware Self-Supervised Learning”, ICCV 2021
[Bourcier24] Bourcier, Jules, et al. "Learning representations of satellite images from metadata supervision." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.
