Exploring the capability of text-to-image diffusion models with structural edge guidance for multi-spectral satellite image inpainting

Mikolaj Czerkawski; Christos Tachtatzis

doi:10.1109/LGRS.2024.3370212

Exploring the capability of text-to-image diffusion models with structural edge guidance for multi-spectral satellite image inpainting

Mikolaj Czerkawski, Christos Tachtatzis

Research output: Contribution to journal › Letter › peer-review

2 Citations (Scopus)

42 Downloads (Pure)

Abstract

The letter investigates the utility of text-to-image inpainting models for satellite image data. Two technical challenges of injecting structural guiding signals into the generative process as well as translating the inpainted RGB pixels to a wider set of MSI bands are addressed by introducing a novel inpainting framework based on StableDiffusion and ControlNet as well as a novel method for RGB-to-MSI translation. The results on a wider set of data suggest that the inpainting synthesized via StableDiffusion suffers from undesired artifacts and that a simple alternative of self-supervised internal inpainting achieves a higher quality of synthesis.

Original language	English
Article number	5001905
Number of pages	5
Journal	IEEE Geoscience and Remote Sensing Letters
Volume	21
Early online date	26 Feb 2024
DOIs	https://doi.org/10.1109/LGRS.2024.3370212
Publication status	E-pub ahead of print - 26 Feb 2024

Keywords

image inpainting
image completion
generative models

Access to Document

10.1109/LGRS.2024.3370212

Czerkawski-Tachtatzis-IEEE-GRSL-2024-Exploring-the-capability-of-text-to-image-diffusionAccepted author manuscript, 5.85 MBLicence: Strath_1

Cite this

@article{3aeb11f7a298488e8fa3833f6b674486,

title = "Exploring the capability of text-to-image diffusion models with structural edge guidance for multi-spectral satellite image inpainting",

abstract = "The letter investigates the utility of text-to-image inpainting models for satellite image data. Two technical challenges of injecting structural guiding signals into the generative process as well as translating the inpainted RGB pixels to a wider set of MSI bands are addressed by introducing a novel inpainting framework based on StableDiffusion and ControlNet as well as a novel method for RGB-to-MSI translation. The results on a wider set of data suggest that the inpainting synthesized via StableDiffusion suffers from undesired artifacts and that a simple alternative of self-supervised internal inpainting achieves a higher quality of synthesis.",

keywords = "image inpainting, image completion, generative models",

author = "Mikolaj Czerkawski and Christos Tachtatzis",

note = "{\textcopyright} 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting /republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.",

year = "2024",

month = feb,

day = "26",

doi = "10.1109/LGRS.2024.3370212",

language = "English",

volume = "21",

journal = "IEEE Geoscience and Remote Sensing Letters",

issn = "1545-598X",

}

TY - JOUR

T1 - Exploring the capability of text-to-image diffusion models with structural edge guidance for multi-spectral satellite image inpainting

AU - Czerkawski, Mikolaj

AU - Tachtatzis, Christos

N1 - © 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting /republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

PY - 2024/2/26

Y1 - 2024/2/26

N2 - The letter investigates the utility of text-to-image inpainting models for satellite image data. Two technical challenges of injecting structural guiding signals into the generative process as well as translating the inpainted RGB pixels to a wider set of MSI bands are addressed by introducing a novel inpainting framework based on StableDiffusion and ControlNet as well as a novel method for RGB-to-MSI translation. The results on a wider set of data suggest that the inpainting synthesized via StableDiffusion suffers from undesired artifacts and that a simple alternative of self-supervised internal inpainting achieves a higher quality of synthesis.

AB - The letter investigates the utility of text-to-image inpainting models for satellite image data. Two technical challenges of injecting structural guiding signals into the generative process as well as translating the inpainted RGB pixels to a wider set of MSI bands are addressed by introducing a novel inpainting framework based on StableDiffusion and ControlNet as well as a novel method for RGB-to-MSI translation. The results on a wider set of data suggest that the inpainting synthesized via StableDiffusion suffers from undesired artifacts and that a simple alternative of self-supervised internal inpainting achieves a higher quality of synthesis.

KW - image inpainting

KW - image completion

KW - generative models

UR - https://ieeexplore.ieee.org/document/10445344

U2 - 10.1109/LGRS.2024.3370212

DO - 10.1109/LGRS.2024.3370212

M3 - Letter

SN - 1545-598X

VL - 21

JO - IEEE Geoscience and Remote Sensing Letters

JF - IEEE Geoscience and Remote Sensing Letters

M1 - 5001905

ER -

Exploring the capability of text-to-image diffusion models with structural edge guidance for multi-spectral satellite image inpainting

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Fostering Precision Agriculture and Livestock Farming through Secure Access to Large-Scale HPC-Enabled Virtual Industrial Experimentation Environment Empowering Scalable Big Data Analytics (H2020-ICT-2018-2020) (CYBELE) | Czerkawski, Mikolaj

Fostering Precision Agriculture and Livestock Farming through Secure Access to Large-Scale HPC-Enabled Virtual Industrial Experimentation Environment Empowering Scalable Big Data Analytics (H2020-ICT-2018-2020) (CYBELE)

Cite this

Exploring the capability of text-to-image diffusion models with structural edge guidance for multi-spectral satellite image inpainting

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Projects

Fostering Precision Agriculture and Livestock Farming through Secure Access to Large-Scale HPC-Enabled Virtual Industrial Experimentation Environment Empowering Scalable Big Data Analytics (H2020-ICT-2018-2020) (CYBELE) | Czerkawski, Mikolaj

Fostering Precision Agriculture and Livestock Farming through Secure Access to Large-Scale HPC-Enabled Virtual Industrial Experimentation Environment Empowering Scalable Big Data Analytics (H2020-ICT-2018-2020) (CYBELE)

Cite this