Infrared and visible image fusion with ResNet and zero-phase component analysis

Hui Li, Xiao jun Wu, Tariq S. Durrani

Research output: Contribution to journalArticle

3 Citations (Scopus)

Abstract

In image fusion approaches, feature extraction and processing are key tasks, and the fusion performance is directly affected by the different features and processing methods undertaken. However, most of deep learning-based methods use deep features directly without them. This leads to the fusion performance degradation in some cases. To solve these drawbacks, in our paper, a deep features and zero-phase component analysis (ZCA) based novel fusion framework is proposed. Firstly, the residual network (ResNet) is used to extract deep features from source images. Then ZCA and l1-norm are utilized to normalize the deep features and obtain initial weight maps. The final weight maps are obtained by employing a soft-max operation in association with the initial weight maps. Finally, the fused image is reconstructed using a weighted-averaging strategy. Compared with the existing fusion methods, experimental results demonstrate that the proposed framework achieves better performance in both objective assessment and visual quality. The code of our fusion algorithm is available at https://github.com/hli1221/imagefusion_resnet50.

Original languageEnglish
Article number103039
Number of pages10
JournalInfrared Physics and Technology
Volume102
Early online date12 Sep 2019
DOIs
Publication statusPublished - 30 Nov 2019

Fingerprint

Image fusion
fusion
Infrared radiation
Processing
Feature extraction
Association reactions
Degradation
norms
pattern recognition
learning
degradation

Keywords

  • deep learning
  • image fusion
  • infrared image
  • residual network
  • visible image
  • zero-phase component analysis

Cite this

@article{6063775743ce4d9e80c8342f0a8fd268,
title = "Infrared and visible image fusion with ResNet and zero-phase component analysis",
abstract = "In image fusion approaches, feature extraction and processing are key tasks, and the fusion performance is directly affected by the different features and processing methods undertaken. However, most of deep learning-based methods use deep features directly without them. This leads to the fusion performance degradation in some cases. To solve these drawbacks, in our paper, a deep features and zero-phase component analysis (ZCA) based novel fusion framework is proposed. Firstly, the residual network (ResNet) is used to extract deep features from source images. Then ZCA and l1-norm are utilized to normalize the deep features and obtain initial weight maps. The final weight maps are obtained by employing a soft-max operation in association with the initial weight maps. Finally, the fused image is reconstructed using a weighted-averaging strategy. Compared with the existing fusion methods, experimental results demonstrate that the proposed framework achieves better performance in both objective assessment and visual quality. The code of our fusion algorithm is available at https://github.com/hli1221/imagefusion_resnet50.",
keywords = "deep learning, image fusion, infrared image, residual network, visible image, zero-phase component analysis",
author = "Hui Li and Wu, {Xiao jun} and Durrani, {Tariq S.}",
year = "2019",
month = "11",
day = "30",
doi = "10.1016/j.infrared.2019.103039",
language = "English",
volume = "102",
journal = "Infrared Physics and Technology",
issn = "1350-4495",
publisher = "Elsevier Science",

}

Infrared and visible image fusion with ResNet and zero-phase component analysis. / Li, Hui; Wu, Xiao jun; Durrani, Tariq S.

In: Infrared Physics and Technology, Vol. 102, 103039, 30.11.2019.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Infrared and visible image fusion with ResNet and zero-phase component analysis

AU - Li, Hui

AU - Wu, Xiao jun

AU - Durrani, Tariq S.

PY - 2019/11/30

Y1 - 2019/11/30

N2 - In image fusion approaches, feature extraction and processing are key tasks, and the fusion performance is directly affected by the different features and processing methods undertaken. However, most of deep learning-based methods use deep features directly without them. This leads to the fusion performance degradation in some cases. To solve these drawbacks, in our paper, a deep features and zero-phase component analysis (ZCA) based novel fusion framework is proposed. Firstly, the residual network (ResNet) is used to extract deep features from source images. Then ZCA and l1-norm are utilized to normalize the deep features and obtain initial weight maps. The final weight maps are obtained by employing a soft-max operation in association with the initial weight maps. Finally, the fused image is reconstructed using a weighted-averaging strategy. Compared with the existing fusion methods, experimental results demonstrate that the proposed framework achieves better performance in both objective assessment and visual quality. The code of our fusion algorithm is available at https://github.com/hli1221/imagefusion_resnet50.

AB - In image fusion approaches, feature extraction and processing are key tasks, and the fusion performance is directly affected by the different features and processing methods undertaken. However, most of deep learning-based methods use deep features directly without them. This leads to the fusion performance degradation in some cases. To solve these drawbacks, in our paper, a deep features and zero-phase component analysis (ZCA) based novel fusion framework is proposed. Firstly, the residual network (ResNet) is used to extract deep features from source images. Then ZCA and l1-norm are utilized to normalize the deep features and obtain initial weight maps. The final weight maps are obtained by employing a soft-max operation in association with the initial weight maps. Finally, the fused image is reconstructed using a weighted-averaging strategy. Compared with the existing fusion methods, experimental results demonstrate that the proposed framework achieves better performance in both objective assessment and visual quality. The code of our fusion algorithm is available at https://github.com/hli1221/imagefusion_resnet50.

KW - deep learning

KW - image fusion

KW - infrared image

KW - residual network

KW - visible image

KW - zero-phase component analysis

UR - https://github.com/hli1221/imagefusion_resnet50

U2 - 10.1016/j.infrared.2019.103039

DO - 10.1016/j.infrared.2019.103039

M3 - Article

VL - 102

JO - Infrared Physics and Technology

JF - Infrared Physics and Technology

SN - 1350-4495

M1 - 103039

ER -