Cognitive fusion of thermal and visible imagery for effective detection and tracking of pedestrians in videos

Yijun Yan, Jinchang Ren, Huimin Zhao, Genyun Sun, Zheng Wang, Jiangbin Zheng, Stephen Marshall, John Soraghan

Research output: Contribution to journalArticle

22 Citations (Scopus)
48 Downloads (Pure)

Abstract

BACKGROUND INTRODUCTION
In this paper, we present an efficient framework to cognitively detect and track salient objects from videos. In general, colored visible image in red-green-blue (RGB) has better distinguishability in human visual perception, yet it suffers from the effect of illumination noise and shadows. On the contrary, the thermal image is less sensitive to these noise effects though its distinguishability varies according to environmental settings. To this end, cognitive fusion of these two modalities provides an effective solution to tackle this problem.
METHODS
First, a background model is extracted followed by two stage background-subtraction for foreground detection in visible and thermal images. To deal with cases of occlusion or overlap, knowledge based forward tracking and backward tracking are employed to identify separate objects even the foreground detection fails.
RESULTS
To evaluate the proposed method, a publicly available color-thermal benchmark dataset OTCBVS is employed here. For our foreground detection evaluation, objective and subjective analysis against several state-of-the-art methods have been done on our manually segmented ground truth. For our object tracking evaluation, comprehensive qualitative experiments have also been done on all video sequences.
CONCLUSIONS
Promising results have shown that the proposed fusion based approach can successfully detect and track multiple human objects in most scenes regardless of any light change or occlusion problem.
Original languageEnglish
Number of pages11
JournalCognitive Computation
Early online date4 Dec 2017
DOIs
Publication statusE-pub ahead of print - 4 Dec 2017

Fingerprint

Imagery (Psychotherapy)
Fusion reactions
Hot Temperature
Noise
Benchmarking
Visual Perception
Lighting
Color
Light
Pedestrians
Experiments

Keywords

  • multiple objects detection
  • pedestrian detection/tracking
  • cognitive fusion
  • visible image
  • thermal image

Cite this

@article{574a6c2c22034dc6a7209ed9c23d8249,
title = "Cognitive fusion of thermal and visible imagery for effective detection and tracking of pedestrians in videos",
abstract = "BACKGROUND INTRODUCTIONIn this paper, we present an efficient framework to cognitively detect and track salient objects from videos. In general, colored visible image in red-green-blue (RGB) has better distinguishability in human visual perception, yet it suffers from the effect of illumination noise and shadows. On the contrary, the thermal image is less sensitive to these noise effects though its distinguishability varies according to environmental settings. To this end, cognitive fusion of these two modalities provides an effective solution to tackle this problem.METHODSFirst, a background model is extracted followed by two stage background-subtraction for foreground detection in visible and thermal images. To deal with cases of occlusion or overlap, knowledge based forward tracking and backward tracking are employed to identify separate objects even the foreground detection fails.RESULTSTo evaluate the proposed method, a publicly available color-thermal benchmark dataset OTCBVS is employed here. For our foreground detection evaluation, objective and subjective analysis against several state-of-the-art methods have been done on our manually segmented ground truth. For our object tracking evaluation, comprehensive qualitative experiments have also been done on all video sequences.CONCLUSIONSPromising results have shown that the proposed fusion based approach can successfully detect and track multiple human objects in most scenes regardless of any light change or occlusion problem.",
keywords = "multiple objects detection, pedestrian detection/tracking, cognitive fusion, visible image, thermal image",
author = "Yijun Yan and Jinchang Ren and Huimin Zhao and Genyun Sun and Zheng Wang and Jiangbin Zheng and Stephen Marshall and John Soraghan",
year = "2017",
month = "12",
day = "4",
doi = "10.1007/s12559-017-9529-6",
language = "English",
journal = "Cognitive Computation",
issn = "1866-9956",

}

Cognitive fusion of thermal and visible imagery for effective detection and tracking of pedestrians in videos. / Yan, Yijun; Ren, Jinchang; Zhao, Huimin; Sun, Genyun; Wang, Zheng; Zheng, Jiangbin; Marshall, Stephen; Soraghan, John.

In: Cognitive Computation, 04.12.2017.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Cognitive fusion of thermal and visible imagery for effective detection and tracking of pedestrians in videos

AU - Yan, Yijun

AU - Ren, Jinchang

AU - Zhao, Huimin

AU - Sun, Genyun

AU - Wang, Zheng

AU - Zheng, Jiangbin

AU - Marshall, Stephen

AU - Soraghan, John

PY - 2017/12/4

Y1 - 2017/12/4

N2 - BACKGROUND INTRODUCTIONIn this paper, we present an efficient framework to cognitively detect and track salient objects from videos. In general, colored visible image in red-green-blue (RGB) has better distinguishability in human visual perception, yet it suffers from the effect of illumination noise and shadows. On the contrary, the thermal image is less sensitive to these noise effects though its distinguishability varies according to environmental settings. To this end, cognitive fusion of these two modalities provides an effective solution to tackle this problem.METHODSFirst, a background model is extracted followed by two stage background-subtraction for foreground detection in visible and thermal images. To deal with cases of occlusion or overlap, knowledge based forward tracking and backward tracking are employed to identify separate objects even the foreground detection fails.RESULTSTo evaluate the proposed method, a publicly available color-thermal benchmark dataset OTCBVS is employed here. For our foreground detection evaluation, objective and subjective analysis against several state-of-the-art methods have been done on our manually segmented ground truth. For our object tracking evaluation, comprehensive qualitative experiments have also been done on all video sequences.CONCLUSIONSPromising results have shown that the proposed fusion based approach can successfully detect and track multiple human objects in most scenes regardless of any light change or occlusion problem.

AB - BACKGROUND INTRODUCTIONIn this paper, we present an efficient framework to cognitively detect and track salient objects from videos. In general, colored visible image in red-green-blue (RGB) has better distinguishability in human visual perception, yet it suffers from the effect of illumination noise and shadows. On the contrary, the thermal image is less sensitive to these noise effects though its distinguishability varies according to environmental settings. To this end, cognitive fusion of these two modalities provides an effective solution to tackle this problem.METHODSFirst, a background model is extracted followed by two stage background-subtraction for foreground detection in visible and thermal images. To deal with cases of occlusion or overlap, knowledge based forward tracking and backward tracking are employed to identify separate objects even the foreground detection fails.RESULTSTo evaluate the proposed method, a publicly available color-thermal benchmark dataset OTCBVS is employed here. For our foreground detection evaluation, objective and subjective analysis against several state-of-the-art methods have been done on our manually segmented ground truth. For our object tracking evaluation, comprehensive qualitative experiments have also been done on all video sequences.CONCLUSIONSPromising results have shown that the proposed fusion based approach can successfully detect and track multiple human objects in most scenes regardless of any light change or occlusion problem.

KW - multiple objects detection

KW - pedestrian detection/tracking

KW - cognitive fusion

KW - visible image

KW - thermal image

UR - https://link.springer.com/journal/12559

U2 - 10.1007/s12559-017-9529-6

DO - 10.1007/s12559-017-9529-6

M3 - Article

JO - Cognitive Computation

JF - Cognitive Computation

SN - 1866-9956

ER -