Imaging from temporal data via spiking convolutional neural networks

Paul Kirkland, Valentin Kapitany, Ashley Lyons, John Soraghan, Alex Turpin, Daniele Faccio, Gaetano Di Caterina

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

2 Downloads (Pure)

Abstract

A new approach for imaging that is solely based on the time of flight of photons coming from the entire imaged scene, combined with a novel machine learning algorithm for image reconstruction: a spiking convolutional neural network (SCNN) named Spike-SPI (Spiking - Single Pixel Imager). The approach uses a single point detector and the corresponding time-counting electronics, which provide the arrival time of photons in the form of spikes distributed over time. This data is transformed into a temporal histogram containing the number of photons per arrival time. A SCNN that converts the 1D temporal histograms into a 3D image (2D image with depth map) by exploiting the feature extraction capabilities of convolutional neural networks (CNNs), the high dimensional compressed latent space representations of a variational encoder-decoder network structure, and the asynchronous processing capabilities of a spiking neural network (SNN). The performance of the proposed SCNN is analysed to demonstrate the state-of-the-art feature extraction capabilities of CNNs and the low latency asynchronous processing of SNNs that offer both higher throughput and higher accuracy in image reconstruction from the ToF data, when compared to standard ANNs. The results of Spike-SPI show an increase in spatial accuracy of 15% over then ANN, using the Intersection of Union (IoU) for the objects in the scene. While also delivering a 100% increase over then ANN in object reconstruction signal to noise ratio (RSNR) from ~3dB to ~6dB. These results are also consistent across a range of IRF (Instrument Response Functions) values and photo counts, highlighting the robust nature of the new network structure. Moreover, the asynchronous processing nature of the spiking neurons allow for a faster throughput and less computational overhead, benefiting from the operational sparsity in the single point sensor.
Original languageEnglish
Title of host publicationEmerging Imaging and Sensing Technologies for Security and Defence V; and Advanced Manufacturing Technologies for Micro- and Nanosystems in Security and Defence III
EditorsGerald S. Buller, Richard C. Hollins, Robert A. Lamb, Martin Laurenzis, Andrea Camposseo, Maria Farsari, Luana Persano, Lynda E. Busse
Place of PublicationBellingham, Washington
Number of pages20
Volume11540
DOIs
Publication statusPublished - 20 Sep 2020

Keywords

  • convolutional neural network
  • sensors
  • CCD image sensors
  • data processing
  • feature extraction
  • image restoration
  • image retrieval

Fingerprint Dive into the research topics of 'Imaging from temporal data via spiking convolutional neural networks'. Together they form a unique fingerprint.

Cite this