Machine learning and Artificial intelligence have already revolutionised the world
we live in. Nevertheless, these technologies are expected to progress even further and advance living standards far beyond today’s reality. To achieve this
revolution, several limitations of current AI need to be addressed. Demands on
computing resources and energy supply are a major obstacle, which ultimately
limit the capabilities of the AI we can deploy at the edge. Furthermore, current
systems are ill-suited for continual learning (CL) on real world data, as they require to train with all available data as an independent and identically distributed
(i.i.d.) set. This thesis focuses on these problems by working on visual feature
extraction, and contributing to more efficient and accurate algorithms, with the
capacity for CL.
With the objective of developing energy efficient feature extraction, a major
part of the thesis is focused on Spiking Neural Networks (SNNs). SNNs have become an interesting alternative to conventional artificial neural networks (ANN)
thanks to their temporal processing capabilities and energy efficient implementations in neuromorphic hardware. However, the challenges involved in training
them have limited their performance in terms of accuracy and thus their applications. Improving learning algorithms and neural architectures for a more accurate
feature extraction is therefore a priority. Contributing towards this aim, this work
presents a study on the key components of modern spiking architectures, an indepth study on the possible implementations of spiking residual connections, and
a novel spiking version of the successful residual network architecture. Additionally, the effect of different state of the art techniques are empirically compared in
image classification tasks to provide a state of the art guide to SNN design. Finally, the proposed network outperforms previous SNN architectures in multiple
datasets, while using less parameters.
In order to exploit SNNs for more efficient AI, it is also of interest to understand the full scope of their exploitable properties. These networks are characterised by their unique temporal dynamics, but the properties and advantages of such computations are still not fully understood. In order to provide answers, in
this work it is demonstrated how spiking neurons can enable temporal feature extraction in feed-forward neural networks without the need for recurrent synapses,
and how recurrent SNNs can achieve comparable results to LSTM with a smaller
number of parameters. This shows how their bio-inspired computing principles
can be successfully exploited beyond energy efficiency gains, and evidences their
differences with respect to conventional artificial neural networks. These results
are obtained through a new task, DVS-Gesture-Chain (DVS-GC), which allows,
for the first time, to evaluate the perception of temporal dependencies in a real
event-based action recognition dataset. Furthermore, this setup allows to reveal
the role of the leakage rate in spiking neurons for temporal processing tasks and
demonstrated the benefits of ”hard reset” mechanisms.
Finally, the focus is switched to the capacity for training feature extractors
in continual learning scenarios, a major milestone for the development of truly
autonomous systems and artificial general intelligence. The challenge in this
setup is to avoid catastrophic forgetting, where artificial systems forget previous
knowledge if they are trained in new data without revisiting the old. Often, the
methods used to alleviate forgetting make use of either rehearsal buffers, pretrained backbones or external indication of the task to solve. However, these requirements result in severe limitations regarding scalability, privacy preservation
and efficient deployment. This work explores how to eliminate the need for such
requirements and proposes a new method, Low Interference Feature Extraction
Sub-networks (LIFES). Additionally, the study breaks down the Catastrophic
Forgetting (CF) problem into 4 causes, allowing to better understand the effect
of CL methods. The proposed LIFES algorithm achieves competitive results in
standard incremental learning scenarios, providing an alternative to approaches
with more restrictive requirements. Moreover, it provides solutions for specific
causes of the CF problem, making it complementary to other methods
| Date of Award | 25 Feb 2025 |
|---|
| Original language | English |
|---|
| Awarding Institution | - University Of Strathclyde
|
|---|
| Sponsors | University of Strathclyde |
|---|
| Supervisor | Gaetano Di Caterina (Supervisor) & John Soraghan (Supervisor) |
|---|