Advances in optimisation algorithms and techniques for deep learning

Research output: Contribution to journalReview articlepeer-review

7 Citations (Scopus)
17 Downloads (Pure)


In the last decade, deep learning(DL) has witnessed excellent performances on a variety of problems, including speech recognition, object recognition, detection, and natural language processing (NLP) among many others. Of these applications, one common challenge is to obtain ideal parameters during the training of the deep neural networks (DNN). These typical parameters are obtained by some optimisation techniques which have been studied extensively. These research have produced state-of-art(SOTA) results on speed and memory improvements for deep neural networks(NN) architectures. However, the SOTA optimisers have continued to be an active research area with no compilations of the existing optimisers reported in the literature.
This paper provides an overview of the recent advances in optimisation algorithms and techniques used in DNN, highlighting the current SOTA optimisers, improvements made on these optimisation algorithms and techniques, alongside the trends in the development of optimisers used in training DL based models. The results of the search of the Scopus database for the optimisers in DL provides the articles reported as the summary of the DL optimisers. From what we can tell, there is no comprehensive compilation of the optimisation algorithms and techniques so far developed and used in DL research and applications, and this paper summarises these facts.
Original languageEnglish
Pages (from-to)563 - 577
Number of pages15
JournalAdvances in Science, Technology and Engineering Systems Journal
Issue number5
Publication statusPublished - 5 Oct 2020


  • deep learning
  • optimisers
  • deep learning optimisation
  • deep learning optimisers
  • optimisation algorithms
  • neural network optimisers


Dive into the research topics of 'Advances in optimisation algorithms and techniques for deep learning'. Together they form a unique fingerprint.

Cite this