Activation functions: comparison of trends in practice and research for deep learning

Research output: Contribution to conferenceProceedingpeer-review

1505 Downloads (Pure)


Deep neural networks (DNN) have been successfully used in diverse emerging domains to solve real world complex problems with may more deep learning (DL) architectures, being developed to date. To achieve this state-of-the-art (SOTA) performances, the DL architectures use activation functions (AFs), to perform diverse computations between the hidden layers and the output layers of any given DL architecture. This paper presents a survey on the existing
AFs used in deep learning applications and highlights the recent trends in the use of the AFs for DL applications. The novelty of this paper is that it compiles the majority of the AFs used in DL and outlines the current trends in the
applications and usage of these functions in practical deep learning deployments against the SOTA research results. This compilation will aid in making effective decisions in the choice of the most suitable and appropriate AF for a given
application, ready for deployment. This paper is timely because majority of the research papers on AF highlights similar works and results while this paper will be the first, to compile the trends in AF applications in practice against the
research results from the literature, found in DL research to date.
Original languageEnglish
Pages124 - 133
Number of pages10
Publication statusPublished - 11 Jan 2021
Event2nd International Conference on Computational Sciences and Technology - Jamshoro, Jamshoro, Pakistan
Duration: 17 Dec 202019 Dec 2020
Conference number: 2


Conference2nd International Conference on Computational Sciences and Technology
Abbreviated title (INCCST)
Internet address


  • activation function
  • activation function types
  • activation function choices
  • deep learning
  • neural networks
  • learning algorithms


Dive into the research topics of 'Activation functions: comparison of trends in practice and research for deep learning'. Together they form a unique fingerprint.

Cite this