Activation functions: comparison of trends in practice and research for deep learning

Chigozie Enyinna Nwankpa; Winifred Ijomah; Anthony Gachagan; Stephen Marshall

Activation functions: comparison of trends in practice and research for deep learning

Chigozie Enyinna Nwankpa, Winifred Ijomah, Anthony Gachagan, Stephen Marshall

Research output: Contribution to conference › Proceeding › peer-review

2165 Downloads (Pure)

Abstract

Deep neural networks (DNN) have been successfully used in diverse emerging domains to solve real world complex problems with may more deep learning (DL) architectures, being developed to date. To achieve this state-of-the-art (SOTA) performances, the DL architectures use activation functions (AFs), to perform diverse computations between the hidden layers and the output layers of any given DL architecture. This paper presents a survey on the existing
AFs used in deep learning applications and highlights the recent trends in the use of the AFs for DL applications. The novelty of this paper is that it compiles the majority of the AFs used in DL and outlines the current trends in the
applications and usage of these functions in practical deep learning deployments against the SOTA research results. This compilation will aid in making effective decisions in the choice of the most suitable and appropriate AF for a given
application, ready for deployment. This paper is timely because majority of the research papers on AF highlights similar works and results while this paper will be the first, to compile the trends in AF applications in practice against the
research results from the literature, found in DL research to date.

Original language	English
Pages	124 - 133
Number of pages	10
Publication status	Published - 11 Jan 2021
Event	2nd International Conference on Computational Sciences and Technology - Jamshoro, Jamshoro, Pakistan Duration: 17 Dec 2020 → 19 Dec 2020 Conference number: 2 https://inccst.muet.edu.pk/pdf/23.%20Activation%20Functions%20Comparison%20of%20Trends%20in%20Practice%20and%20Research%20for%20Deep%20Learning.pdf

Conference

Conference	2nd International Conference on Computational Sciences and Technology
Abbreviated title	(INCCST)
Country/Territory	Pakistan
City	Jamshoro
Period	17/12/20 → 19/12/20
Internet address	https://inccst.muet.edu.pk/pdf/23.%20Activation%20Functions%20Comparison%20of%20Trends%20in%20Practice%20and%20Research%20for%20Deep%20Learning.pdf

Keywords

activation function
activation function types
activation function choices
deep learning
neural networks
learning algorithms

Access to Document

Nwankpa-etal-ICCST-2021-Activation-functions-comparison-of-trends-in-practiceAccepted author manuscript, 279 KB

Cite this

@conference{0706bf17a845490688ef4d7d19df65ba,

title = "Activation functions: comparison of trends in practice and research for deep learning",

abstract = "Deep neural networks (DNN) have been successfully used in diverse emerging domains to solve real world complex problems with may more deep learning (DL) architectures, being developed to date. To achieve this state-of-the-art (SOTA) performances, the DL architectures use activation functions (AFs), to perform diverse computations between the hidden layers and the output layers of any given DL architecture. This paper presents a survey on the existingAFs used in deep learning applications and highlights the recent trends in the use of the AFs for DL applications. The novelty of this paper is that it compiles the majority of the AFs used in DL and outlines the current trends in theapplications and usage of these functions in practical deep learning deployments against the SOTA research results. This compilation will aid in making effective decisions in the choice of the most suitable and appropriate AF for a givenapplication, ready for deployment. This paper is timely because majority of the research papers on AF highlights similar works and results while this paper will be the first, to compile the trends in AF applications in practice against theresearch results from the literature, found in DL research to date.",

keywords = "activation function, activation function types, activation function choices, deep learning, neural networks, learning algorithms",

author = "Nwankpa, {Chigozie Enyinna} and Winifred Ijomah and Anthony Gachagan and Stephen Marshall",

year = "2021",

month = jan,

day = "11",

language = "English",

pages = "124 -- 133",

note = "2nd International Conference on Computational Sciences and Technology, (INCCST) ; Conference date: 17-12-2020 Through 19-12-2020",

url = "https://inccst.muet.edu.pk/pdf/23.%20Activation%20Functions%20Comparison%20of%20Trends%20in%20Practice%20and%20Research%20for%20Deep%20Learning.pdf",

}

TY - CONF

T1 - Activation functions

T2 - 2nd International Conference on Computational Sciences and Technology

AU - Nwankpa, Chigozie Enyinna

AU - Ijomah, Winifred

AU - Gachagan, Anthony

AU - Marshall, Stephen

N1 - Conference code: 2

PY - 2021/1/11

Y1 - 2021/1/11

N2 - Deep neural networks (DNN) have been successfully used in diverse emerging domains to solve real world complex problems with may more deep learning (DL) architectures, being developed to date. To achieve this state-of-the-art (SOTA) performances, the DL architectures use activation functions (AFs), to perform diverse computations between the hidden layers and the output layers of any given DL architecture. This paper presents a survey on the existingAFs used in deep learning applications and highlights the recent trends in the use of the AFs for DL applications. The novelty of this paper is that it compiles the majority of the AFs used in DL and outlines the current trends in theapplications and usage of these functions in practical deep learning deployments against the SOTA research results. This compilation will aid in making effective decisions in the choice of the most suitable and appropriate AF for a givenapplication, ready for deployment. This paper is timely because majority of the research papers on AF highlights similar works and results while this paper will be the first, to compile the trends in AF applications in practice against theresearch results from the literature, found in DL research to date.

AB - Deep neural networks (DNN) have been successfully used in diverse emerging domains to solve real world complex problems with may more deep learning (DL) architectures, being developed to date. To achieve this state-of-the-art (SOTA) performances, the DL architectures use activation functions (AFs), to perform diverse computations between the hidden layers and the output layers of any given DL architecture. This paper presents a survey on the existingAFs used in deep learning applications and highlights the recent trends in the use of the AFs for DL applications. The novelty of this paper is that it compiles the majority of the AFs used in DL and outlines the current trends in theapplications and usage of these functions in practical deep learning deployments against the SOTA research results. This compilation will aid in making effective decisions in the choice of the most suitable and appropriate AF for a givenapplication, ready for deployment. This paper is timely because majority of the research papers on AF highlights similar works and results while this paper will be the first, to compile the trends in AF applications in practice against theresearch results from the literature, found in DL research to date.

KW - activation function

KW - activation function types

KW - activation function choices

KW - deep learning

KW - neural networks

KW - learning algorithms

M3 - Proceeding

SP - 124

EP - 133

Y2 - 17 December 2020 through 19 December 2020

ER -

Activation functions: comparison of trends in practice and research for deep learning

Abstract

Conference

Keywords

Access to Document

Fingerprint

Cite this