O(N2)-Operation approximation of covariance matrix inverse in Gaussian process regression based on quasi-Newton BFGS method

W.E. Leithead, Yunong Zhang, Science Foundation Ireland grant, 00/PI.1/C067 (Funder), EPSRC, GR/M76379/01 (Funder)

Research output: Contribution to journalArticle

48 Citations (Scopus)

Abstract

Gaussian process (GP) is a Bayesian nonparametric regression model, showing good performance in various applications. However, during its model-tuning procedure, the GP implementation suffers from numerous covariance-matrix inversions of expensive O(N3) operations, where N is the matrix dimension. In this article, we propose using the quasi-Newton BFGS O(N2)-operation formula to approximate/replace recursively the inverse of covariance matrix at every iteration. The implementation accuracy is guaranteed carefully by a matrix-trace criterion and by the restarts technique to generate good initial guesses. A number of numerical tests are then performed based on the sinusoidal regression example and the Wiener-Hammerstein identification example. It is shown that by using the proposed implementation, more than 80% O(N3) operations could be eliminated, and a typical speedup of 5-9 could be achieved as compared to the standard maximum-likelihood-estimation (MLE) implementation commonly used in Gaussian process regression.
LanguageEnglish
Pages367-380
Number of pages13
JournalCommunications in Statistics - Simulation and Computation
Volume36
Issue number2
DOIs
Publication statusPublished - 2 Mar 2007

Fingerprint

BFGS Method
Quasi-Newton Method
Newton-Raphson method
Covariance matrix
Gaussian Process
Regression
Maximum likelihood estimation
Approximation
Tuning
Bayesian Nonparametrics
Quasi-Newton
Matrix Inversion
Restart
Nonparametric Model
Guess
Nonparametric Regression
Maximum Likelihood Estimation
Regression Model
Speedup
Trace

Keywords

  • Gaussian process regression
  • matrix inverse
  • optimization
  • O(N2) operations
  • quasi-Newton BFGS method

Cite this

Leithead, W.E. ; Zhang, Yunong ; Science Foundation Ireland grant, 00/PI.1/C067 (Funder) ; EPSRC, GR/M76379/01 (Funder). / O(N2)-Operation approximation of covariance matrix inverse in Gaussian process regression based on quasi-Newton BFGS method. In: Communications in Statistics - Simulation and Computation. 2007 ; Vol. 36, No. 2. pp. 367-380.
@article{a05c0adb70d345eca2547cb05bc34295,
title = "O(N2)-Operation approximation of covariance matrix inverse in Gaussian process regression based on quasi-Newton BFGS method",
abstract = "Gaussian process (GP) is a Bayesian nonparametric regression model, showing good performance in various applications. However, during its model-tuning procedure, the GP implementation suffers from numerous covariance-matrix inversions of expensive O(N3) operations, where N is the matrix dimension. In this article, we propose using the quasi-Newton BFGS O(N2)-operation formula to approximate/replace recursively the inverse of covariance matrix at every iteration. The implementation accuracy is guaranteed carefully by a matrix-trace criterion and by the restarts technique to generate good initial guesses. A number of numerical tests are then performed based on the sinusoidal regression example and the Wiener-Hammerstein identification example. It is shown that by using the proposed implementation, more than 80{\%} O(N3) operations could be eliminated, and a typical speedup of 5-9 could be achieved as compared to the standard maximum-likelihood-estimation (MLE) implementation commonly used in Gaussian process regression.",
keywords = "Gaussian process regression, matrix inverse, optimization, O(N2) operations, quasi-Newton BFGS method",
author = "W.E. Leithead and Yunong Zhang and {Science Foundation Ireland grant, 00/PI.1/C067 (Funder)} and {EPSRC, GR/M76379/01 (Funder)}",
year = "2007",
month = "3",
day = "2",
doi = "10.1080/03610910601161298",
language = "English",
volume = "36",
pages = "367--380",
journal = "Communications in Statistics - Simulation and Computation",
issn = "0361-0918",
number = "2",

}

O(N2)-Operation approximation of covariance matrix inverse in Gaussian process regression based on quasi-Newton BFGS method. / Leithead, W.E.; Zhang, Yunong; Science Foundation Ireland grant, 00/PI.1/C067 (Funder); EPSRC, GR/M76379/01 (Funder).

In: Communications in Statistics - Simulation and Computation, Vol. 36, No. 2, 02.03.2007, p. 367-380.

Research output: Contribution to journalArticle

TY - JOUR

T1 - O(N2)-Operation approximation of covariance matrix inverse in Gaussian process regression based on quasi-Newton BFGS method

AU - Leithead, W.E.

AU - Zhang, Yunong

AU - Science Foundation Ireland grant, 00/PI.1/C067 (Funder)

AU - EPSRC, GR/M76379/01 (Funder)

PY - 2007/3/2

Y1 - 2007/3/2

N2 - Gaussian process (GP) is a Bayesian nonparametric regression model, showing good performance in various applications. However, during its model-tuning procedure, the GP implementation suffers from numerous covariance-matrix inversions of expensive O(N3) operations, where N is the matrix dimension. In this article, we propose using the quasi-Newton BFGS O(N2)-operation formula to approximate/replace recursively the inverse of covariance matrix at every iteration. The implementation accuracy is guaranteed carefully by a matrix-trace criterion and by the restarts technique to generate good initial guesses. A number of numerical tests are then performed based on the sinusoidal regression example and the Wiener-Hammerstein identification example. It is shown that by using the proposed implementation, more than 80% O(N3) operations could be eliminated, and a typical speedup of 5-9 could be achieved as compared to the standard maximum-likelihood-estimation (MLE) implementation commonly used in Gaussian process regression.

AB - Gaussian process (GP) is a Bayesian nonparametric regression model, showing good performance in various applications. However, during its model-tuning procedure, the GP implementation suffers from numerous covariance-matrix inversions of expensive O(N3) operations, where N is the matrix dimension. In this article, we propose using the quasi-Newton BFGS O(N2)-operation formula to approximate/replace recursively the inverse of covariance matrix at every iteration. The implementation accuracy is guaranteed carefully by a matrix-trace criterion and by the restarts technique to generate good initial guesses. A number of numerical tests are then performed based on the sinusoidal regression example and the Wiener-Hammerstein identification example. It is shown that by using the proposed implementation, more than 80% O(N3) operations could be eliminated, and a typical speedup of 5-9 could be achieved as compared to the standard maximum-likelihood-estimation (MLE) implementation commonly used in Gaussian process regression.

KW - Gaussian process regression

KW - matrix inverse

KW - optimization

KW - O(N2) operations

KW - quasi-Newton BFGS method

U2 - 10.1080/03610910601161298

DO - 10.1080/03610910601161298

M3 - Article

VL - 36

SP - 367

EP - 380

JO - Communications in Statistics - Simulation and Computation

T2 - Communications in Statistics - Simulation and Computation

JF - Communications in Statistics - Simulation and Computation

SN - 0361-0918

IS - 2

ER -