Random scaling of quasi-Newton BFGS method to improve the o(N^2)-operation approximation of covariance-matrix inverse in Gaussian process

Y. Zhang, W.E. Leithead, D. Leith

Research output: Contribution to conference › Paper

Abstract

Gaussian process (GP) is a Bayesian nonparametric regression model, showing good performance in various applications. Similar to other computational models, Gaussian process frequently encounters the matrix-inverse problem during its model-tuning procedure. The matrix inversion is generally of O(N3) operations where N is the matrix dimension. We proposed using the O(N2)-operation quasi-Newton BFGS method to approximate/replace the exact inverse of covariance matrix in the GP context. As inspired during a paper revision, in this paper we show that by using the random-scaling technique, the accuracy and effectiveness of such a BFGS matrix-inverse approximation could be further improved. These random-scaling BFGS techniques could be widely generalized to other machine-learning systems which rely on explicit matrix-inverse.
Original languageEnglish
Pages452-457
Number of pages6
DOIs
Publication statusPublished - 1 Oct 2007
EventIEEE 22nd International Symposium on Intelligent Control, 2007. ISIC 2007. - Guangzhou, China
Duration: 1 Oct 2007 → …

Conference

ConferenceIEEE 22nd International Symposium on Intelligent Control, 2007. ISIC 2007.
CountryChina
CityGuangzhou
Period1/10/07 → …

Fingerprint

Newton-Raphson method
Covariance matrix
Learning systems
Inverse problems
Tuning

Keywords

  • gaussian process
  • matrix inverse approximation
  • quasi-Newton BFGS method
  • random scaling

Cite this

Zhang, Y., Leithead, W. E., & Leith, D. (2007). Random scaling of quasi-Newton BFGS method to improve the o(N^2)-operation approximation of covariance-matrix inverse in Gaussian process. 452-457. Paper presented at IEEE 22nd International Symposium on Intelligent Control, 2007. ISIC 2007. , Guangzhou, China. https://doi.org/10.1109/ISIC.2007.4450928
Zhang, Y. ; Leithead, W.E. ; Leith, D. / Random scaling of quasi-Newton BFGS method to improve the o(N^2)-operation approximation of covariance-matrix inverse in Gaussian process. Paper presented at IEEE 22nd International Symposium on Intelligent Control, 2007. ISIC 2007. , Guangzhou, China.6 p.
@conference{e7bbfcd55b82444bb69967499cd14957,
title = "Random scaling of quasi-Newton BFGS method to improve the o(N^2)-operation approximation of covariance-matrix inverse in Gaussian process",
abstract = "Gaussian process (GP) is a Bayesian nonparametric regression model, showing good performance in various applications. Similar to other computational models, Gaussian process frequently encounters the matrix-inverse problem during its model-tuning procedure. The matrix inversion is generally of O(N3) operations where N is the matrix dimension. We proposed using the O(N2)-operation quasi-Newton BFGS method to approximate/replace the exact inverse of covariance matrix in the GP context. As inspired during a paper revision, in this paper we show that by using the random-scaling technique, the accuracy and effectiveness of such a BFGS matrix-inverse approximation could be further improved. These random-scaling BFGS techniques could be widely generalized to other machine-learning systems which rely on explicit matrix-inverse.",
keywords = "gaussian process, matrix inverse approximation, quasi-Newton BFGS method, random scaling",
author = "Y. Zhang and W.E. Leithead and D. Leith",
year = "2007",
month = "10",
day = "1",
doi = "10.1109/ISIC.2007.4450928",
language = "English",
pages = "452--457",
note = "IEEE 22nd International Symposium on Intelligent Control, 2007. ISIC 2007. ; Conference date: 01-10-2007",

}

Zhang, Y, Leithead, WE & Leith, D 2007, 'Random scaling of quasi-Newton BFGS method to improve the o(N^2)-operation approximation of covariance-matrix inverse in Gaussian process' Paper presented at IEEE 22nd International Symposium on Intelligent Control, 2007. ISIC 2007. , Guangzhou, China, 1/10/07, pp. 452-457. https://doi.org/10.1109/ISIC.2007.4450928

Random scaling of quasi-Newton BFGS method to improve the o(N^2)-operation approximation of covariance-matrix inverse in Gaussian process. / Zhang, Y.; Leithead, W.E.; Leith, D.

2007. 452-457 Paper presented at IEEE 22nd International Symposium on Intelligent Control, 2007. ISIC 2007. , Guangzhou, China.

Research output: Contribution to conference › Paper

TY - CONF

T1 - Random scaling of quasi-Newton BFGS method to improve the o(N^2)-operation approximation of covariance-matrix inverse in Gaussian process

AU - Zhang, Y.

AU - Leithead, W.E.

AU - Leith, D.

PY - 2007/10/1

Y1 - 2007/10/1

N2 - Gaussian process (GP) is a Bayesian nonparametric regression model, showing good performance in various applications. Similar to other computational models, Gaussian process frequently encounters the matrix-inverse problem during its model-tuning procedure. The matrix inversion is generally of O(N3) operations where N is the matrix dimension. We proposed using the O(N2)-operation quasi-Newton BFGS method to approximate/replace the exact inverse of covariance matrix in the GP context. As inspired during a paper revision, in this paper we show that by using the random-scaling technique, the accuracy and effectiveness of such a BFGS matrix-inverse approximation could be further improved. These random-scaling BFGS techniques could be widely generalized to other machine-learning systems which rely on explicit matrix-inverse.

AB - Gaussian process (GP) is a Bayesian nonparametric regression model, showing good performance in various applications. Similar to other computational models, Gaussian process frequently encounters the matrix-inverse problem during its model-tuning procedure. The matrix inversion is generally of O(N3) operations where N is the matrix dimension. We proposed using the O(N2)-operation quasi-Newton BFGS method to approximate/replace the exact inverse of covariance matrix in the GP context. As inspired during a paper revision, in this paper we show that by using the random-scaling technique, the accuracy and effectiveness of such a BFGS matrix-inverse approximation could be further improved. These random-scaling BFGS techniques could be widely generalized to other machine-learning systems which rely on explicit matrix-inverse.

KW - gaussian process

KW - matrix inverse approximation

KW - quasi-Newton BFGS method

KW - random scaling

U2 - 10.1109/ISIC.2007.4450928

DO - 10.1109/ISIC.2007.4450928

M3 - Paper

SP - 452

EP - 457

ER -

Zhang Y, Leithead WE, Leith D. Random scaling of quasi-Newton BFGS method to improve the o(N^2)-operation approximation of covariance-matrix inverse in Gaussian process. 2007. Paper presented at IEEE 22nd International Symposium on Intelligent Control, 2007. ISIC 2007. , Guangzhou, China. https://doi.org/10.1109/ISIC.2007.4450928

Access to Document