A data driven approach to audiovisual speech mapping

Andrew Abel; Ricard Marxer; Jon Barker; Roger Watt; Bill Whitmer; Peter Derleth; Amir Hussain

doi:10.1007/978-3-319-49685-6_30

A data driven approach to audiovisual speech mapping

Andrew Abel, Ricard Marxer, Jon Barker, Roger Watt, Bill Whitmer, Peter Derleth, Amir Hussain^*

^*Corresponding author for this work

Computer And Information Sciences

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution book

7 Citations (Scopus)

16 Downloads (Pure)

Abstract

The concept of using visual information as part of audio speech processing has been of significant recent interest. This paper presents a data driven approach that considers estimating audio speech acoustics using only temporal visual information without considering linguistic features such as phonemes and visemes. Audio (log filterbank) and visual (2D-DCT) features are extracted, and various configurations of MLP and datasets are used to identify optimal results, showing that given a sequence of prior visual frames an equivalent reasonably accurate audio frame estimation can be mapped.

Original language	English
Title of host publication	Advances in Brain Inspired Cognitive Systems
Subtitle of host publication	8th International Conference, BICS 2016, Beijing, China, November 28-30, 2016, Proceedings
Editors	Cheng-Lin Liu, Amir Hussain, Bin Luo, Kay Chen Tan, Yi Zeng, Zhaoxiang Zhang
Place of Publication	Cham, Switzerland
Publisher	Springer-Verlag
Pages	331-342
Number of pages	12
ISBN (Electronic)	9783319496856
ISBN (Print)	9783319496849
DOIs	https://doi.org/10.1007/978-3-319-49685-6_30
Publication status	Published - 13 Nov 2016
Event	8th International Conference on Brain Inspired Cognitive Systems, BICS 2016 - Beijing, China Duration: 28 Nov 2016 → 30 Nov 2016

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	10023 LNAI
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	8th International Conference on Brain Inspired Cognitive Systems, BICS 2016
Country/Territory	China
City	Beijing
Period	28/11/16 → 30/11/16

Funding

This work was supported by the UK Engineering and Physical Sciences Research Council (EPSRC) Grant No. EP/M026981/1 (CogAVHearing- http://cogavhearing.cs.stir.ac.uk ). In accordance with EPSRC policy, all experimental data used in the project simulations is available at http://hdl.handle.net/11667/81 . The authors would also like to gratefully acknowledge Prof. Leslie Smith and Dr Ahsan Adeel at the University of Stirling, Dr Kristína Malinovská at Comenius University in Bratislava, and the anonymous reviewers for their helpful comments and suggestions.

Keywords

ANNs
audiovisual
speech mapping
speech processing

Access to Document

10.1007/978-3-319-49685-6_30

Abel-etal-BICS2016-A-data-driven-approach-to-audiovisual-speech-mappingAccepted author manuscript, 162 KBLicence: Strath_1

Cite this

Abel, A., Marxer, R., Barker, J., Watt, R., Whitmer, B., Derleth, P., & Hussain, A. (2016). A data driven approach to audiovisual speech mapping. In C.-L. Liu, A. Hussain, B. Luo, K. C. Tan, Y. Zeng, & Z. Zhang (Eds.), Advances in Brain Inspired Cognitive Systems: 8th International Conference, BICS 2016, Beijing, China, November 28-30, 2016, Proceedings (pp. 331-342). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10023 LNAI). Springer-Verlag. https://doi.org/10.1007/978-3-319-49685-6_30

Abel, Andrew ; Marxer, Ricard ; Barker, Jon et al. / A data driven approach to audiovisual speech mapping. Advances in Brain Inspired Cognitive Systems: 8th International Conference, BICS 2016, Beijing, China, November 28-30, 2016, Proceedings. editor / Cheng-Lin Liu ; Amir Hussain ; Bin Luo ; Kay Chen Tan ; Yi Zeng ; Zhaoxiang Zhang. Cham, Switzerland : Springer-Verlag, 2016. pp. 331-342 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{c95d9fb1d57a4dc28d3f0f8f48e11ba8,

title = "A data driven approach to audiovisual speech mapping",

abstract = "The concept of using visual information as part of audio speech processing has been of significant recent interest. This paper presents a data driven approach that considers estimating audio speech acoustics using only temporal visual information without considering linguistic features such as phonemes and visemes. Audio (log filterbank) and visual (2D-DCT) features are extracted, and various configurations of MLP and datasets are used to identify optimal results, showing that given a sequence of prior visual frames an equivalent reasonably accurate audio frame estimation can be mapped.",

keywords = "ANNs, audiovisual, speech mapping, speech processing",

author = "Andrew Abel and Ricard Marxer and Jon Barker and Roger Watt and Bill Whitmer and Peter Derleth and Amir Hussain",

year = "2016",

month = nov,

day = "13",

doi = "10.1007/978-3-319-49685-6\_30",

language = "English",

isbn = "9783319496849",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer-Verlag",

pages = "331--342",

editor = "Cheng-Lin Liu and Amir Hussain and Bin Luo and Tan, \{Kay Chen\} and Yi Zeng and Zhaoxiang Zhang",

booktitle = "Advances in Brain Inspired Cognitive Systems",

note = "8th International Conference on Brain Inspired Cognitive Systems, BICS 2016 ; Conference date: 28-11-2016 Through 30-11-2016",

}

Abel, A, Marxer, R, Barker, J, Watt, R, Whitmer, B, Derleth, P & Hussain, A 2016, A data driven approach to audiovisual speech mapping. in C-L Liu, A Hussain, B Luo, KC Tan, Y Zeng & Z Zhang (eds), Advances in Brain Inspired Cognitive Systems: 8th International Conference, BICS 2016, Beijing, China, November 28-30, 2016, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 10023 LNAI, Springer-Verlag, Cham, Switzerland, pp. 331-342, 8th International Conference on Brain Inspired Cognitive Systems, BICS 2016, Beijing, China, 28/11/16. https://doi.org/10.1007/978-3-319-49685-6_30

A data driven approach to audiovisual speech mapping. / Abel, Andrew; Marxer, Ricard; Barker, Jon et al.
Advances in Brain Inspired Cognitive Systems: 8th International Conference, BICS 2016, Beijing, China, November 28-30, 2016, Proceedings. ed. / Cheng-Lin Liu; Amir Hussain; Bin Luo; Kay Chen Tan; Yi Zeng; Zhaoxiang Zhang. Cham, Switzerland: Springer-Verlag, 2016. p. 331-342 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10023 LNAI).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution book

TY - GEN

T1 - A data driven approach to audiovisual speech mapping

AU - Abel, Andrew

AU - Marxer, Ricard

AU - Barker, Jon

AU - Watt, Roger

AU - Whitmer, Bill

AU - Derleth, Peter

AU - Hussain, Amir

PY - 2016/11/13

Y1 - 2016/11/13

N2 - The concept of using visual information as part of audio speech processing has been of significant recent interest. This paper presents a data driven approach that considers estimating audio speech acoustics using only temporal visual information without considering linguistic features such as phonemes and visemes. Audio (log filterbank) and visual (2D-DCT) features are extracted, and various configurations of MLP and datasets are used to identify optimal results, showing that given a sequence of prior visual frames an equivalent reasonably accurate audio frame estimation can be mapped.

AB - The concept of using visual information as part of audio speech processing has been of significant recent interest. This paper presents a data driven approach that considers estimating audio speech acoustics using only temporal visual information without considering linguistic features such as phonemes and visemes. Audio (log filterbank) and visual (2D-DCT) features are extracted, and various configurations of MLP and datasets are used to identify optimal results, showing that given a sequence of prior visual frames an equivalent reasonably accurate audio frame estimation can be mapped.

KW - ANNs

KW - audiovisual

KW - speech mapping

KW - speech processing

U2 - 10.1007/978-3-319-49685-6_30

DO - 10.1007/978-3-319-49685-6_30

M3 - Conference contribution book

AN - SCOPUS:84997282854

SN - 9783319496849

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 331

EP - 342

BT - Advances in Brain Inspired Cognitive Systems

A2 - Liu, Cheng-Lin

A2 - Hussain, Amir

A2 - Luo, Bin

A2 - Tan, Kay Chen

A2 - Zeng, Yi

A2 - Zhang, Zhaoxiang

PB - Springer-Verlag

CY - Cham, Switzerland

T2 - 8th International Conference on Brain Inspired Cognitive Systems, BICS 2016

Y2 - 28 November 2016 through 30 November 2016

ER -

Abel A, Marxer R, Barker J, Watt R, Whitmer B, Derleth P et al. A data driven approach to audiovisual speech mapping. In Liu CL, Hussain A, Luo B, Tan KC, Zeng Y, Zhang Z, editors, Advances in Brain Inspired Cognitive Systems: 8th International Conference, BICS 2016, Beijing, China, November 28-30, 2016, Proceedings. Cham, Switzerland: Springer-Verlag. 2016. p. 331-342. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-319-49685-6_30

A data driven approach to audiovisual speech mapping

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Fingerprint

Cite this