Using Bottleneck Adapters to Identify Cancer in Clinical Notes under Low-Resource Constraints

ISARIC Clinical Characterisation Group; Omid Rohanian; Hannah Jauncey; Mohammadmahdi Nouriborji; Vinod Kumar Chauhan; Bronner P. Gonçalves; Christiana Kartsonaki; Laura Merson; David Clifton

doi:10.18653/v1/2023.bionlp-1.5

Using Bottleneck Adapters to Identify Cancer in Clinical Notes under Low-Resource Constraints

ISARIC Clinical Characterisation Group, Omid Rohanian, Hannah Jauncey, Mohammadmahdi Nouriborji, Vinod Kumar Chauhan, Bronner P. Gonçalves, Christiana Kartsonaki, Laura Merson, David Clifton

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution book

5 Citations (Scopus)

Abstract

Processing information locked within clinical health records is a challenging task that remains an active area of research in biomedical NLP. In this work, we evaluate a broad set of machine learning techniques ranging from simple RNNs to specialised transformers such as BioBERT on a dataset containing clinical notes along with a set of annotations indicating whether a sample is cancer-related or not. Furthermore, we specifically employ efficient fine-tuning methods from NLP, namely, bottleneck adapters and prompt tuning, to adapt the models to our specialised task. Our evaluations suggest that fine-tuning a frozen BERT model pre-trained on natural language and with bottleneck adapters outperforms all other strategies, including full fine-tuning of the specialised BioBERT model. Based on our findings, we suggest that using bottleneck adapters in low-resource situations with limited access to labelled data or processing capacity could be a viable strategy in biomedical text mining. The code used in the experiments are going to be made available at [LINK ANONYMIZED].

Original language	English
Title of host publication	BioNLP 2023 - BioNLP and BioNLP-ST, Proceedings of the Workshop
Editors	Dina Demner-fushman, Sophia Ananiadou, Kevin Cohen
Pages	62-78
Number of pages	17
ISBN (Electronic)	9781959429852
DOIs	https://doi.org/10.18653/v1/2023.bionlp-1.5
Publication status	Published - 31 Jul 2023
Externally published	Yes
Event	22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, BioNLP 2023 - Toronto, Canada Duration: 13 Jul 2023 → …

Publication series

Name	Proceedings of the Annual Meeting of the Association for Computational Linguistics
ISSN (Print)	0736-587X

Conference

Conference	22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, BioNLP 2023
Country/Territory	Canada
City	Toronto
Period	13/07/23 → …

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 3 Good Health and Well-being

Keywords

Computational linguistics
Natural language processing systems
Clinical notes

Access to Document

10.18653/v1/2023.bionlp-1.5Licence: CC BY 4.0

2023.bionlp-Using-Bottleneck-Adapters-to-Identify-Cancer-in-Clinical-Notes-under-Low-Resource-ConstraintsFinal published version, 411 KBLicence: CC BY 4.0

Cite this

ISARIC Clinical Characterisation Group, Rohanian, O., Jauncey, H., Nouriborji, M., Chauhan, V. K., Gonçalves, B. P., Kartsonaki, C., Merson, L., & Clifton, D. (2023). Using Bottleneck Adapters to Identify Cancer in Clinical Notes under Low-Resource Constraints. In D. Demner-fushman, S. Ananiadou, & K. Cohen (Eds.), BioNLP 2023 - BioNLP and BioNLP-ST, Proceedings of the Workshop (pp. 62-78). (Proceedings of the Annual Meeting of the Association for Computational Linguistics). https://doi.org/10.18653/v1/2023.bionlp-1.5

ISARIC Clinical Characterisation Group ; Rohanian, Omid ; Jauncey, Hannah et al. / Using Bottleneck Adapters to Identify Cancer in Clinical Notes under Low-Resource Constraints. BioNLP 2023 - BioNLP and BioNLP-ST, Proceedings of the Workshop. editor / Dina Demner-fushman ; Sophia Ananiadou ; Kevin Cohen. 2023. pp. 62-78 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

@inproceedings{b6b02bfecbf340c399569daa1b31f0db,

title = "Using Bottleneck Adapters to Identify Cancer in Clinical Notes under Low-Resource Constraints",

abstract = "Processing information locked within clinical health records is a challenging task that remains an active area of research in biomedical NLP. In this work, we evaluate a broad set of machine learning techniques ranging from simple RNNs to specialised transformers such as BioBERT on a dataset containing clinical notes along with a set of annotations indicating whether a sample is cancer-related or not. Furthermore, we specifically employ efficient fine-tuning methods from NLP, namely, bottleneck adapters and prompt tuning, to adapt the models to our specialised task. Our evaluations suggest that fine-tuning a frozen BERT model pre-trained on natural language and with bottleneck adapters outperforms all other strategies, including full fine-tuning of the specialised BioBERT model. Based on our findings, we suggest that using bottleneck adapters in low-resource situations with limited access to labelled data or processing capacity could be a viable strategy in biomedical text mining. The code used in the experiments are going to be made available at [LINK ANONYMIZED].",

keywords = "Computational linguistics, Natural language processing systems, Clinical notes",

author = "\{ISARIC Clinical Characterisation Group\} and Omid Rohanian and Hannah Jauncey and Mohammadmahdi Nouriborji and Chauhan, \{Vinod Kumar\} and Gon{\c c}alves, \{Bronner P.\} and Christiana Kartsonaki and Laura Merson and David Clifton",

year = "2023",

month = jul,

day = "31",

doi = "10.18653/v1/2023.bionlp-1.5",

language = "English",

series = "Proceedings of the Annual Meeting of the Association for Computational Linguistics",

pages = "62--78",

editor = "Dina Demner-fushman and Sophia Ananiadou and Kevin Cohen",

booktitle = "BioNLP 2023 - BioNLP and BioNLP-ST, Proceedings of the Workshop",

note = "22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, BioNLP 2023 ; Conference date: 13-07-2023",

}

ISARIC Clinical Characterisation Group, Rohanian, O, Jauncey, H, Nouriborji, M, Chauhan, VK, Gonçalves, BP, Kartsonaki, C, Merson, L & Clifton, D 2023, Using Bottleneck Adapters to Identify Cancer in Clinical Notes under Low-Resource Constraints. in D Demner-fushman, S Ananiadou & K Cohen (eds), BioNLP 2023 - BioNLP and BioNLP-ST, Proceedings of the Workshop. Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 62-78, 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, BioNLP 2023, Toronto, Canada, 13/07/23. https://doi.org/10.18653/v1/2023.bionlp-1.5

Using Bottleneck Adapters to Identify Cancer in Clinical Notes under Low-Resource Constraints. / ISARIC Clinical Characterisation Group; Rohanian, Omid; Jauncey, Hannah et al.
BioNLP 2023 - BioNLP and BioNLP-ST, Proceedings of the Workshop. ed. / Dina Demner-fushman; Sophia Ananiadou; Kevin Cohen. 2023. p. 62-78 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution book

TY - GEN

T1 - Using Bottleneck Adapters to Identify Cancer in Clinical Notes under Low-Resource Constraints

AU - ISARIC Clinical Characterisation Group

AU - Rohanian, Omid

AU - Jauncey, Hannah

AU - Nouriborji, Mohammadmahdi

AU - Chauhan, Vinod Kumar

AU - Gonçalves, Bronner P.

AU - Kartsonaki, Christiana

AU - Merson, Laura

AU - Clifton, David

PY - 2023/7/31

Y1 - 2023/7/31

N2 - Processing information locked within clinical health records is a challenging task that remains an active area of research in biomedical NLP. In this work, we evaluate a broad set of machine learning techniques ranging from simple RNNs to specialised transformers such as BioBERT on a dataset containing clinical notes along with a set of annotations indicating whether a sample is cancer-related or not. Furthermore, we specifically employ efficient fine-tuning methods from NLP, namely, bottleneck adapters and prompt tuning, to adapt the models to our specialised task. Our evaluations suggest that fine-tuning a frozen BERT model pre-trained on natural language and with bottleneck adapters outperforms all other strategies, including full fine-tuning of the specialised BioBERT model. Based on our findings, we suggest that using bottleneck adapters in low-resource situations with limited access to labelled data or processing capacity could be a viable strategy in biomedical text mining. The code used in the experiments are going to be made available at [LINK ANONYMIZED].

AB - Processing information locked within clinical health records is a challenging task that remains an active area of research in biomedical NLP. In this work, we evaluate a broad set of machine learning techniques ranging from simple RNNs to specialised transformers such as BioBERT on a dataset containing clinical notes along with a set of annotations indicating whether a sample is cancer-related or not. Furthermore, we specifically employ efficient fine-tuning methods from NLP, namely, bottleneck adapters and prompt tuning, to adapt the models to our specialised task. Our evaluations suggest that fine-tuning a frozen BERT model pre-trained on natural language and with bottleneck adapters outperforms all other strategies, including full fine-tuning of the specialised BioBERT model. Based on our findings, we suggest that using bottleneck adapters in low-resource situations with limited access to labelled data or processing capacity could be a viable strategy in biomedical text mining. The code used in the experiments are going to be made available at [LINK ANONYMIZED].

KW - Computational linguistics

KW - Natural language processing systems

KW - Clinical notes

UR - https://www.scopus.com/pages/publications/85171966867

UR - https://aclanthology.org/2023.bionlp-1.5/

U2 - 10.18653/v1/2023.bionlp-1.5

DO - 10.18653/v1/2023.bionlp-1.5

M3 - Conference contribution book

AN - SCOPUS:85171966867

T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics

SP - 62

EP - 78

BT - BioNLP 2023 - BioNLP and BioNLP-ST, Proceedings of the Workshop

A2 - Demner-fushman, Dina

A2 - Ananiadou, Sophia

A2 - Cohen, Kevin

T2 - 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, BioNLP 2023

Y2 - 13 July 2023

ER -

ISARIC Clinical Characterisation Group, Rohanian O, Jauncey H, Nouriborji M, Chauhan VK, Gonçalves BP et al. Using Bottleneck Adapters to Identify Cancer in Clinical Notes under Low-Resource Constraints. In Demner-fushman D, Ananiadou S, Cohen K, editors, BioNLP 2023 - BioNLP and BioNLP-ST, Proceedings of the Workshop. 2023. p. 62-78. (Proceedings of the Annual Meeting of the Association for Computational Linguistics). doi: 10.18653/v1/2023.bionlp-1.5

Using Bottleneck Adapters to Identify Cancer in Clinical Notes under Low-Resource Constraints

Abstract

Publication series

Conference

UN SDGs

Keywords

Access to Document

Other files and links

Fingerprint

Cite this