There are significant differences among artificial intelligence large language models when answering scientific questions

Francisco Javier Álvarez-Martínez; Luis Esteban; Lucas Frungillo; Estefanía Butassi; Alessandro Zambon; María Herranz-López; Mario Aranda; Federica Pollastro; Anne Sylvie Tixier; Jose V Garcia-Perez; David Arráez-Román; Andrew Ross; Pedro Mena; Ru Angelie Edrada-Ebel; James Lyng; Vicente Micol; Fernando Borrás-Rocher; Enrique Barrajón-Catalán

doi:10.3389/frai.2025.1664303

There are significant differences among artificial intelligence large language models when answering scientific questions

Francisco Javier Álvarez-Martínez, Luis Esteban, Lucas Frungillo, Estefanía Butassi, Alessandro Zambon, María Herranz-López, Mario Aranda, Federica Pollastro, Anne Sylvie Tixier, Jose V Garcia-Perez, David Arráez-Román, Andrew Ross, Pedro Mena, Ru Angelie Edrada-Ebel, James Lyng, Vicente Micol, Fernando Borrás-Rocher, Enrique Barrajón-Catalán

Strathclyde Institute Of Pharmacy And Biomedical Sciences

Research output: Contribution to journal › Article › peer-review

Abstract

Introduction: This study investigates the efficacy of large language models (LLMs) for generating accurate scientific responses through a comparative evaluation of five prominent free models: Claude 3.5 Sonnet, Gemini, ChatGPT 4o, Mistral Large 2, and Llama 3.1 70B.

Methods: Sixteen expert scientific reviewers assessed these models in terms of depth, accuracy, relevance, and clarity.

Results: Claude 3.5 Sonnet emerged as the highest scoring model, followed by Gemini, with notable variability among the other models. Additionally, retrieval-augmented generation (RAG) techniques were applied to improve LLM performance, and prompts were refined to improve answers. The results indicate that although LLMs such as Claude 3.5 Sonnet have potential for scientific tasks, other models may require more development or additional prompt engineering to reach comparable accuracy. Reviewers’ perceptions of artificial intelligence (AI) utility and trustworthiness showed a positive shift after evaluation. However, ethical concerns, particularly with respect to transparency and disclosure, remained consistent.

Discussion: The study highlights the need for structured frameworks for evaluating LLMs and ethical considerations essential for responsible AI integration in scientific research. These findings should be interpreted with caution, as the limited sample size and domain-specific focus of the exam questions restrict the generalizability of the results.

Original language	English
Article number	1664303
Number of pages	11
Journal	Frontiers in Artificial Intelligence
Volume	8
DOIs	https://doi.org/10.3389/frai.2025.1664303
Publication status	Published - 9 Oct 2025

Funding

The author(s) declare that financial support was received for the research and/or publication of this article. This work was supported by RTI2018-096724-B-C22, PID2021-125188OB-C32 and TED2021-129932B-C21 projects funded by Spanish Ministry of Economy and Competitiveness and CB12/03/30038 (CIBER Fisiopatologia de la Obesidad y la Nutrición, CIBERobn, Instituto de Salud Carlos III). This research was also funded by Generalitat Valenciana (grant number PROMETEO/2021/059) and Agencia Valenciana de la Innovación (grant number INNEST/2022/103). FÁ-M were supported by “Margarita Salas” fellowship from the Spanish Ministry of Universities. EB-C was supported by the “Requalification for university teachers grant” from the Spanish Ministry of Universities and European Union Next Generation program.

Keywords

Artificial Intelligence
scientific evaluation
prompt engineering
large language models
retrieval-augmented Generation

Access to Document

10.3389/frai.2025.1664303Licence: CC BY 4.0

Alvarez-Martinez-FiAI-2025-significant-differences-among-artificial-intelligence-large-language-models-when-answering-scientific-questionsFinal published version, 1.88 MBLicence: CC BY 4.0

Cite this

Álvarez-Martínez, F. J., Esteban, L., Frungillo, L., Butassi, E., Zambon, A., Herranz-López, M., Aranda, M., Pollastro, F., Tixier, A. S., Garcia-Perez, J. V., Arráez-Román, D., Ross, A., Mena, P., Edrada-Ebel, R. A., Lyng, J., Micol, V., Borrás-Rocher, F., & Barrajón-Catalán, E. (2025). There are significant differences among artificial intelligence large language models when answering scientific questions. Frontiers in Artificial Intelligence , 8, Article 1664303. https://doi.org/10.3389/frai.2025.1664303

@article{6cd78d07966241748839642cae33bde2,

title = "There are significant differences among artificial intelligence large language models when answering scientific questions",

abstract = "Introduction: This study investigates the efficacy of large language models (LLMs) for generating accurate scientific responses through a comparative evaluation of five prominent free models: Claude 3.5 Sonnet, Gemini, ChatGPT 4o, Mistral Large 2, and Llama 3.1 70B. Methods: Sixteen expert scientific reviewers assessed these models in terms of depth, accuracy, relevance, and clarity. Results: Claude 3.5 Sonnet emerged as the highest scoring model, followed by Gemini, with notable variability among the other models. Additionally, retrieval-augmented generation (RAG) techniques were applied to improve LLM performance, and prompts were refined to improve answers. The results indicate that although LLMs such as Claude 3.5 Sonnet have potential for scientific tasks, other models may require more development or additional prompt engineering to reach comparable accuracy. Reviewers{\textquoteright} perceptions of artificial intelligence (AI) utility and trustworthiness showed a positive shift after evaluation. However, ethical concerns, particularly with respect to transparency and disclosure, remained consistent. Discussion: The study highlights the need for structured frameworks for evaluating LLMs and ethical considerations essential for responsible AI integration in scientific research. These findings should be interpreted with caution, as the limited sample size and domain-specific focus of the exam questions restrict the generalizability of the results.",

keywords = "Artificial Intelligence, scientific evaluation, prompt engineering, large language models, retrieval-augmented Generation",

author = "{\'A}lvarez-Mart{\'i}nez, \{Francisco Javier\} and Luis Esteban and Lucas Frungillo and Estefan{\'i}a Butassi and Alessandro Zambon and Mar{\'i}a Herranz-L{\'o}pez and Mario Aranda and Federica Pollastro and Tixier, \{Anne Sylvie\} and Garcia-Perez, \{Jose V\} and David Arr{\'a}ez-Rom{\'a}n and Andrew Ross and Pedro Mena and Edrada-Ebel, \{Ru Angelie\} and James Lyng and Vicente Micol and Fernando Borr{\'a}s-Rocher and Enrique Barraj{\'o}n-Catal{\'a}n",

year = "2025",

month = oct,

day = "9",

doi = "10.3389/frai.2025.1664303",

language = "English",

volume = "8",

}

Álvarez-Martínez, FJ, Esteban, L, Frungillo, L, Butassi, E, Zambon, A, Herranz-López, M, Aranda, M, Pollastro, F, Tixier, AS, Garcia-Perez, JV, Arráez-Román, D, Ross, A, Mena, P, Edrada-Ebel, RA, Lyng, J, Micol, V, Borrás-Rocher, F & Barrajón-Catalán, E 2025, 'There are significant differences among artificial intelligence large language models when answering scientific questions', Frontiers in Artificial Intelligence , vol. 8, 1664303. https://doi.org/10.3389/frai.2025.1664303

TY - JOUR

T1 - There are significant differences among artificial intelligence large language models when answering scientific questions

AU - Álvarez-Martínez, Francisco Javier

AU - Esteban, Luis

AU - Frungillo, Lucas

AU - Butassi, Estefanía

AU - Zambon, Alessandro

AU - Herranz-López, María

AU - Aranda, Mario

AU - Pollastro, Federica

AU - Tixier, Anne Sylvie

AU - Garcia-Perez, Jose V

AU - Arráez-Román, David

AU - Ross, Andrew

AU - Mena, Pedro

AU - Edrada-Ebel, Ru Angelie

AU - Lyng, James

AU - Micol, Vicente

AU - Borrás-Rocher, Fernando

AU - Barrajón-Catalán, Enrique

PY - 2025/10/9

Y1 - 2025/10/9

N2 - Introduction: This study investigates the efficacy of large language models (LLMs) for generating accurate scientific responses through a comparative evaluation of five prominent free models: Claude 3.5 Sonnet, Gemini, ChatGPT 4o, Mistral Large 2, and Llama 3.1 70B. Methods: Sixteen expert scientific reviewers assessed these models in terms of depth, accuracy, relevance, and clarity. Results: Claude 3.5 Sonnet emerged as the highest scoring model, followed by Gemini, with notable variability among the other models. Additionally, retrieval-augmented generation (RAG) techniques were applied to improve LLM performance, and prompts were refined to improve answers. The results indicate that although LLMs such as Claude 3.5 Sonnet have potential for scientific tasks, other models may require more development or additional prompt engineering to reach comparable accuracy. Reviewers’ perceptions of artificial intelligence (AI) utility and trustworthiness showed a positive shift after evaluation. However, ethical concerns, particularly with respect to transparency and disclosure, remained consistent. Discussion: The study highlights the need for structured frameworks for evaluating LLMs and ethical considerations essential for responsible AI integration in scientific research. These findings should be interpreted with caution, as the limited sample size and domain-specific focus of the exam questions restrict the generalizability of the results.

AB - Introduction: This study investigates the efficacy of large language models (LLMs) for generating accurate scientific responses through a comparative evaluation of five prominent free models: Claude 3.5 Sonnet, Gemini, ChatGPT 4o, Mistral Large 2, and Llama 3.1 70B. Methods: Sixteen expert scientific reviewers assessed these models in terms of depth, accuracy, relevance, and clarity. Results: Claude 3.5 Sonnet emerged as the highest scoring model, followed by Gemini, with notable variability among the other models. Additionally, retrieval-augmented generation (RAG) techniques were applied to improve LLM performance, and prompts were refined to improve answers. The results indicate that although LLMs such as Claude 3.5 Sonnet have potential for scientific tasks, other models may require more development or additional prompt engineering to reach comparable accuracy. Reviewers’ perceptions of artificial intelligence (AI) utility and trustworthiness showed a positive shift after evaluation. However, ethical concerns, particularly with respect to transparency and disclosure, remained consistent. Discussion: The study highlights the need for structured frameworks for evaluating LLMs and ethical considerations essential for responsible AI integration in scientific research. These findings should be interpreted with caution, as the limited sample size and domain-specific focus of the exam questions restrict the generalizability of the results.

KW - Artificial Intelligence

KW - scientific evaluation

KW - prompt engineering

KW - large language models

KW - retrieval-augmented Generation

U2 - 10.3389/frai.2025.1664303

DO - 10.3389/frai.2025.1664303

M3 - Article

SN - 2624-8212

VL - 8

JO - Frontiers in Artificial Intelligence

JF - Frontiers in Artificial Intelligence

M1 - 1664303

ER -

There are significant differences among artificial intelligence large language models when answering scientific questions

Abstract

Funding

Keywords

Access to Document

Fingerprint

Cite this