Role of punctuation in semantic mapping between brain and transformer models

Modern neural networks specialised in natural language processing (NLP) are not implemented with any explicit rules regarding language. It has been hypothesised that they might learn something generic about language. Because of this property much research has been conducted on interpreting their inner representations. A novel approach has utilised an experimental procedure that uses human brain recordings to investigate if a mapping from brain to neural network representations can be learned. Since this novel approach has been introduced, more advanced models in NLP have been introduced. In this research we are using this novel approach to test four new NLP models to try and find the most brain aligned model. Moreover, in our effort to unravel important information on how the brain processes text semantically, we modify the text in the hope of getting a better mapping out of the models. We remove punctuation using four different scenarios to determine the effect of punctuation on semantic understanding by the human brain. Our results show that the RoBERTa model is most brain aligned. RoBERTa achieves a higher accuracy score on our evaluation than BERT. Our results also show for BERT that when punctuation was removed a higher accuracy was achieved and that as the context length increased the accuracy did not decrease as much as the original results that include punctuation.