1) Ph. D. of Philology, Associate Professor, Head of the Department of Lin-guistics and Translation, South Ural State University, Russia, Chelyabinsk, babinaoi@susu.ru 2) Master’s Student of the Institute of Linguistics and International Commu-nications, South Ural State University, Russia, Chelyabinsk, anastb6@mail.ru
The paper proposes a method for automating the technical assessment of employees in an IT company using a model for scoring responses to open-ended technical as-sessment test questions. The suggested method employs retrieval augmented genera-tion by the large language model GPT-4o to create a reference answer for the test question. Subsequently, a custom neural network model assesses competencies by comparing a user response with the reference answer, both of which have been vec-torized using a pre-trained Russian-language model FastText. The developed model is implemented as a Siamese neural network based on a multilayer perceptron, which outputs a regressor predicting values ranging from 1 to 10. The experimental part of the study was conducted on a dataset consisting of pairs of questions and answers from the technical assessment test, scored by experts. The model perfor-mance, evaluated using the mean squared error (MSE) on the test split, achieves 0.036, demonstrating a high correlation between the model predictions and expert scores. The developed model and method for automating technical assessment can contribute to optimizing recruitment and monitoring employees’ competencies, as well as providing a deeper understanding of their actual knowledge and skills in the context of rapidly changing technologies.
transformer; generative neural network; large language models; GPT-4o; Siamese network; multilayer perceptron; technical assessment; retrieval augmented genera-tion.
Download textFor citing: Babina O.I., Bykova A.S. (2025) Modeling technical assessment using GPT agent’s responses. Human being: Image and essence. Humanitarian aspects. Moscow. INION RAN.Vol. 2 (62). pp. 156-172. DOI: 10.31249/chel/2025.02.09