A study conducted at the Professor Giacomo Rizzolatti Biological Research Laboratory at Francisco de Vitoria University (UFV) has analyzed how different artificial intelligence models interpret complex mental states based on images of faces.
The study, published in the scientific journal Royal Society Open Science, compared the responses of 230 human participants with those of five artificial intelligence models—ChatGPT-4o, Claude, Gemini, Grok, and Mistral—to a task designed to assess processes related to social cognition.
The study did not focus on recognizing basic emotions, such as joy or sadness, but rather on interpreting more complex situations: whether a person appeared distrustful, regretful, threatening, self-confident, or willing to forgive.
A Study on Social Cognition Based on Portraits
To conduct the experiment, the research team selected eight images: three portraits by Diego de Velázquez and five photographs by Arnold Abner Newman and Roger George Clark.
Among the works featured are Portrait of Man, Mother Jerónima de la Fuente, and Saint Paul, along with photographic portraits of Robert Frost, Bill Brandt, Dwight D. Eisenhower, Shimon Peres, and Pablo Picasso.
Each image was paired with a question that had two possible answers. In one of the exercises, for example, participants were asked to imagine that the person in the picture had just witnessed a specific action and to decide whether they thought it was good or bad. In other cases, participants had to choose between concepts such as trust or threat, regret or satisfaction, worry or security, or revenge or forgiveness.
The study’s lead author, Carlota Márquez-Pedregal, a member of the lab and a student pursuingBachelor's Degree Psychology**](https://www.ufv.es/estudiar-grado-psicologia-madrid/) at UFV, explains that the goal was not to test whether a machine is capable of identifying basic emotions, but rather to apply the assessment to situations more closely related to everyday life. “We were interested in that moment when a person looks at a face and interprets—almost without realizing it—whether there is trust, suspicion, guilt, a threat, or moral judgment,” she notes.
The research was conducted under the direction of Dr. Raúl Alelú-Paz, a professor at UFV, director of the Professor Giacomo Rizzolatti Biological Research Laboratory, and the corresponding author of the article.
Beyond Recognizing a Facial Expression
One of the study's main contributions is that it does not ask artificial intelligence models about basic emotions, but rather about social situations that humans consistently interpret.
When looking at a face, people don't necessarily limit themselves to identifying whether someone is sad or happy. They can also infer whether that person is suspicious of someone, expects a certain reaction, feels guilty, or is making a judgment about what they consider right or wrong.
For Dr. Raúl Alelú-Paz, the central issue lies precisely in that difference. It is not enough for a system to detect a facial expression; rather, it is necessary to analyze the extent to which it can approximate the way people attribute intention, trust, or moral judgment based on visual cues.
In other words, the study examines whether artificial intelligence models can respond in a similar way to humans when asked to interpret a social situation based on a face.
AI models exhibit different behaviors
Human participants provided highly consistent responses across the eight dimensions analyzed. As the authors explain, this consistency made it possible to establish a baseline against which to subsequently compare the responses of the various artificial intelligence systems.
The results showed that ChatGPT-4o, Grok, and Gemini formed a cluster very close to the human pattern. Mistral fell somewhere in the middle, with partial overlaps, while Claude exhibited a more distinct profile in some dimensions, particularly in the interpretation of trust and threat.
This variability is one of the study’s most significant findings. The results suggest that there is no single way in which “artificial intelligence” interprets a face. Two different models can analyze the same image and reach different conclusions about what a person appears to be thinking or feeling. These differences could be related to the architecture of each model, the data used during their training, or the criteria employed in their tuning processes.
The issue takes on particular significance given the potential application of these systems in fields such as education, mental health, virtual care, social robotics, and care for the elderly. “It won’t be enough to simply state that artificial intelligence accurately interprets human signals. We’ll need to know which model does so, under what conditions, and within what limits,” notes Carlota Márquez-Pedregal.
Infographic summarizing the study: 230 human participants, five artificial intelligence models, and eight images to compare how complex mental states are attributed based on faces.
Seeming empathetic doesn't mean having empathy
The authors emphasize that the results should be interpreted with caution. The research does not conclude that artificial intelligence possesses empathy in the human sense, but rather that some models can functionally reproduce certain patterns associated with social cognition.
Artificial intelligence can provide a response similar to that of a person because it has learned complex associations between images, language, and social situations. It can identify patterns, select a coherent response, and express it convincingly. However, this does not mean that it has a subjective experience of what it observes or that it understands others in the same way a human does.
This nuance takes on increasing importance as conversational artificial intelligence systems generate responses that can be perceived as attentive, sensitive, or understanding. “The coherence of a response should not be confused with genuine empathy,” the researchers warn.
A New Tool for Evaluating a Sensitive Frontier
The study also addresses a methodological challenge. Many of the classic tests used to assess social cognition were designed for people, not for artificial intelligence systems.
Furthermore, some of these tests may have been part of the data used to train large language models. In that case, a correct answer could be due to prior exposure to the test and not necessarily to a new ability to reason.
To reduce this risk, the team designed a specific task using images and questions selected specifically for the study. This contribution is particularly relevant given the growing use of artificial intelligence in fields where the interpretation of human signals can have practical consequences, such as education, mental health, virtual assistance, and social robotics.
However, the authors themselves point out the study’s limitations. The human sample consisted of young adults between 19 and 28 years old who shared the same cultural and linguistic background. Furthermore, the test included only eight images and binary responses.
“The paradigm is preliminary, and further research will be needed using more diverse samples, a greater number of stimuli, and situations that more closely resemble real-life interactions,” the authors note.
The study does not aim to settle the debate over whether artificial intelligence understands people, but rather to raise an increasingly relevant question: if some models are already capable of accurately reproducing certain human patterns of social interpretation, we will need to determine how to evaluate them before entrusting them with functions that directly affect interpersonal relationships.
