A new technological era is emerging, in which the ability to interpret non-verbal expressions becomes a crucial differentiator. While artificial intelligence (AI) and various devices become increasingly present in our daily lives, non-verbal communication continues to be a complex barrier between humans and machines. However, recent initiatives, such as an innovative video game, are paving a way to overcome this difficulty.
A doctoral student at Northeastern University has developed a first-of-its-kind virtual reality game focused on recording players’ emotions. The main objective is to foster empathy towards the LGBTQIA+ community, immersing the participant in the experience of a hero who faces a world full of prejudice. The memories and feelings evoked during the game serve as essential rewards for achieving victory.
It is widely recognized that putting yourself in someone else’s shoes represents the most effective way to overcome disagreements and understand different perspectives, promoting mutual respect. Despite being a challenge in most situations, technology offers a valuable resource for creating immersive experiences. They not only transport the individual out of everyday reality, but also help them to decipher and interact better with the real world.
An image shows Mark Zuckerberg, CEO of Meta, using his smart glasses. There are reports that Meta is testing prototypes of glasses equipped with “super sensors” powered by artificial intelligence, capable of recording every moment experienced by the user.
The virtual reality devices used in this context are capable of capturing facial movements linked to a repertoire of six basic emotions, such as joy, sadness and surprise. Furthermore, they identify fifteen compound emotions, which combine sensations such as “happily surprised”, “sadly angry” or “afraid and disgusted”, revealing complex nuances of human feeling.
The name of the game is Rekindle, and it was designed by Hector Fan, a PhD student in Interdisciplinary Design and Media. Fan counted on the support and collaboration of professors Mark Sivak and Casper Harteveld, both professors at Northeastern University, to carry out the project.
The story of the Rekindle game and the protagonist’s immersion in reality
Rekindle’s narrative unfolds in a dystopian future, where a totalitarian government manipulates memories to eradicate any trace of sexual identities considered dissident or sinful. The player takes on the role of the protagonist, whose mission is to retrieve his own memories to progress and ultimately win the game. These recovered souvenirs serve as collectibles crucial to advancing the story.
By having their emotional responses recorded with each memory discovered, the player becomes deeply involved in the narrative. He establishes an emotional connection with the hero, experiencing the perspective of a person from the LGBTQIA+ community who faces daily discrimination. This methodology turns the game into a powerful tool for building empathy.
A graphic illustrates how the game captures user feelings at Northeastern University. The creators of Rekindle, in their introductory article published at Augmented World Expo, highlight a paradigmatic shift: traditionally, emotions were treated as “input” in games, with players reacting passively. The innovation is in inverting this concept, where the player’s emotions become an active part of the development of the experience.
The game’s story structure is carefully crafted with instructions that motivate the user to actively express the emotions corresponding to each collectible “memory fragment.” This dynamic interaction not only drives progress in the story, but also strengthens the emotional bond with the main character.
A scene from the Rekindle video game is featured in one image, capturing moments of gameplay. In one specific segment, a combat scene alludes to the historic Stonewall riots of 1969 or the recent mass altercations in West Hollywood in March. These memories are collectible and, unlike Mario coins or Pac-Man points, they are designed to represent situations that players can identify with. They evoke significant events that are “culturally shared” within the LGBTQIA+ community.
Rekindle’s innovative methodology can be adapted to countless other technological variables, opening up a promising panorama for the future. In it, technology will assist human beings not only through commands and buttons, but also by interpreting non-verbal information, increasing mutual understanding.
How virtual reality glasses analyze gamer feelings
Virtual reality glasses are capable of identifying facial muscle movements and translating them into “action units” (AUs). For example, raising your eyebrows or squinting your eyes are considered distinct UAs. This information is then fed into a system that classifies each of the 21 possible emotions with a score ranging from 0 to 1, indicating their relative intensity. A score of 0 means the complete absence of a specific emotion at that moment.
For the development of this study, two models of virtual reality headsets were used. The first is Meta Quest 3, released in 2023, which can be seen in use by video game design professor Casper Harteveld. The second model used is one of the most recent releases on the market, the Samsung Galaxy XR.
The following image displays the Samsung Galaxy XR immersive glasses. The South Korean giant announced this device last year, positioning it as a direct competitor to the Apple Vision Pro. Recent tests with the device have shown that, thanks to its advanced sensors, cameras and hardware, it offers accurate tracking of head, hand and eye movements, providing fully immersive experiences. In the context of the Rekindle video game, however, the system is specifically used to analyze players’ emotions, and not just their physical movements.
This model is currently only available in the United States and South Korea, costing US$1,799.99, which is approximately €1,575.89. There is still no concrete information about the launch date or availability of these glasses in Spain and other international markets.
