Measuring cognitive load in VR — the power of three

For this eye-for-innovation post, I talked to Julia Kern, director of strategic partnerships at Soma Reality — an Austrian startup developing neuropsychological insights for digital environments.

I met Kern during the latter part of 2022, right about the time when ChatGPT and other generative AI bots were monopolizing the headlines. Fast forward a couple of months to the release of GPT-4 earlier this year, I was struck by the comprehensive changes that this multimodal language model will have on the way that we work, learn, and live. The way GPT-4 “exhibits human-level performance on various professional and academic benchmarks”, has caused me (and I am sure I am not alone here) to reflect on how much we understand human cognitive abilities and how quickly we can enhance our learning efficiency and effectiveness.

When I spoke with Kern, she talked about the rising interest in cognitive load, their first digital biomarker based on eye-tracking, and how her company has leveraged that trend. In this post, I will share Kern’s insights on measuring cognitive load in VR, how education and healthcare stand to benefit, and how the power of three — know-how, VR, and eye tracking — has enabled Soma to build a robust solution for calculating neurological insights for anyone, anywhere.

Rising interest in cognitive load

If you ask ChatGPT for a definition of cognitive load, it responds with the following:

“Cognitive load refers to the amount of mental effort or resources that are required to process information in working memory. It is a measure of how much information a person can effectively process and retain in their memory while completing a task or solving a problem.”

The scientific community is keen to expand understanding of cognitive load because of its broad application, especially in areas like learning, interface design, and healthcare. Public awareness is also a factor. Neurodegenerative diseases are becoming more prevalent as populations age, touching most people’s lives. Governments and organizations are pouring funds into longevity and cognitive-related research, creating opportunities for the tech industry to develop innovative early detection and treatment solutions. But any solution — manual or computerized — that aims to deliver insights about a person’s learning capabilities or their health and wellbeing requires good data and well-established research to ensure positive outcomes.

Measuring cognitive load in VR — the know-how

Kern explains the research into task-invoked pupillary response which shows the linear relationship between pupil size and demand for cognitive processes. As demand rises, our eyes react with tiny involuntary pupil fluctuations — a response we cannot control. She notes that our inability to influence those movements makes pupil size a reliable and quantifiable biomarker for cognitive load.

Light exposure also causes changes in pupil size, known as pupillary light reflex (PLR) — becoming smaller in bright scenarios and larger in dark situations. So, to measure cognitive load accurately in a VR setting, Kern explains the need to separate neurological impact from light-induced effects on pupil size, which you can calculate from the luminance of the VR display.

The VR

As a delivery method, VR brings several things to the table. Its closed environment helps to cut out external influences, so you can use it to assess cognitive load, safe in the knowledge that you’re measuring is indeed CL and not a response to other people, events, or objects. And while its mobility factor brings testing capabilities to the point of need, I think the sense of realism and presence that gives VR an edge over traditional computing. VR puts people in environments as close to reality as possible, meaning that any resulting cognitive load assessment will be as accurate as technically feasible. VR enables you to prepare people for stressful experiences they might encounter in their chosen careers, for example. And while eye tracking can provide the data to calculate cognitive load on other platforms, VR can deliver useful sensory information, such as motion, heart rate, and skin response — without additional hardware or sensors.

While Soma Reality has designed its proprietary cognitive load algorithm to be hardware agnostic, in practical terms, the need for advanced eye tracking signals led Soma to develop with the Pico Neo 3 Pro Eye headset — a platform that has become popular among developers owing to its untethered mobility, processing capabilities, built-in eye tracking from Tobii, and favorable cost/value ratio.

Why partner with Tobii and why Tobii Ocumen — the eye tracking

When asked why they chose Tobii, Kern replied,

“Tobii is already known to many of our partners and clients for its quality and long-standing research in the field of eye tracking, making it the obvious choice for expanding our work in VR. Tobii Ocumen comes with packages for Unity 3D and has a clean interface that makes it easy access to the data – a big plus for everyone working on the development end.”

To assess cognitive load, the Soma algorithm leverages real-time pupil diameter measurements delivered by Tobii Ocumen. Kern says that the robustness and granularity of the data Tobii Ocumen provides, together with the mobility and processing capabilities of the Pico Neo 3 Pro Eye VR headset, is a powerful combination.

Perhaps you are considering developing with Tobii Ocumen and want to know what kind of data you can expect. In that case, I can say that in addition to standard eye tracking data like gaze vector and origin, we’ve designed Tobii Ocumen to deliver advanced metrics like pupil diameter and gaze — separated for each eye. We’ve worked on making the product easy to use by incorporating capabilities like a filter library and record and replay features for data. By measuring the right biomarkers and doing so accurately, we make the most of our expertise and enable solution developers to make the most of their know-how.

 Which brings me to my next point: what can you do with the powerful trio of know-how, VR, and eye tracking?

Learning and skills training

According to Kern, the retention rate for new skills learned in a VR setting can be up to four times that of a traditional one. Soma’s cognitive load algorithm adds the ability to adapt VR learning scenarios to user behavior, optimizing learning to occur when the student is in an ideal learning state. Learning applications can leverage the algo to, for example, prevent students from becoming bored or overloaded by providing feedback about the student’s cognitive state, which an instructor (or the application) can use to adjust content accordingly.

Healthcare and wellbeing

As cognitive load is an indicator of brain activity, it’s possible to use Soma’s technology in various applications, such as personalized therapy, remote clinical trials, and early detection of neurodegenerative diseases. It’s my hope that their solution matures quickly, offering the potential to delay the onset of diseases like Alzheimer's and bring relief to millions of people and their families.

Gaming showcase

Soma has developed two mini-games — Flying Colors and Soma Math — to showcase how cognitive load assessment works in VR. Compared with Train of Thought, a similar game from Lumosity that I used to play on my mobile, I feel that Flying Colors offers a more comprehensive visual chart displaying cognitive load at different levels. This chart can be compared to other games that train different aspects of cognitive ability, providing a more nuanced understanding of cognitive ability.

My takeaway, the power of three

As we talked, it occurred to me how complex tech can be, but when we separate expertise — with Tobii focusing on eye tracking and delivering robust data for everyone, OEMs providing a computationally powerful platform that supports convenience and realistic measurement, and Soma leveraging their knowledge of cognitive processes — we enable the creation of innovative solutions that are life-changing for a lot of people.

Kern also shared their plans to empower next-gen digital applications by making them aware and adaptive. Their roadmap includes experiments to measure cognitive function and develop adaptive training programs that utilize cognitive load assessment to increase engagement and retention. Following up on their initial cognitive load algorithm, Soma intends to expand their portfolio of digital biomarkers to cover attention and other cognitive processes. I eagerly anticipate the results of these projects and experiments and want to thank Julia for taking the time to chat about this fascinating topic.

If you want to learn more about Soma Reality’s solution, use the link in her bio to reach out.