Fraunhofer IAO
Fraunhofer IAO

CogniAffect: Interactions and Relationships Between Affective and Cognitive States

A neuropsychological study investigating mental states

Challenge

Within the initiative »KI-Fortschrittszentrum LERNENDE SYSTEME UND KOGNITIVE ROBOTIK«, the Team Applied Neurocognitive System at Fraunhofer IAO investigated the decoding and feedback of users’ mental states during a cognitive task with concurrent emotional distraction. Identifying users’ mental states is a decisive task for many human-machine applications (HMI) like in industrial production, semi-autonomous vehicles, medical surgery, or in the context of learning and educational systems. While affective states (i.e., emotions) are related to satisfaction and motivation, cognitive states (e.g., effort and workload) are related to exhaustion, stress, and fatigue. In this study we wanted to identify and continuously monitor these mental states by measuring and decoding brain activity. Furthermore, we explored interactions between affective and cognitive states (c.f., Moore et al., 2019) and developed an intuitive feedback of the recognized states to the users. By providing users with insights on their current affective and cognitive state, we were able to investigate the importance of the feedback’s reliability, appropriateness, and accuracy for the users’ attitude towards the system and their performance. Users’ reactions to the feedback were captured on a neurophysiological, subjective, and behavioral level.

Methodology

During the experiment, participants performed arithmetic tasks of different complexity (high vs. low) with concurrent auditory emotional distractions (negatively, neutrally, and positively associated sounds; see Figure 1). We measured neurophysiological correlates in a 2 (i.e., low vs. high working memory load) × 3 (i.e., low/negative, neutral, and high/positive valence/emotionality) design resulting in six experimental conditions. Based on electroencephalographic (EEG) data of eight participants, we implemented a real-time neurofeedback routine consisting of preprocessing, analyzing, and visualization steps (e.g., Enriquez-Geppert et al., 2017).

Figure 1. Experimental procedure for left: Low Working Memory Load and right: High Working Memory Load.
Figure 1. Experimental procedure for left: Low Working Memory Load and right: High Working Memory Load.

In a subsequent study, we investigated neurophysiological and behavioral effects (i.e., response time and accuracy) of either appropriate or inappropriate (i.e., erroneous), sham visual feedback allegedly based on users’ brain activity (Logemann et al., 2010) in the same 2 × 3 study design. We explore how participants accept and experience the real-time feedback by asking them to correct the feedbacked states according to their own perception by clicking in the respective field (see Figure 2 left). After the experiment, we asked participants in a semi-structured qualitative interview how they perceived the feedback and whether they used it to adapt their behavior.

Figure 2. left: Experimental setup. right: Visualization of the recognized affective and cognitive state.
Figure 2. left: Experimental setup. right: Visualization of the recognized affective and cognitive state.

Results

On a behavioral level, participants answered faster and more accurately in the low working memory load condition than in the high working memory load condition. Furthermore, on a subjective level, participants perceived the high working memory load condition as more demanding. They also perceived the high valence condition as more positive than the neutral valence and the low valence condition (see Figure 3).

Figure 3. left: Performance (accuracy and response time). right: Subjective ratings (effort and affect). HH: High Valence – High Working Memory Load, HL: High Valence – Low Working Memory Load, LH: Low Valence – High Working Memory Load, LL: Low Valence – Low Working Memory Load, NH: Neutral Valence – High Working Memory Load, NL: Neutral Valence – Low Working Memory Load. Error bars = standard deviation.
Figure 3. left: Performance (accuracy and response time). right: Subjective ratings (effort and affect). HH: High Valence – High Working Memory Load, HL: High Valence – Low Working Memory Load, LH: Low Valence – High Working Memory Load, LL: Low Valence – Low Working Memory Load, NH: Neutral Valence – High Working Memory Load, NL: Neutral Valence – Low Working Memory Load. Error bars = standard deviation.

To decode mental states, we used established neuronal correlates of the EEG signals and compared several supervised machine learning methods. We were able to discriminate experimental conditions with high classification accuracy (see Figure 4 left). Interestingly, the classification accuracy decreases to a random level when the annotations are corrected on the basis of the subjective assessments which we collected via questionnaires presented after each stimulus event and arithmetic task (post-hoc rating; see Figure 4 right). Thus, influences not represented in the neurophysiological signals or identified by the classification model seem to affect a subjective evaluation of experienced stimuli and perceived states.

Figure 4. Average classification accuracy of the training set (green) and test set (light green) compared to an empirical baseline (dummy classifier; blue). Features: Channels and frequency bands of the frontal alpha asymmetry and working memory load coefficient (Smith et al., 2017; Käthner et al., 2014). left: Prediction of the experimental condition. right: Prediction of the subjective rating. LR: Logistic Regression, SVM: Support-Vector Machine, KNN: k-Nearest Neighbor, RFC: Random Forest- Classifier, GBC: Gradient Boosting Classifier, GNB: Gaussian Naïve Bayes.
Figure 4. Average classification accuracy of the training set (green) and test set (light green) compared to an empirical baseline (dummy classifier; blue). Features: Channels and frequency bands of the frontal alpha asymmetry and working memory load coefficient (Smith et al., 2017; Käthner et al., 2014). left: Prediction of the experimental condition. right: Prediction of the subjective rating. LR: Logistic Regression, SVM: Support-Vector Machine, KNN: k-Nearest Neighbor, RFC: Random Forest- Classifier, GBC: Gradient Boosting Classifier, GNB: Gaussian Naïve Bayes.

When we showed participants their alleged mental states, participants were more likely to correct odd feedback compared to reasonable one. Whether they received appropriate feedback or inappropriate one, had, however, no significant influence on their performance in the next trial (see Figure 5). Interestingly, increased working memory load did not change the perceived correctness of and probability to adjust an inappropriate feedback score. No differences between the feedback conditions (appropriate vs. inappropriate) could be revealed on a neurophysiological level.

Figure 5. Effects of odd feedback on performance (accuracy and response time) in the subsequent trial. Error bars = standard deviation.
Figure 5. Effects of odd feedback on performance (accuracy and response time) in the subsequent trial. Error bars = standard deviation.

Participants evaluated the feedbacked scores as interesting and asked for a more detailed explanation on the underlying computations. Some perceived the odd feedback as irritating. Still, many participants reported further potentials of adaptive feedback systems to enhance effectiveness in learning and training scenarios. Since participants reported irritation in response to the inaccurate feedback and the wish to get more information regarding the score computation, it is likely that they had limited trust in the feedback system. Probably only systems perceived as reliable, consistent, and relevant, are able to induce effects on a behavioral and neuronal level (Kluger & DeNisi, 1996).

Conclusion

In sum, our real-time EEG-based neurofeedback approach contributes to the development of closed-loop HMI-applications and neuro-adaptive systems allowing to recognize users’ state, provide feedback, and adapt the system behavior to individual capabilities and demands. We also highlighted the importance of feedback reliability and associated challenges and research potentials for robust affective-emotional and cognitive state recognition during the human-machine interaction with a focus on performance-oriented contexts (learning environment, safety-critical control environments, during driving).

References

  • Enriquez-Geppert, S., Huster, R.J., & Herrmann, C.S. (2017). EEG-neurofeedback as a tool to modulate cognition and behavior: A review tutorial. Frontiers in Human Neuroscience, 11(51), 1-19.
  • Käthner, I., Wriessnegger, S. C., Müller-Putz, G. R., Kübler, A., & Halder, S. (2014). Effects of mental workload and fatigue on the P300, alpha and theta band power during operation of an ERP (P300) brain-computer interface. Biological Psychology, 102, 118-129.
  • Kluger, A.N. & DeNisi, A. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254-284.
  • Logemann, H.N.A., Lansbergen, M.M., Van Os, T.W.D.P., Böcker, K.B.E., & Kenemans, J.L. (2010). The effectiveness of EEG-feedback on attention, impulsivity and EEG: A sham feedback controlled study. Neuroscience Letters, 479(1), 49-53.
  • Moore, M., Shafer, A.T., Bakhtiari, R., Dolcos, F., & Singhal, A. (2019). Integration of spatio-temporal dynamics in emotion-cognition interactions: A simultaneous fMRI-ERP investigation using the emotional oddball task. NeuroImage, 202, 116078.
  • Smith, E. E., Reznik, S. J., Stewart, J. L., & Allen, J. J. B. (2017). Assessing and conceptualizing frontal EEG asymmetry: An updated primer on recording, processing, analyzing, and interpreting frontal alpha asymmetry. International Journal of Psychophysiology 111, 98-114.