For Pete’s Sake: Measuring Frustration

If you’ve ever met me then this is obvious, but for those who haven’t: I’m not a patient person. Patience may be a virtue, but unless reincarnation is a thing, I’ve got one life to live and I don’t like to waste it. Did you know that over the average lifespan, you will spend six months of your life waiting in line, for your plane, train, etc¹? That’s half a year wasted, in my opinion. 

Thanks to breathing exercises and new age music, I’m a work in progress and getting better with age; however, one thing that really tests all those breathing exercises is traffic. I loathe traffic so much that I work from home! This is worse in Cincinnati, which consistently ranks as having the worst drivers in the US². For any Cincy folks reading this, you know who you are; the left lane is for passing, not cruising five miles per hour below the speed limit! Whew. Breathe in, breathe out. I’m ok. 

The point I’m trying to make is that for me, traffic is very frustrating. It causes a host of emotions: anger, anxiety, stress, etc, very much akin to the opening scene to Office Space³. As part of my own self-reflection into why I am so bothered by things like traffic, I began to peel back the onion, so to speak. We begin there: what exactly is frustration?

Our emotional reactions

First things first: Frustration is not anger. It’s related to anger, but also disappointment and annoyance, and arises when one’s needs are blocked. Put simply: it’s an emotional reaction to things (internal or external) getting in your way. 

Hi, my name is Jason and I have control issues whereby I don’t like it when people or things block my ability to get to where I’m going. Granted, driving isn’t the only source of frustration. Spouses, kids, co-workers, video games, etc, all can create frustrations at times. 

It’s perfectly natural and I know my Aussiedoodle, Winston, certainly experiences his own version of frustration, especially when I take too long to get ready for his walks. He huffs and grumbles rather loudly. It’s rather funny. But I digress. 

The influence of frustration

If frustration is natural, then why does it matter? We all experience it, so why can’t we just focus on the soothing sounds of Peter Kater, David Arkenstone, or my favorite, Constancy Demby? Well, within the field of human factors and usability, frustration poses an interesting challenge. For example, if I’m shopping online and can’t find that for which I’m looking, how likely am I to continue shopping on a website? If I get frustrated, not very likely, especially for someone like me. 

This goes for any task, be it computer-based, hand-held, wearable, etc (for an excellent summary of frustration in HCI, see Euijung Yang’s thesis work⁴). What is important is that frustration can be a barrier for learning. 

Minimizing frustration is the key to creating a better user interface, smartphone app, retail experience, etc. Great, so we all experience frustration, but minimizing it can help us learn quicker and find greater satisfaction with our technological world. But in order to minimize something, we have to look at how we measure it. 

How to measure frustration

Of course, we could put someone in a scenario and then ask them how he/she feels about it. Easy enough. Ride along with me on Beechmont Ave at 5pm on a rainy Friday and I’ll tell you all about how I’m feeling, and how I feel about everyone around me! But beyond simple self-report, we can start by looking at someone’s physiological responses, as frustration is characterized by changes in GSR and heart rate⁵. 

Think to the last time your flight was delayed, you had a tight connection, and you know there is nothing you can do about it. You might miss that connection, miss that meeting, or your kids soccer game. As you sit there and ponder all the ways you cannot change your situational outcome, how was your breathing, pulse, and overall mood? As someone who lacks the Zen-like patience of my colleague, Peter, I can tell you that my breathing is quickened and my palms are sweaty. 

More than just ‘basic’ emotions

Another option would be to scan someone’s brain while performing a frustrating task⁶. As a neuroscientist, I think that’s super cool. Brain scans are fun. But who has access to a magnet? In fact, some of my friends tell me that waiting for MRI time can itself be a frustrating experience. So let’s think simpler than sensors, wires, and magnets. If you’ve ever read anything I’ve written, you know my preferred strategy. 

You guessed it: measure someone’s facial expression. But, Jason, I hear you saying, I thought automated facial expression analysis only measured the “basic” emotions (happy, sad, angry, surprised, scared, disgusted)? Well good news, dear reader, not anymore.  

Measure frustration by using facial expression analysis

A recent update to FaceReader now allows researchers the opportunity to design your own algorithms by combining variables such as facial expressions, Action Units (AU), and even heart rate to create “Custom Expressions”. In English that means you can create your own affective attitudes, emotional states, or whatever you choose to call it. 

FREE WHITE PAPER: FaceReader methodology

Download the free FaceReader methodology note to learn more about facial expression analysis theory.

• How FaceReader works
• More about the calibration
• Insight in quality of analysis & output

 Download

Remember a previous post looking at false vs felt smiles? To refresh your memory, the FaceReader analytical outputs were imported into The Observer XT for subsequent analysis. Now we can calculate those expressions directly in FaceReader. Pretty cool, eh? So let’s look at an example. 

Facial Action Coding System (FACS) tells us more about how and why the face moves

FaceReader can measure twenty different facial muscles, known as Action Units, or AUs. A real smile combines the mouth (AU12) with the eyes (AU6 and/or AU7). Going back to frustration, recent studies have shown the engagement of both AU4, the ‘brow lowerer’ muscle between your eyebrows, and AU14, the ‘dimpler’ ⁷. 

Try it for yourself: furrow the ‘unibrow’ muscle between your eyebrows, and pull back on side of your mouth, like a contemptuous face. Finish it by putting your hands on your head and ask someone to guess what expression you’re showing. 

Now, in the interest of your health, take the next few minutes to smile and combat any unintentional negative influences from making a frustrated face. It’s ok, I’ll wait. 

Good? Ok. Funny enough, there is also data to show that people sometimes smile when they’re frustrated ⁸. In my experience, that’s usually accompanied by sarcasm or a cynical remark. But even in my own research or projects we’ve done for companies, I’ve noticed that some people indeed smile or chuckle when they eat something they don’t like or can’t figure out how to accomplish their task. 

Automatically analyze facial expressions with FaceReader

Important disclaimer: I am not a facial expression researcher. I’m a curious behavioral neuroscientist with access to FaceReader, the most powerful facial expression analysis software in the market. That means that I leave it for you to decide what exactly constitutes a frustrated face. 

The point is that you can now automate the analysis of such without the need for heart rate sensors, expensive MRI or MEG machines. Imagine the possibilities of what you can do with a simple camera instead of wires, sensors, and pesky artifacts? Now measuring facial expressions does not have to become a frustrating experience! See what I did there? 

References

1. https://logisticsmgepsupv.wordpress.com/2017/04/04/how-much-time-of-an-average-life-is-spent-waiting/
2. https://www.allstate.com/americas-best-drivers/index.htm
3. https://www.youtube.com/watch?v=fk2YRpLnmdU
4. Yang, E. & Dorneich, M.C. (2018). Evaluating Human–Automation Etiquette Strategies to Mitigate User Frustration and Improve Learning in Affect-Aware Tutoring. Applied Sciences, 8 (6), 895, https://doi.org/10.3390/app8060895
5. Jost, H. (1941). Some Physiological Changes during Frustration. Child Development, 12 (1), 9-15
6. Bierzynska, M.; Bielecki, M.; Marchewka, A.; Debowska, W.; Duszyk, A.; Zajkowski, W.; Falkiewicz, M.; Nowicka, A.; Strelau, J. & Kossut, M. (2016). Effect of frustration on brain activation pattern in subjects with different temperament. Frontiers in Psychology, 6. https://doi.org/10.3389/fpsyg.2015.01989 
7. Grafsgaard, J.F.; Wiggins, J.B.; Boyer, K.E.; Wiebe, E.N. & Lester, J.C. (2013). Automatically Recognizing Facial Expression: Predicting Engagement and Frustration. Retrieved from: https://www.intellimedia.ncsu.edu/wp-content/uploads/grafsgaard-edm-2013.pdf
8. http://hdl.handle.net/1721.1/66534