Should you use a combination of modes to optimize behavioral studies?

If you ask a scientist what they think of when you say, ‘behavioral study’, they will come up with wildly differing answers, depending on their own type of work. Behavioral studies of people can be roughly divided into four categories:

• Observational studies are about watching and recording people’s behaviors.
• Implicit measures are measurements of physiology, facial expressions, etc.
• Explicit measures are where you ask individual test participants or subjects questions either during a task or afterwards.
• Surveys are similar to explicit measures but in this case you ask questions to large groups of people and the questions are not normally related to a task.

Each type has specific advantages and disadvantages.

Observational studies

These are the classic type of study in psychological research. They are objective (and therefore potentially unbiassed), take place in controlled conditions, can be replicated and are often designed to test a falsifiable hypothesis (and so fall under the usual definition of scientific experiments).

However, they can be more difficult to interpret than might appear at first sight and can be subject to various biases. Many classic psychological experiments have proved difficult to replicate.

Stanford marshmallow experiment

For example, the famous Stanford marshmallow experiment first carried out in the 60s and published in 1970 [1] was designed to be a test of will power. Young children were offered marshmallows and told that if they didn’t eat the one in front of them that they would get two later. The children who did better in the test, scored better in school tests ten years later. That was called delayed gratification.

During the experiment itself, it became clear that the successful children were the ones who devised techniques to distract themselves (one child even succeeded in falling asleep), not those with the greatest will power. The children were also not a representative sample of the population, they were all from the university day care center.

Later it was shown that results varied enormously across different countries [2], proving that the results were at least in part culturally determined and that if family background was corrected for, the correlation with doing better in school tests disappeared.

Standard in psychology

Nevertheless, if enough attention is paid to the study design (and scientists have learnt a lot about that since the ‘60s), it is possible to gain a lot of insights and this type of research has been the standard in psychology for the past decades. Psychologists often use The Observer XT for those studies, either as a stand-alone program or as part of a complete observational lab.

Explicit measures

A disadvantage of a purely observational study is that it is difficult to know what is going on inside the subjects’ heads. One solution to that is simply to ask the participants in order to get their subjective insights. That can be done after the experiment, or at critical moments in a task (‘how do you feel now?’).

In longer studies, the questions can be put at set times each day or related to the behaviors or physiological signals of the subjects (’why did you do that?’). That is sometimes referred to as ecological momentary assessment (EMA) if it is carried out whilst the subjects’ go about their daily life.

However, this method also has more problems than you might think at first. If the questions are put whilst the subjects are carrying out a task, they can distract from the task itself. If carried out afterwards, studies have shown that the responses relate primarily to the last few minutes and not the entire task. Subjects will subconsciously give answers to that they think the researchers want to hear (out of a desire to please) or give answers that they think gives a better impression of themselves (studies on unhealthy eating are notorious in that respect).

Mental states

It is also surprising how little insight people normally have into their own mental states. I remember hearing a talk at Measuring Behavior in which Jennifer Healey told us that a subject had heard gunfire whilst driving along and said that they were completely unbothered by it, whereas in fact their heart rate went through the roof.

Surveys

A big disadvantage of the above two methods is that it is hard to know if their findings apply to the general population or only the group on which the study was carried out (which is typically psychology students). A solution to that is to ask questions to a much larger representative group by means of a survey. This can go out to hundreds of people living in many different places and circumstances.

However, those questions usually cannot be related to a task, and they still have the disadvantage that the answers may not be both honest and with sufficient self-understanding. If asked about the price of a product, many people will say that it is too expensive, even if they don’t really believe that, because they don’t want the manufacturer to put the price up.

Psychologists often refer to the questionnaires that they use in surveys or in studies with small groups as validated. However, remarkably often that turns out to mean that they give the same results as other questionaries, rather than be validated by an independent measure. Surveys comparing attitudes in different countries or cultures need to be treated with particular caution, as the interpretation of questions is often culturally dependent, and even more so if the survey is translated.

Surveys are particularly useful for following trends and other changes over time. It is unlikely that their biases will change from one year to the next, so an increase in whatever they are asking about is probably a real increase.

Implicit (objective) measures

In the past, measures like physiology (heart rate, skin conductance, etc), eye-tracking, EEG, fNIRS, and facial expression analysis were largely confined to laboratory studies of subjects more or less sitting still. However, in recent years, those technologies have become increasingly mobile. In addition, technologies like NoldusHub have made them much easier to use.

Their big advantage is that they are entirely objective and therefore much less susceptible to all kinds of biases.

Another advantage is that they can pick up effects which explicit questionnaires miss. For example, de Wijk and Noldus [3] carried out an experiment where they measured the eating experience of participants in a laboratory and the participants’ home, with everything else (food, plates, cutlery, measurement devices, etc) kept exactly the same. There was almost no effect on the explicit measures but a large effect on heart rate and facial expressions.

They then tried to see if they could replicate that effect with an experiment in which the participants were in a laboratory, a simulated café and a real café. The replication succeeded: the explicit measures showed no effects of location, but the implicit measures did show an effect.

However, use of just implicit measures alone is often insufficient, and the technical needs of the measurement can form a constraint for the experimental design.

Mixed-methods design: optimizing objectivity and insights

It is clear that the researcher should not just use one measurement technique, but wherever possible use multiple ways of measuring the responses of their subjects to the experimental conditions. This will both give more insights and more objective results. Of course, if a fundamental element of good experimental design like an unrepresentative group of subjects or insufficient replication (or pseudoreplication) is not taken care of, then the results might still be misleading. Nevertheless, a mixture of methods can be very useful in optimizing but the objectivity and insights.

References

1. Mischel, Walter; Ebbesen, Ebbe B. (1970). "Attention in delay of gratification". Journal of Personality and Social Psychology. 16 (2): 329–337. doi:10.1037/h0029815.

2. Yanaoka, Kaichi; Michaelson, Laura E.; Guild, Ryan Mori; Dostart, Grace; Yonehiro, Jade; Saito, Satoru; Munakata, Yuko (22 June 2022). "Cultures Crossing: The Power of Habit in Delaying Gratification". Psychological Science. 33 (7): 1172–1181. doi:10.1177/09567976221074650.

3. René A. de Wijk, Lucas P.J.J. Noldus (2021), Using implicit rather than explicit measures of emotions, Food Quality and Preference 92, 104125. doi: 10.1016/j.foodqual.2020.104125.