What are the most common user testing biases?

2 Hawthorne effect

This is the phenomenon where people change their behavior or performance when they know they are being observed or evaluated. The Hawthorne effect can make the user testing results unreliable, as the users might act differently than they would in a natural setting. For example, a user might try to please the tester, avoid mistakes, or exaggerate their reactions. To mitigate the Hawthorne effect, you should try to create a comfortable and realistic environment for the user testing, explain the purpose and process clearly, and encourage honest and constructive feedback.

3 Observer effect

This is the influence that the presence or actions of the tester have on the user's behavior or performance. The observer effect can also skew the user testing results, as the tester might unintentionally guide, distract, or pressure the user. For example, a tester might give hints, cues, or feedback that affect the user's actions, or a tester might stare, interrupt, or rush the user. To avoid the observer effect, you should follow a consistent and structured protocol, limit your interactions and interventions, and use remote or unmoderated methods when possible.

4 Social desirability bias

This is the tendency to answer questions or behave in a way that is perceived as socially acceptable or desirable, rather than truthful or authentic. Social desirability bias can compromise the validity and quality of the user testing feedback, as the users might hide their opinions, preferences, or emotions. For example, a user might give positive or polite comments, even if they are dissatisfied or frustrated with the product, or a user might avoid expressing their needs, concerns, or suggestions. To overcome social desirability bias, you should build rapport and trust with the user, use indirect or anonymous methods, and ask for specific and concrete examples.

5 Novelty effect

This is the tendency to react more positively or enthusiastically to something new or unfamiliar, rather than something routine or familiar. Novelty effect can inflate the user testing results, as the users might overestimate their interest, satisfaction, or engagement with the product. For example, a user might rate the product highly or use it frequently, just because it is novel or different, but not because it is useful or valuable. To counteract the novelty effect, you should test the product over a longer period of time, compare it with existing or alternative solutions, and measure the actual outcomes and impacts.

6 Memory bias

This is the distortion or alteration of the user's memory or recall of their experience with the product, due to various factors such as time, emotions, or expectations. Memory bias can affect the user testing results, as the users might misremember or forget some aspects of the product or their feedback. For example, a user might recall only the positive or negative features, or a user might change their opinion or impression over time. To prevent memory bias, you should collect the feedback as soon as possible, use multiple sources and methods, and document and review the data carefully.

7 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?