More User Testing in the Time of Pandemic

In the first part of this article, we talked about the importance of planning and what to focus on as you draw the grand scheme of your user test. We also outlined some differences between A/B testing as an automated match-up done mostly with real apps and usability testing allowing you to test even with medium-fidelity artifacts to avoid code waste. In this second part, we will introduce additional best practices that will save you a lot of headaches and time: creating a script, testing your test, and having a broader strategy for user testing.

Use a Script 

People often ask about the most important quality of a good designer and my response is always empathy. Our biggest strength often becomes our greatest weakness and that is the case with empathy while running a user test face to face. As researchers, we can become trapped in users’ emotions and frustrations, making subtle changes in the conditions that affect the outcome of the test without even noticing. But by creating a script and sticking to it, we can alleviate this. And doing so is even more crucial in the new normal. Remote user testing, without direct moderation, requires a script to help guide participants, making sure they are subjected to analogous conditions. This alleviates potential areas of frustration, while also preventing unintended bias from spoiling the outcome of the test.

A good script should prepare the user for the study with clear expectations of what lies ahead and how long it is going to last. It should provide a scenario for each task and may give additional hints or guidance upon user request. And finally, it should contain some closing remarks and clarifications about the test that also express gratitude for people’s time and effort. A way to implement your script via the user testing mechanisms of Indigo.Design is to first define your tasks, providing a scenario for each one, and then to write the welcome and thank you messages for your participants. As we also mentioned in part one, it is a good idea to add a feedback survey link in the closing notes of your study. Having a script is not a silver bullet for running user tests smoothly.

Run Pilot Tests ️

No matter how good our plans are, occasionally they will fail us. To avoid disappointment, we must test our user tests (the play of words is intentional). With just one or two pilot runs of your user test, you will learn and be prepared for any potential misunderstandings and frustrations that users in your larger study might experience. This lets you adjust your script and plans accordingly making the whole experience more pleasant and predictable, while at the same time your data will be cleaner and more accurate.

"Plans are nothing; planning is everything"
Dwight Eisenhower

In our team, we always run our user tests internally before moving on to representative users. Even a fellow designer can provide constructive feedback about the clarity of the script, task scenarios, and questions in the feedback form. Such pilot tests may add a couple of hours to your preparation but always end up saving a much larger waste of time because as humans we are burdened not to see our own mistakes. And speaking about planning, it is time to broaden this activity’s horizon and arrive at the final best practice that I would like to share.

Do it Regularly 

At the end of the day, everything boils down to having a bigger strategy when it comes to user testing, which is the only practice not affected by the presence or absence of a pandemic. Collecting feedback and conducting studies can truly take you from failure to success but only after you stop treating it like a one-time-event. Once you start consistently following these practices and executing them regularly driven by a long-term effort to learn and improve, you will reap the full benefit of user testing. Consider your product roadmap, underline the strategic milestones on it, reinforce them with the right type and amount of user testing, and you are on track to success.

I believe that by sharing our team’s best practices in this two-part article we can inspire others to commit to user testing and do it with the right mindset. Remember to create a plan for your experiment with a hypothesis and proper setup, consider different ways for collecting user feedback, but always one that combines quantitative and qualitative input. Always prepare a script and make sure users stick to it, run pilot tests to make sure everything is easy to comprehend and correctly understood. And last but not least, always think broad and strategic about your efforts with a commitment to continuously dedicate time for user testing. Godspeed for your upcoming user tests my fellow UX designers and developers!