User Testing: Top 5 Myths vs Reality

For every $1 invested in user experience (UX), the ROI is between $2 and $100, notes Forrester Research¹. Why? Because with users now engaging with most brands via web or mobile device, user experience is not just about how to find a button or place an order, it is becoming about how customers define brands for better or worse.

And with COVID-19, that has become even more true. As David Truog, VP, Research at Forrester Research wrote in a recent blog, “With customers worldwide coping with the pandemic, companies need to recalibrate customer experience (CX) efforts. With most people cooped up at home, it’s suddenly more important than ever to get digital experiences right. That means urgently redirecting funds toward the people, processes, and technologies required for digital user experience (UX).”

Why are User Testing Rates Low?

So while user testing seems increasingly important, it is estimated that only about 55% of companies are currently conducting any user experience testing or user research for their applications. Why? In this blog we will unpack some of the reasons offered and turn them around to give you our top 5 reasons not to skip user testing! 

Myth 1: Product Maturity

Many companies say that they know their users and have evolved their products based on their needs.

The Reality:

No matter how well you think you know your customers and users, customers and markets are constantly evolving. New trends in both consumer and business applications are always influencing user expectations. Add in changes in industries, society and geographic regions and you can see trends and transformations quickly reaching the “tipping point.” So how can you be sure that your applications are truly delivering the best user experience based on current user expectations and needs? Consistent, periodic testing.

Myth 2: Agile Development Eliminates Usability Testing Need

Some feel that with agile development they can fix usability issues after release and roll-out an update.

The Reality: 

Fixing a problem in development costs 10x as much as fixing it in design, notes Roger Pressman in his book, Software Engineering: A Practitioner’s Approach, and 100x as much if you’re fixing it in an existing product. That means the earlier you can identify issues with usability, the less expensive it is to fix. We also need to get strategic about what we CAN test before code, and what can be tested with the real app.  Testing with early design artifacts can lead to insights about the product which the team can turn into design improvements. Allowing teams to review test results together builds empathy for your customers and helps everyone understand that we are indeed building software for human beings!

Myth 3: Lack of Resources Needed for Usability Studies

Many companies say they don't set up usability studies because they lack the resources to manage and execute a usability study.

The Reality:

Did you know that developers spend about 50% of their time fixing issues which could have been avoided? That means if you can catch and identify issues in the design or prototype, you reduce the time that your developers spend fixing issues and iterations of the same application and allow them to have more time to focus on crafting good, bug-free code. The net gain: resources are freed up for other tasks. 

Myth 4: Takes Too Much Time

The time required to set up and execute usability studies is another myth that many people believe, some thinking usability studies can take weeks or months to plan and execute.

The Reality:

User testing does not have to be a long, drawn out affair. While some types of testing can take longer to plan and execute than others, UX/UI designers and developers can execute remote, unmoderated user testing to validate designs, test prototypes and ensure usability in a matter of hours or minutes using tools like Indigo.Design.

Myth 5: Usability Testing Costs Too Much

Cost is frequently mentioned as a headwind with user testing, claiming a lack of budget to find a user panel to participate and then paying for the logistics.

The Reality:

If you have the time and budget, conducting broader user testing with a larger user panel can be beneficial. However the truth is that very often 85% of issues related to UX can be detected by performing a usability test on a group of only 5 users². Especially if users represent the same persona and are testing an app using an identical closed-ended task (for example, "add a 40 inch Samsung Flat-Screen TV to the shopping cart"). And for most designers and developers, time and budget are often at a premium. So, the more testing that they can conduct earlier in the app creation process, the more insights and intelligence they can leverage to improve designs and usability and the easier it is to implement those improvements.

Also, some testing services charge not only by the test and the number of users, they also charge for incidentals such as travel facilities and more, leading to test budgets that can quickly balloon and can thus be very limited in their scope. Conducting remote, unmoderated user testing can be done with a much lower fixed cost, using tools such as Indigo.Design, which provides tools to test development, video capture, test analytics and more. All with a fixed cost for an annual subscription, enabling designers and developers to conduct unlimited tests, with unlimited users.

Which brings us back to our main point: with the cost to fix issues dramatically higher in development or post release, the question becomes – How can you afford not to do user testing and ensure the usability of your applications?

If you haven't already, I invite you to try Indigo.Design, a complete design to code system, integrating a design system, prototyping, user testing and code generation in a single common platform for designers and developers. You can try user testing in Indigo.Design free for 14 days, access sample user tests and analytics create your own tests.

1. The Six Steps For Justifying Better UX, Forrester Research, 2016
2. Will Five Users Really Find 85% Of All Usability Problems?, MeasuringU