Are developers good judges of User Experience?

I recently redesigned a community site for developers. The site has the typical blog and discussion boards, meeting information and general background information about the community.  The client said the site was old and unattractive, she also noted that people had trouble searching for meeting locations and submitting requests. I also saw issues with poor navigation, the organization of content areas and the writing of its copious amount of information that were difficult to read.

One of the first things we did was to perform an expert review by two UX Architects and a Visual Designer, we quickly recognized several major issues but we wanted to check the site with the target audience to see if we could discover any new issues and perhaps gain some insights to help us when we begin redesigning the site. We created eight tasks to investigate. The tasks addressed how well people can find their way around the site, how well they can search for a geographic location and how error free they can complete a form.

This is the first time I conducted a usability study for a website specifically targeted to just developers. Here at Infragistics we have many developers so it was easy to find several to participate in this study. All the developers said participating in the study was an eye-opening experience for them, many had not been part of a usability study before so this was a great opportunity to experience firsthand what it feels like to be in one.  It was an interesting experience for me because of the unusual results I saw. We listened to and observed the developers as they performed each task. At the end of each session we asked each participant to complete the System Usability Scale (SUS).

The developers were quite outspoken, most said they disliked the visual design, the layout of the navigation and pages and the way the web components were used within the site. A minority of the developers, two of the seven, were positive in their remarks. We observed that the developers were able to complete every task, some with little difficulty but for the most part they were able to complete each task easily. Each task the developers attempted was rated by an observer for ease of completion on a 7 point scale, most tasks were scored as a 6 or 7 – which was great. If we looked at just the ability of the participant to complete the task, it would appear that there were no usability problems. When the developers were discussing the site using the “think aloud” technique they clearly expressed frustration with the design yet they very quickly discovered how to accomplish each task.

The results of the SUS was in agreement with the UX expert review, the average score was a very poor 34.5. The SUS measures ease of use, system satisfaction and learnability. The chart below should help to put the score into perspective, thanks to Bangor, Kortum and Miller “Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale” for this scale.

The developer’s score of 34.5 clearly reflected their frustration with the site, but not their ability to get the tasks done.  Although the developers were able to complete all the tasks rather easily, they rated the web site as very poor. In the last 10 years of using the SUS I have never seen a score this low! I typically see users having problems completing tasks with scores in the 50’s. With scores in the 40’s users are generally dead in the water! I think that the developer’s success with the tasks, no matter how poorly the application was designed, reflects the ability of these developers to quickly recognize software patterns and process and then overcome any design issues.

So, are developers good judges of User Experience? Anybody can give you their opinion of the software they just engaged with, but remember it is just their own opinion. This study clearly showed that developers are very knowledgeable about the patterns and controls used in software products (they easily accomplished all the tasks) but because they know so much they are also biased. This bias is known as the Curse of Knowledge , when you have knowledgeable people (software professionals) who find it difficult to think about problems from the perspective of less informed people (users). Not all the developers showed this bias, but about a quarter of them did. Developers are not the best judges of User Experience, but they are no worse than any other software professional who comes to the table with their extensive knowledge of software. In the end, it is the job of a UX professional to review the software, consolidate findings and present actionable solutions. Early on we came to the conclusion that the site was in desperate need of a redesign, each step in the research process helped us verify our initial thoughts and then identify the goals we’ll need to accomplish to insure a positive User Experience.