On this episode of the Human Tech podcast we talk with our guest Danielle Cooley about the challenges of user testing. What works, what doesn’t, and why.
At The Team W we’ve updated our User Testing online video course. This latest version of the course has been filmed in our new studio. We’ve expanded and updated the content. The video clip below will give you an idea of what’s in the course.
You can get details on the course, preview some lessons, and/or register for the course at the User Testing online video course page. The User Testing course is also included in our User Experience Certificate curriculum.
To see the catalog of all of our online video courses, go to the main course catalog page.
User Testing as a way to get feedback from people about a product is still going strong. In this episode Susan quizzes Guthrie about the what and how of user testing, and talks about some of her fun and more memorable moments of the hundreds of tests she’s conducted.
HumanTech is a podcast at the intersection of humans, brain science, and technology. Your hosts Guthrie and Dr. Susan Weinschenk explore how behavioral and brain science affects our technologies and how technologies affect our brains.
You can subscribe to the HumanTech podcast through iTunes, Stitcher, or where ever you listen to podcasts.
User testing is a great way to get feedback from actual users/customers about your product. Whether you are new to user testing, or a seasoned testing professional, you want to get the most out of your user testing research. It’s easy to fall into some bad habits though, that will either make your testing time consuming, ineffective, or expensive. So here are 10 bad habits to watch out for when you are doing user testing:
#10 Skip the pilot
A pilot test is a user test you run before you run your “real” test. You run a pilot test so that you can try out your prototype and your instructions – it’s a trial run for everything. Then you can make any changes necessary before you run your “real” test. Sometimes your pilots go without a hitch and then it can be easy to say the next time, “Oh, maybe I’ll skip the pilot”. Don’t skip the pilot! Otherwise you may have to redo the whole test. Pilots are fast and inexpensive and worth it to do.
#9 Draw conclusions from early and insufficient data
People get excited when results start coming in, but don’t start changing things after 1 or 2 participants. You’ve got to wait to see what everyone does or doesn’t do before you start making decisions. And watch out for the confirmation bias – deciding that you know what’s going on after 2 participants and then ignoring other data that comes in later.
#8 Test too many people
If you are used to quantitative measures you might be used to running studies with large numbers of people. But user testing is often qualitative rather than quantitative (there are exceptions). If you aren’t running statistical analyses, so you don’t need lots of people. 7 to 10 people (per cell, see #7 below) will get you the data you need most of the time.
#7 Too many cells
A cell is the smallest unit of categorization for a user test. Let’s say that you want to run your test on men and women, and you want to be able to draw conclusions about differences in men and women. That means you have 2 cells – one for men and one for women and you need to run 7 people per cell. Now you already have 14 people. Next you decide to add young people versus older people, so now you have 4 cells of 7 each. Then you add people who are current customers vs. not current customers…. You can see that this is headed to too many people. The mistake here is a misunderstanding between cells and variation. I can have just one cell of 10 people, and within that cell I can make sure that I have some men, some women, some older people, some younger people – they only have to be a separate cell if I am going to draw conclusions about the variables. If I just want variation, but don’t need conclusions about the variability, then I don’t need all these cells.
#6 Do a data dump
When you conduct a user testing study you are familiar with everything, the instructions, the tasks, the results, and you may not realize that if you just hand the data and the video recordings to someone else they may be overwhelmed. You need to summarize the results, draw conclusions, and present a cohesive summary to your team and stakeholders, not just hand them a lot of data.
#5 Too much stock in after-test surveys
People have bad memories, and they also tend to over-inflate ratings on a survey. Watch out for putting too much stock in a survey that you give them after the task portion of the test.
#4 Test too late
Don’t wait till the product is designed to test. You can test wireframes, prototypes, and even sketches.
#3 Skip over surprises
Some of the best data from user testing comes not from the tasks themselves, but the places that people wander off, or the off-hand comments they make about something you weren’t even officially testing. Don’t ignore these surprises. They can be extremely valuable.
#2 Draw conclusions not supported by the data
You have to interpret the data, but watch out for drawing conclusions (usually with your pet theories) that really aren’t supported from the data.
#1 Skip the highlights video
A picture is worth 1000 words and a video is worth even more. These days highlight videos (made up of video clips) are easy to assemble using tools (for example usertesting.com). Highlight videos are much more persuasive than you just saying what happened. Make a habit of creating video clips the first time you watch the videos. Then you don’t have to go through them again to create a highlights video.
What do you think? Do you have any bad habits to add to the list?
If you are interested in learning more about user testing consider these two courses:
User Testing: The What Why and How as an in-person workshop I’m teaching it in San Francisco on July 31, 2014.
an online video course on User Testing.
With Lean UX all the rage (deservedly in my opinion — see my recent slideshare on Lean UX), user testing (an important part of the Lean UX process) is getting even more popular. If you need to convince someone(s) in your organization that user testing is important — well, not just important but CRITICAL — try this video below. It’s an introduction video to my User Testing course. And if you are interested in the course I’ll be teaching it in San Francisco on July 31, 2014. Bring the whole team! If you are already convinced about how important usability testing is, then stay tuned for an upcoming blog post on Bad Usability Testing Habits To Avoid.