Recently I was talking to someone who is relatively new to the field of usability and user experience. He has developed a web application and wanted some ideas for getting feedback from users. He commented that he was planning on sending out a survey to users to see what they think about the web application. That was his plan for user testing. I’m so entrenched in the concept of usability and user testing that I have to stop sometimes and remember that not everyone else is.
“Well, you do have other choices besides doing a survey, you know”, I said.
“Oh, really?” he asked, “like what?”.
“I’ll send you some ideas,” I replied, and then I thought, “That would make a good blog post”, and, here we are.
1. “Traditional” moderated usability test – Let’s start with the most well-known and most used method of getting feedback from users. In a moderated usability test the user sits down in front of the software, web site web application, or other product that you are testing and uses the product, site or item to get one or more tasks done. The tester designs the test with real-life scenarios and asks the user to use the product or tool or site to go through and actually do the scenarios. The user is asked to talk out loud while they are completing the scenarios, so that the tester can understand what they are thinking and experiencing as they complete the activities they have been asked to do. It’s called moderated because there is a facilitator to moderate the testing.
It’s important in a moderated usability test that:
- Users must be representative of the actual user. It doesn’t work to use you or friend in the next cubicle, or your sister. The idea is to have a representative user try to use the site or product to get real tasks done.
- Although you may be collecting other data, such as time to complete the task or number or types of errors made, the main data comes from the comments users make while they are working (called the “think aloud” technique).
- Tests are done one-on-one. This isn’t a focus group.
- Some facilitators “probe” with questions during the test, but this is tricky to do. You don’t want your questions to influence the user. Some facilitators wait until the tasks are completed before asking questions (called “de-briefing”).
Pros: Gives you lots of great data on what the usability issues are
Cons: Fairly expensive to conduct. You do these one at a time, so if you are testing 10 users that’s a lot of your time to be at the sessions, plan them, analyze and report on data, etc. You may also need to pay for recruiting users and you need to give them an “incentive” (pay them in some way with cash, gift certificates etc).
2. Un-moderated usability test – As the name suggests, in an un-moderated usability test there is no person facilitating (“moderating”). Special software is used that shows the tasks and steps that the user is supposed to go through. These instructions show on the users computer screen while they are using the web site or software, and the special software is tracking what they do, where they go, and what they click on. The newest type of un-moderated test includes a screen capture video and also an audio recording while the user “thinks out loud”.
Pros: Less expensive, since the data is collected “automatically”.
Cons: You don’t have the ability to probe and ask questions. Users have to download special software and sometimes there are technical glitches. In addition you miss the non-verbal communication, such as frowning, looking confused, or tears (yes, I’ve seen people so frustrated with poorly designed, unusable software that they start to cry).
3. Remote usability test – A remote usability test is like a regular moderated test, except rather than being in the same room with the participant, the moderator is interacting with an audio/video conferencing application. You can do this using tele-conferencing applications, or with special applications such as Techsmith’s UserVue (http://www.techsmith.com/uservue.asp)
Pros: Save on travel costs.
Cons: Users have to connect and/or download and use the tele-conferencing/remote testing application, and sometimes there are technical glitches with that. Unless the user has a camera, you miss seeing their facial expressions.
4. Eye tracking usability test – An eye tracking test is a lot like a regular moderated usability test, except special equipment is used that can track and report on where people actually looked on a screen, for how long and in what path. Check out the blog I wrote about eye tracking.
Pros: Can collect some user experience data while also getting the eye gaze data.
Cons: Have to change the methodology somewhat (see the blog article on this), and you can’t have people do the think aloud technique.
5. Card Sort – A card sort isn’t a usability test. It’s a method to get feedback from users about how information should be organized, for example, at a web site. You take all the possible items of information and put one item on a card. For example, if you were designing a web site for travel services you would have a card that says “Cars” and another that says “Flight status” and another “Hotels” etc. The user is given all these cards and is asked to organize them into categories with what they believe “goes together”.
Pros: It’s a very effective way to get feedback on how information should be organized. It only requires cards, so it’s not difficult to put together this kind of study.
Cons: You only get information on how information should be organized.
6. Reverse Card Sort – As you might guess from the name it’s the opposite of a card sort. Instead of giving the user cards with words and phrases on them, you show them the navigation structure of a web site and then ask questions such as “Show me where you would go to make a hotel reservation?”
Pros: It’s fast and quick to get feedback on the design you have or the one you have in mind.
Cons: You only get feedback on what you have already put together. You don’t get a “clean slate” as you do with a card sort.
7. User Interviews – User interviews are a very important part of making a software application, web site, or technology device usable. But they are most important early on, before your product or web site has been designed. You can use user interviews after the fact to get feedback on an existing design, but usability testing is a better method. If you just ask users what they think about a product you cannot rely on what they are telling you. (See my blog post on reconstructive memory). It is better to ask users to do a task and watch them do it then just to have a conversation.
Pros: Important method early in the design process to get information on requirements, needs, problems with current methods.
Cons: Doesn’t give you information on how usable the product actually is.
8. Contextual Interviews and Observations – If you go out to where users are using your software or web site, or cell phone, and watch them use the product in their natural environment then you are conducting a contextual observation study. If you are looking for feedback for a web site or software application a usability test will likely gain you better data, since you can control which tasks the person does and be sure of getting feedback on the issues you are most interested in. But for some products such as cell phones or medical technology, you may need to see the user using the product in the environment that they normally do. Bringing someone into a “lab” for a usability test won’t get you the richness of data that seeing use in context will. You can also combine observation with an interview in their environment.
Pros: Get to see the user use the product in a “natural” environment
Cons: Can be costly to do if you have to travel to many locations. The issues you are interested in may not come up during the “naturalistic” observation.
9. Surveys – Surveys are a great tool for getting ideas about what is important to people (why don’t you check out and take my survey, if you haven’t already, that asks about what people want to see me write about in my blog posts?: (http://bit.ly/2mIEzn), but they are a poor tool for getting usability feedback. People don’t always say how they really feel in a survey. Some people tend to always rate things high and others tend to always rate things low. How you word the questions can affect the response. You can’t collect data on how easy or hard a task was to accomplish.
Pros: Using today’s survey tools for example survey monkey, you can send out a survey inexpensively to a lot of people.
Cons: You can’t collect true usability data.
10. user testing .com – Usertesting.com is an un-moderated usability testing service, but I’ve put them in a category of their own because they have some unique features. Users download special software and see the instructions on their screen. The entire session is recorded, both the screen capture video as well as audio. Users fill out a one-page form as well with their impression. Users come “with” the service. You don’t have to recruit users; the usertesting.com folks have a database of people they go to. Because of this you don’t have to worry about technical glitches. Each user has already downloaded the special software and knows how to use it (they have to demonstrate that in order to be in the usertesting.com user data base).
Pros: A really inexpensive method to do usability testing. It costs $39 (as of March 2010) to recruit and pay the user (the comparable rate for a regular moderated usability test is $200), the test is remote, so there is no travel, and it is very fast (you usually get results in a few hours (sometimes a few minutes!).
Cons: There is no camera view of the person, so you don’t get non-verbal information or facial reactions. The sessions only run about 15-20 minutes so you can’t test everything you might want to test. And you can’t do a “de-brief” and ask questions since this is an un-moderated test. Still, this is a great service, and if you haven’t tried it I suggest you check it out. Here is an affiliate link: (www.usertesting.com).
Do I need to be an expert? — These methods of getting user feedback vary in how much expertise you have to have with the method and/or with the technology in order to do an effective session. Eye tracking requires the most expertise, and Usertesting.com un-moderated testing probably requires the least. With one exception, I do suggest that you engage an expert if you are new to this, at least for your first time out. The exception is usertesting.com. If you are brand new to the methods of user testing then I suggest you go run one test with one participant at usertesting.com. You will “get your feet wet” and get a good idea of the rich and interesting feedback you can and will get from users when you undertake doing user feedback research.
What have you tried? Have I left out your favorite method? Comment and let me know.
Did you find this post interesting? If you did, please consider doing one or more of the following:
add your comment
subscribe to the blog via RSS or email
sign up for the Brain Lady newsletter
share this post