How To Test A Web Site Design In An Hour And On a Shoestring Budget

I have a friend who volunteers to be on an advisory board for a land trust conservancy organization. They have been designing a web site for the land trust. But they are all volunteers, and the organization doesn’t have a budget for web site design. They have a programmer donating her time to put together the website.

Can you get user feedback when the site doesn’t even exist yet? — My friend’s background is in usability, and she was concerned that the web site that the programmer was putting together had usability problems. But the group has virtually no budget to do user centered design or get user feedback on the prototype. And all she had were some pictures of a draft of some of the pages. For example, here’s what she had for the home page:

Picture of home page for Conservancy site

The menus didn’t “work” because it was just a picture, so she put together this page showing what would be in the drop downs if you did click on the main navigation on the home page:

Picture of home page with drop down menus

Continue reading “How To Test A Web Site Design In An Hour And On a Shoestring Budget”

How To Save Money And Time On User Testing: Run Multiple, Iterative Pilots

In my last blog post I reported on a study I recently conducted about differences between men and women in what they planned to purchase online for Valentine’s Day. (see Who is the Most Romantic). I used (affiliate) to collect the data, and I had an interesting insight about running user tests while I was doing the study.

Brief description of the service I used — In case you don’t know, it’s a service that lets you run what is called an un-moderated user test. Un-moderated means you are not there to moderate or facilitate the test session. You set up the test scenario and specify the web site and tasks you want the user to do by entering this information into a form at the site. Then the people recruit the users you have specified (meaning they post it to their database of already screened people), they provide the scenario and tasks to each user, and record the interactions each user has with the web site or sites. You get a notification that your test results are available, and then you can watch the video and the audio of each user session.

It’s very easy to set up and run user tests this way. If you are skilled at writing scenarios and tasks, it takes literally a minute to set up and run a test. It usually takes about 2-5 minutes for users to see the test post and start the test, and I have found that within 20-30 minutes videos are ready for you to watch.  Nice, right?

Running my first pilot for the study — When you fill out the form to set up the test you get to pick how many people you want to run the test. The first time I set up the test, I decided to just run one person. I wanted to make sure I had the wording right in my scenario and tasks. In other words, I was doing what is called a pilot test – I was running a test where I would throw away the data, just to see if my scenario and instructions were clear and would result in getting me the type of data I needed.

Why run a pilot anyway? — Running a pilot is always recommended when you are doing user testing, but I ‘ve seen lots of people skip this step. When you are doing “regular” moderated user testing (i.e.,  you are there in person, you’re renting a facility, you are paying money to recruit users, and you are paying money to the users as incentives), it’s expensive to run a pilot test. You should still do it, but I would say that less than 50% of the people I know even run a pilot test.

But with the system it’s easy and fast and not very expensive to run a pilot. The entire cost is $39 per user – for everything, so why not run a pilot?

How I came to run multiple pilots — In my test last week I ran a pilot, and found that certain wording in my task instructions was causing people to go off in a direction that was not useful in terms of the data I was interested in. I changed the wording of the instructions and ran the pilot again. Still not quite right, so I modified a little more. I  ran 4 pilots before I was convinced that the wording was clear and would result in the test testing what I actually wanted to test. Then I used that wording to run the real test.

How about running un-moderated pilots before a moderated study? — Now I was sure that the data I had coming would be useful and valid, and not just a reflection of some wording or instructional error I had in my tasks. By spending an hour or two to run the pilots I could be sure that the actual test results would be effective. It dawned on me that this ability to run multiple, iterative pilots was really powerful. I usually run one pilot, but I’d never been able to run iterative, multiple pilots. In fact, I’ve decided this is so powerful, that in the future, even when I am conducting  “regular” moderated user testing, I plan to first run a series of pilots with to test out my scenario and task instructions.

Contest to give away a free test session — I have a special idea to encourage comments for this blog: I have a special promotional code that the folks have given to me. You can use the code to run one free test for one user at I’m going to run a little contest here, and give the promotional code away to the person who writes the best comment to this blog (I get to decide which is the “best” comment). So, what do you think? Do you do user testing? Do you run pilot tests when you do? Have you used in this way?



Did you find this post interesting? If you did, please consider doing one or more of the following:

add your comment
subscribe to the blog via RSS or email
sign up for the Brain Lady newsletter
share this post

10 Ways To Get User Feedback

Recently I was talking to someone who is relatively new to the field of usability and user experience. He has developed a web application and wanted some ideas for getting feedback from users. He commented that he was planning on sending out a survey to users to see what they think about the web application. That was his plan for user testing. I’m so entrenched in the concept of usability and user testing that I have to stop sometimes and remember that not everyone else is.

“Well, you do have other choices besides doing a survey, you know”, I said.

“Oh, really?” he asked, “like what?”.

“I’ll send you some ideas,” I replied, and then I thought, “That would make a good blog post”, and, here we are.

1. “Traditional” moderated usability test – Let’s start with the most well-known and most used method of getting feedback from users. In a moderated usability test the user sits down in front of the software, web site web application, or other product that you are testing and uses the product, site or item to get one or more tasks done. The tester designs the test with real-life scenarios and asks the user to use the product or tool or site to go through and actually do the scenarios. The user is asked to talk out loud while they are completing the scenarios, so that the tester can understand what they are thinking and experiencing as they complete the activities they have been asked to do.  It’s called moderated because there is a facilitator to moderate the testing.

It’s important in a moderated usability test that:

  • Users must be representative of the actual user. It doesn’t work to use you or friend in the next cubicle, or your sister. The idea is to have a representative user try to use the site or product to get real tasks done.
  • Although you may be collecting other data, such as time to complete the task or number or types of errors made, the main data comes from the comments users make while they are working (called the “think aloud” technique).
  • Tests are done one-on-one. This isn’t a focus group.
  • Some facilitators “probe” with questions during the test, but this is tricky to do. You don’t want your questions to influence the user. Some facilitators wait until the tasks are completed before asking questions (called “de-briefing”).

Pros: Gives you lots of great data on what the usability issues are

Cons: Fairly expensive to conduct. You do these one at a time, so if you are testing 10 users that’s a lot of your time to be at the sessions, plan them,  analyze and report on data, etc. You may also need to pay for recruiting users and you need to give them an “incentive” (pay them in some way with cash, gift certificates etc). Continue reading “10 Ways To Get User Feedback”