How To Save Money And Time On User Testing: Run Multiple, Iterative Pilots

In my last blog post I reported on a study I recently conducted about differences between men and women in what they planned to purchase online for Valentine’s Day. (see Who is the Most Romantic). I used (affiliate) to collect the data, and I had an interesting insight about running user tests while I was doing the study.

Brief description of the service I used — In case you don’t know, it’s a service that lets you run what is called an un-moderated user test. Un-moderated means you are not there to moderate or facilitate the test session. You set up the test scenario and specify the web site and tasks you want the user to do by entering this information into a form at the site. Then the people recruit the users you have specified (meaning they post it to their database of already screened people), they provide the scenario and tasks to each user, and record the interactions each user has with the web site or sites. You get a notification that your test results are available, and then you can watch the video and the audio of each user session.

It’s very easy to set up and run user tests this way. If you are skilled at writing scenarios and tasks, it takes literally a minute to set up and run a test. It usually takes about 2-5 minutes for users to see the test post and start the test, and I have found that within 20-30 minutes videos are ready for you to watch.  Nice, right?

Running my first pilot for the study — When you fill out the form to set up the test you get to pick how many people you want to run the test. The first time I set up the test, I decided to just run one person. I wanted to make sure I had the wording right in my scenario and tasks. In other words, I was doing what is called a pilot test – I was running a test where I would throw away the data, just to see if my scenario and instructions were clear and would result in getting me the type of data I needed.

Why run a pilot anyway? — Running a pilot is always recommended when you are doing user testing, but I ‘ve seen lots of people skip this step. When you are doing “regular” moderated user testing (i.e.,  you are there in person, you’re renting a facility, you are paying money to recruit users, and you are paying money to the users as incentives), it’s expensive to run a pilot test. You should still do it, but I would say that less than 50% of the people I know even run a pilot test.

But with the system it’s easy and fast and not very expensive to run a pilot. The entire cost is $39 per user – for everything, so why not run a pilot?

How I came to run multiple pilots — In my test last week I ran a pilot, and found that certain wording in my task instructions was causing people to go off in a direction that was not useful in terms of the data I was interested in. I changed the wording of the instructions and ran the pilot again. Still not quite right, so I modified a little more. I  ran 4 pilots before I was convinced that the wording was clear and would result in the test testing what I actually wanted to test. Then I used that wording to run the real test.

How about running un-moderated pilots before a moderated study? — Now I was sure that the data I had coming would be useful and valid, and not just a reflection of some wording or instructional error I had in my tasks. By spending an hour or two to run the pilots I could be sure that the actual test results would be effective. It dawned on me that this ability to run multiple, iterative pilots was really powerful. I usually run one pilot, but I’d never been able to run iterative, multiple pilots. In fact, I’ve decided this is so powerful, that in the future, even when I am conducting  “regular” moderated user testing, I plan to first run a series of pilots with to test out my scenario and task instructions.

Contest to give away a free test session — I have a special idea to encourage comments for this blog: I have a special promotional code that the folks have given to me. You can use the code to run one free test for one user at I’m going to run a little contest here, and give the promotional code away to the person who writes the best comment to this blog (I get to decide which is the “best” comment). So, what do you think? Do you do user testing? Do you run pilot tests when you do? Have you used in this way?



Did you find this post interesting? If you did, please consider doing one or more of the following:

add your comment
subscribe to the blog via RSS or email
sign up for the Brain Lady newsletter
share this post

Book Review of Steve Krug's Rocket Surgery Made Easy
Who is The Most Romantic?: The Brain Science of Valentine's Day

2 Replies to “How To Save Money And Time On User Testing: Run Multiple, Iterative Pilots”

  1. I think this makes perfect sense. Much better than spending a lot of $$$ upfront and then finding out that your instructions were not clear enough.

    Although $39 maybe too much money to spend on this kind of thing. Perhaps the same instructions on Mechanical Turk task would suffice, to get feedback on the wording. You’d certainly spend a lot less money. Once you have your feedback on the wording of the tasks, you could then do the real thing on UserTesting.

  2. I’ve always been a fan of iterative testing. You can quickly dispose of the task-flow roadblocks of early design. It is a fast way to hone down the rough edges of designs and task flows so you can finally test for the small bumps in the road. Your propose multi-pilot proposal has potentially equal value for unmoderated tests and surveys, as well as other methodologies.

    I have run a few surveys and unmoderated tests. I’ve also run hundreds of usability tests. I expect to throw out the first (pilot) participant due to misinterpreting the user’s mental model of a task or poor directions or writing on my part.

    I think the ability to test your protocol for $39 is money well spent considering the cost of one in-lab usability pilot subject can cost $200 or more depending on the participant. It could even be economical for surveys that you intend to later push out by different (cheaper) means to large populations. I’ve had to throw out several areas of a survey when I discovered a question could be easily misinterpreted.

Leave a Reply

Your email address will not be published. Required fields are marked *