The Orbiter

• You don’t need to test more than 5 people at a time.
• Test early and often.
• Focus on specific flows.

Usability experts have an annoying shared trait: they tend to answer a lot of questions by saying it depends. Each website’s unique content, audience, goals, and style ensure that there are no universal solutions to even the most basic questions. This means that there is only one way to know if your site really works for users: you have to test it. Fortunately, if conducted right, these tests shouldn’t require a lot of money or time.

When professionals begin designing a website, they typically follow a series of guidelines or design patterns that help ensure the result is easy to use. In the case of simple interactions, such as a log-in form, this method almost always results in a usable design.

However, when the interface becomes complicated or highly specific to an individual site, it’s unlikely that any existing guidelines or patterns will provide enough direction. At this point, the designer cannot—or, at least, should not—be sure that the interface will work: what they have produced is simply an educated guess at a usable design. Even worse, many inexperienced (and some experienced) designers will base their work on assumptions or personal preferences—either their own or their clients’. Now you can drop the educated and just call the design a guess.

Beyond the Guess

Most major websites rely on user testing to determine if their features work for visitors. Although the specifics vary with each implementation, user-testing basics are generally the same. A representative sample of subjects are exposed to a website (or a portion of a website), data is collected based on the subjects’ interactions, and the results are analyzed to measure effectiveness.

Big sites like Amazon and Netflix collect data from huge numbers of users in a constant stream of specialized tests. If Amazon wants to test a feature with 5,000 new users, they only need to turn the feature on for 45 seconds and 5,000 new users will see it. (Being an 800-pound gorilla has its benefits.) Of course, big websites tend to have big budgets, and this type of ongoing testing isn’t cheap.

But while these tests generally produce a lot of great data, more data doesn’t always mean better results. Just recently Google tested its new social media tool Buzz with 20,000 people. Unfortunately, these 20,000 people were all Google employees—who have a specific set of shared experiences and web-usage patterns, to put it mildly—and this testing therefore failed to turn up a major usability flaw: Buzz automatically connects you to people you have emailed. The resulting—and immediate—storm of complaints had Google publicly apologizing and rushing to make changes.

By any measure, Google’s user testing for Buzz was a massive failure. Think you could do better with five subjects and a couple days? You should.

Cheap, Fast and Good

The idea that user testing can be done on the cheap isn’t new. Jakob Nielsen’s 1989 paper on discount user testing is often seen as the start of the low-cost usability movement, which also gets called guerilla usability.

The basic premise is simple: although larger studies are useful, discount testing produces valuable results and a far greater return on investment. (Predictably, Nielsen goes further and suggests that discount testing often produces better results than bigger studies because it encourages an iterative design process.) In fact, one of Nielsen’s more interesting findings was that while better studies produce better results, even poorly executed quick tests provide significant value.

At Orbit, we’ve distilled a variety of research and approaches into three main principles for guerilla usability.

1. You don’t need to test more than 5 people at a time. If your site (or feature) has a major flaw, the first subject will find it. So will the second. So will the 35th. You don’t need to watch people make the same mistakes over and over, and while five people won’t find every tiny problem, they will provide you with a lot of information about your site’s strengths and weaknesses. (Even testing a single person can be useful, especially since the obvious stuff is often the easiest to miss.) Similarly, while it’s best to recruit subjects who are representative of your actual audience, testing with friends, family, or anyone within reach will also produce worthwhile results.
2. Test early and often. If you wait until your site is complete to test it, any changes that need to be made will be expensive, time consuming, and complicated. But if you first test the flow of your main features by presenting a subject with hand-drawn wireframes (known as paper prototyping), changes require only a good eraser—or at worst, a new sheet of paper. There is no substitute for testing a live site, but running the major functions through a quick battery of paper prototyping will catch the biggest flaws before you start the expensive development process.
3. Focus on specific flows. Testing every feature on a large-scale site, even with a single subject, can take hours or days. (And that’s time for which these people will need to be compensated, once you find volunteers willing to endure such punishment.) After you finish a few of these extensive tests, you’ll be left with a mountain of data to sift through, and a potentially imposing array of changes to be made. But every feature on your site is not equal in importance. Start with the crucial functions—on an e-commerce site, this would probably be the checkout process—and save the less important features for later. If you do need to test a number of features, break up the work into several tests and conduct each just before the design or development of that specific function.

If conducted intelligently, discount usability can provide crucial data to improve your site’s design and greatly improve its effectiveness. Paper prototyping can produce results in a few hours of work, and tests on a live site can be completed and analyzed within a couple days.

These quick guerilla techniques won’t guarantee that your site is perfect, but they will ensure you catch the biggest problems. And that’s perhaps the most important principle of all: doing something is much better than doing nothing. Every user test you perform—however modest—takes you further beyond the guess and closer to the knowledge that your site works.