gLog

Since its release last week, there have been numerous reviews of Silverback (the guerilla usability testing application). Most I’ve read are good overviews of the application but don’t discuss how to go about actually running a guerilla usability test.  What follows below is the process I used recently to test a new product feature. It’s a mixture of a fairly typical usability test protocol and guerilla techniques (I feel like such a rebel already). I utilized Silverback to capture the video, audio, and interactions of each participant.

Have something to test.
It seems a bit silly to mention this but you need something interactive to test. Since the new feature wasn’t implemented yet, I did a screen shot of the web page where the new feature would be implemented. After applying a little digital camouflage, aka HTML, CSS, and Photoshop tweaking, I had a prototype that was ready to test.  I thought it was important to show this new feature in context so that’s why I did this. Not everything on the screen worked but at least it looked like the final product.

Know what you want to test.
There were three specific tasks participants needed to be able to accomplish with this new feature.  I documented these tasks and wrote non-leading questions to see if the participants could accomplish them.  I also wrote a few sentences that described the context of use and how they got to the web page where the new feature was prototyped.

Setup Silverback.
I loaded up the prototype in a web browser, started Silverback, created a new project, created a new session, centered my mug for the camera, and hit the space bar (this will make sense if you’ve watched the demo). I had already decided to run the tests at my desk. I know usability tests are typically run in a quiet meeting room, or in the participant’s workspace, but the conference room was booked and after all this was a guerilla usability test. No mercy.

Find participants.
So then I had to find my victims participants. I knew I wanted to test with at least six people. I also knew the air-conditioning on the third floor was on the fritz so I was fairly positive I’d be able to find people who would enjoy a short reprieve from the heat (I tried to keep each session to a maximum of ten minutes). Also, I’d like to acknowledge that while I obviously didn’t worry about recruiting the exact demographic/audience for the new feature, I did try to select participants of varying ages, genders, and roles within the company.  Let’s call this technique guerilla sampling.

Running the test.
My first participant was one of our DBAs who hadn’t worked on the project. He was accessible (I just had to turn my chair around). I asked him to sit down at my desk and make himself comfortable. I thanked him for helping, explained what we would be doing, told him we’d be recording the session, read a statement of informed consent that outlined his rights as a participant of the test, assured him we’d be testing the interface and not him, and encouraged him to talk out loud about what he was thinking during the test to help me better understand the motivations for his interactions with the prototype.

I then explained the context for the prototype to help him understand how he got to the screen. I asked if his was ready and hit the spacebar to start recording.  I asked him a question to see if he could complete the first task. We went through all the task-related questions and then I asked some more general questions about the interface.  Finally, I asked if he had any other thoughts and concluded the test.

Rinse. Repeat.  I then went around the office recruiting more participants.  When it was all said and done I had tested six people.  Everyone seemed open to doing this.  I’m not sure if it was the short time commitment, lack of air-conditioning, or my charming personality but no one really objected to participating.  The one thing I did notice was that when I mentioned we would be recording the session all of them seemed taken aback.  Once I explained it was for research purposes only, and that it wouldn’t be shared outside of the building, they were fine with being recorded. I think this is important to note.

The results?
After all the sessions were complete, it was time to see whether any usability issues emerged. I exported movies of each participant and watched them one at a time while taking copious notes. While I had noticed some fairly obvious issues during the sessions, it wasn’t until I was able to watch the recorded sessions that I noticed some of the more subtle ones.  When I was finished watching all the videos, I transferred the issues into a spreadsheet (the AK-47 of usability testing).  This helped me quantify the similarities and differences between the participants.  I then prioritized and designed solutions where appropriate.  Finally, I put all of my findings into a usability report which was delivered to project stakeholders. And there was much rejoicing.

In total, I spent about 4 hours: preparing for the test; finding and running tests with participants; evaluating the results; and writing the report.  I’m sure I’ll refine this process over the coming weeks but I can definitely see how useful Silverback is in helping ensure you don’t miss anything. I can’t wait and see what additional features they add in subsequent versions. Viva la revolution!

On a side note, the usability testing protocol I used is based mostly on concepts from the excellent book Observing the User Experience: A Practitioner’s Guide to User Research.  A highly recommended resource.

Posted in: User Experience Design