Page style:

A Blue Perspective: A first foray into user testing

A first foray into user testing
17 December 2004

It may shock, surprise and disappoint you, but I did formal user testing on a website for the first time ever last week. Sure, I'd gotten people to play around with a site and asked them the non-specific question "what do you think of this?" But this was the first time I'd been involved in rounding up a group of prospective users, sitting them down and watching their every twitch, mutter and groan as they jumped through the hoops we'd set for them. And it was fascinating.

Gratifying to see the users understand your carefully structured menus. Mystifying to see users type search commands into an enquiry form. Horrifying to see users wandering circuitously in search of a picture. As with anyone who's done testing, though, I'd highly recommend it.

There's been enough articles written by more experienced people than I on the subject of "guerilla" user testing:

All those articles give you advice on taking usability out from behind the one way mirror, and making it easy (read: cheap) enough to be a viable development method on medium to large projects. However, there's a few things I'd reiterate from my own experience, and which I'll personally try to remedy in my own future dealings with user testing.

Firstly, it's impossible for a human observer to capture everything about what a user does – there's just too much going on to record every mouse movement, button press and search phrase; as well as timecode it all. If you want to capture such data, you'll definitely need to setup a video camera, otherwise, you'll just have to be satisfied with a record of what pages a user visited and what they clicked on. But if you want to find any major deficiencies in your web site, that's probably going to be enough.

Secondly, the wording of tasks can strongly influence the way that someone proceeds towards the goal. Depending upon what you want to achieve, you should try and eliminate any bias expressed in the words you use, and the way that you phrase them. For instance, if you supply a task loaded with keyword lists: "Find a news article about Cambodian locusts" then the tendency is to perform a search for those keywords. However, if you want the user to consider other navigation methods, or to formulate their own search phrase, then you should supply a less directed task: "Locate news about agriculture and how it is affected by crop pests."

There also seems to be two schools of thought to the observation process: get the user to verbalise what they are doing, and why, or let them traverse the site naturally. In this instance I opted for silence, reasoning that if a user has to explain their actions, those actions will be artifically influenced through some sort of Heisenberg usability principle. It was left up to an exit survey to glean the hidden meaning in a user's confusion. It seemed to produce some good results, but I can't be certain which method is better without seeing some user testing of user testing methods.

Oh, and you should clear the history cache in the browser, so that successive users don't get visited links leading them through your well organised tasks :o]


1/3. 19 December 2004 @ 07:52, Jenni wrote:

I hated the first time I had to watch someone test a website I built. You think you made everything clear and easy to use, and they still fumble around lost and confused! :( Good tips, though, thanks!

2/3. 20 December 2004 @ 10:03, Michael Koukoullis wrote:

I asked all the particpants to verbalise their thoughts. It is impossible to consistently follow what they are doing on the site inorder to complete a task whilst you are trying to take notes at the same time. Their actions are a traversal of a decicion tree which you must understand and document.

For example when asked to locate a particular"event relevant to a particular kind of researcher", it was reassuring to hear them naturally browsing the categories on the main menu. If they did that first then used the search engine it is indicative that maybe the menu labelling was not clear enough. Given the absence of video recording during the session and my inability to write in shorthand , having the guinea pigs verbalise their actions helped me document what the hell was going on.

3/3. 10 January 2005 @ 23:20, Andy Budd wrote:

"Oh, and you should clear the history cache in the browser, so that successive users don't get visited links leading them through your well organised tasks :o]"

lol. I figured that one out as well the first time I did a usability test. However the problem wasn't visited links, it was text input boxes auto-filling with the last subjects entered data. Doh!

I personally think asking subjects to talk through their thinking helps a lot. Otherwise you can see that somebody is doing something wrong, but have to guess why.

Post your own comments

Fields marked with an asterisk* are mandatory. All HTML tags will be escaped. http:// strings in comments will be auto-linked.