What really happens in….user testing

FullSizeRender

Ever wondered what really happens at a user testing session?
Wonder no more – I kept a diary at our session this week so I could share the trials and tribulations with you.

Although I kept a diary on the day – it actually doesn’t all happen in a day.  Lots of prep work comes beforehand.  Firstly a brief needs to be written to outline what is being tested and the sort of people we want to test on.  We then have to find the appropriate candidates (as close to the target audience as possible) and book them in, book a venue, and write test-scripts.

On this occasion we’re finding out whether the journey we’ve built is to easy to understand and navigate.  It’s a journey we’ve been working on for some time, so we have some inclination of how the tests will go, but we’re here to see for ourselves.

8.30am

It’s an early start as we have to get the test room ready – cameras, microphones, and the system we’re testing. We get the system we’re testing up on the screen ready, but it’s a bit glitchy. There’s a lot of tension as various different routes are tried to get it up and running without fault. With six candidates booked in for the day (all of whom are paid for their time) it’s not an option to cancel the testing!

9.00am

The good news is the site’s now working, the bad news is that our moderator is running late! We grab some drinks and get ourselves ready with mountains of Post-It notes and pens for the first session.

9.20am

The first candidate’s arrived and they’re briefed on the session – they’ll be asked to get a quote for home insurance as they usually would do at home. They know they’re being watched remotely, but the cameras are very discrete so they don’t feel intimidated.

9.30am

The moderator arrives so we’re ready to start watching. We’ve been told the objective for the test – all user testing starts with a hypothesis, and you’re watching to gather evidence. It’s important to record observations (not our own opinions) and suspend judgement.

We watch the first user and jot down any observations – such as the user misunderstanding copy or not knowing how an interaction works.

10.20am

Despite a few glitches in the system we reach the end of the first test – I’ve written over 20 Post-Its! We brief the moderator on which bits of the system had bugs so he knows which are user issues and which are systems. We want to avoid users getting hung-up on the bits of the journey that we know will be fixed before we go live.

11.20am

The second test is complete. After two hours of frantic note-taking we’re getting hungry but there’s no time to eat! It’s tempting to start analysing the issues we’ve seen, but that will come much later and after all the tests are complete. There’s debate in the room, and some good questions are raised, but we need to focus on observing. What I find interesting is how differently people approach buying home insurance, and how each user so far has had different sticking points.

12.15pm

The third user raced through the journey and stumbled in a couple of the same places that others had. You usually find in user testing that some issues will be repeated through the day and you’ll probably get fewer observations the day progresses.

12.45pm

At last it’s time for lunch! We head out into the local town to forage for falafels, then it’s back into the observation room!

1.30pm

It might be the post-lunch slump or it might be the candidate – but our next session is exhausting to watch! The user vocalises every single thought and doesn’t even make it all the way through the journey in his time slot! Our moderator displays immense patience and doesn’t once hurry him!

2.45pm

Luckily the next candidate is a little more relaxing to watch. She’s so far into the role of buying home insurance that we’re not convinced she realises this is just a test! We get some new observations from this user too so we’re re-energised!

3.45pm

Our final user comes in on time but the session is very slow. We’re testing on a mobile which she’s not used to using, and she’s very thorough. However we get a few last observations to add to our growing collection!

4.30pm

The sessions wrap up and we have a quick debrief with the moderator. All the Post-Its will be taken away and grouped, then any serious issues will be highlighted so that we can decide how to address them.

There’s a second day of testing to come so we don’t spend much time dwelling on the detail now.

What happens next

After all of the testing, once the issues are grouped, a summary of the observations will be created by the moderators.

Out of this we might see serious issues that affected multiple users and could impact the sale of the product, or we might see minor things that could be addressed through A/B testing at a later date.

The great thing about user testing is that it not only highlights issues, but it provides evidence as all sessions are recorded.

This is vital for sharing with stakeholders, and showing them which issues need to be prioritised.  Our next step, once we have the analysis, and have shared it with them, will be to create an action plan to iterate our web journey.