Hi all, I figured this would be the right place to ask this question: what are your best tips and advice on conducting usability tests with end-users?
One thing I was wondering about was how to do effective usability tests in a virtual format. When I was in university, I took courses where we would set up UX interviews for our prototypes and different team members played different roles (if there was more than one person, anyways). Obviously this will not work the exact same these days! So, I am hoping to learn from the ideas and thoughts of folks here.
To be clear, I am looking for advice on practices and ways of working, not tools (other threads were already posted here about that). I am hoping to understand how other designers are approaching things like this in the COVID-19 world for Open Source projects especially.
This is a very interesting question: many organisations (not just software projects) are struggling to adapt to doing research remotely. Luckily, lots of activities with users can be done this way, particularly if your users have access to a computing device and a decent internet connection (not all do!).
Pretty much all research activities I’ve done for FOSS projects (pre-Covid) were done remotely, mostly because that allowed me to reach users all over the world, and because it’s cheaper. Here come some notes based on my experience:
Usabilty testing
All the best practices for co-located usability studies apply to the remote kind.
Make sure to take some time to formulate your research questions, think about the profile of the participants, come up with appropriate tasks, go for the moderated variety to make the best of the time spent with your participants, and use the think aloud protocol.
To do usability studies remotely for desktop software, all you need is a video conferencing application with screensharing. I always recommend web-based ones (e.g. Jitsi) so that participants don’t need to install anything. Ask participants to share their screen with you so that you can observe what they do as they attempt the tasks.
With mobile is a bit more convoluted, but also possible with webcams or document cameras pointing to the participant’s mobile device, or screen mirroring software if your users know how to set it up. There are also commercial products to do this, but my experience with them has been poor.
Interviews
Super easy to do remotely: either voice calls or video calls will do.
There are other approaches and ways that are suitable for remote settings:
Photography, audio and video. Ask your users to show you around using their mobile phone cameras if their physical environment is relevant to your research questions. For instance, their working space may be of interest to you: if they have more than one monitor, if they use alternative input devices, etc. They can also record video or audio of themselves as well as taking photographs and send them to you. For example, you may ask them to record themselves while they do something with your software, and comment as they go along. You may ask them to record a walkthrough of their computing device, telling you what kind of software they have installed and why.
Making. You may ask your users to draw or write something for you, or in some more alternative and less used approaches, to make something physical. For instance, you may ask them to create a diagram depicting the steps and sequence of events when they do a certain thing or complete a specific task with or without software. Or you may ask them to create a network diagram showing the people they work or collaborate with using which software and communication channels. They can then take a photograph of their artifact and send it to you, and explain it in detail during a video call.
Diary studies: remote by design, this method involves your users documenting their experiences in a certain format (e.g. a paper diary, mobile messages, etc) over a period of time.
There are many other ways, I am sure I am curious to see what others come up with.
Hi @belenbarrospena, I forgot to say thank you for this great information! I have shared this thread more widely with my colleagues and other Open Source friends.
Curious if anyone else has ideas or experiences that have worked well for this kind of remote work.
Hey, I was going to reply but then I have a question which would help me share my two cents on this topic. Are you looking for advice on testing remote asychronously testing or testing remote synchronously?
@davidbenque There is tons of stuff out there, as I am sure you know It depends on what you need to read on. I will assume it’s remote research and interviewing.
There are a couple of decent Rosenfeld Media books on the subject, although you may find them a bit too basic:
If you want something meatier, I’ve been using this book on interviewing quite a bit:
Also happy to have a chat if there is anything you want to discuss
Don’t assume anything in terms of my knowledge in this area : ) it’s actually quite lacking. As part of my CryptPad role this is the first time I actually have time for a round of research as part of a funded project… have to start somewhere.
I am curious to see what others come up with.
Curious to know what you will share!