Software as a service for user testing

Design Lounge Usability reviews and research
1

This morning in the #positivepython channel on freenode I heard the following desire for infrastructure for doing using testing on open source projects:

I wish we had a better set of infrastructure for doing f/oss usertesting, seems like most problems in like this (e.g. Debian’s website, Helios, etc), they live in the developer’s blind spot, since they’re not the user and they’re focused on a feature-set that’s interesting to them
…
like travis, but for user-testing

I just tried a couple searches for “user testing saas” and “user testing software as a service” and didn’t find anything I’d heard of. I’m much more familiar with real world, in-person user testing efforts such as SpinachCon or the User Research Center at Harvard Library.

Is there software as a service for user testing that open source projects can use?

For more context on the quote above see BotBot.me + Startup Resources | Startup Resources part of which is captured below:

nedbat: i just did my PSF ballot, and I remain impressed by the over-engineering of the encrypted ballot, and the complete lack of awareness of usability issues.

sigmavirus24: nedbat: Would you care to email vote@python.org with your feedback? Mark and I will work with te Helios team to fix as much as we can

pdurbin: What are the usability issues?

sigmavirus24: pdurbin: well, I can think of a few myself

sigmavirus24: That said, on the whole, I find Helios to be leagues better than eVote

*: pdurbin doesn’t believe he’s used either

nedbat: sigmavirus24: i might send an email, but i feel like i’ve said this in previous years: the election software seems entirely focused on assuring me about the encryption of the results (which I frankly don’t care about), and not concerned much with helping me understand what I am voting for.

paultag: I wish we had a better set of infrastructure for doing f/oss usertesting, seems like most problems in like this (e.g. Debian’s website, Helios, etc), they live in the developer’s blind spot, since they’re not the user and they’re focused on a feature-set that’s interesting to them

paultag: something’s perfectly usable if you’re one of the world’s leading experts in that feature-set

paultag: (I say this, as guilty of the same exact thing)

pdurbin: paultag: have you seen https://discourse.opensourcedesign.net ? That group talks about user testing and such.

paultag: I mean, infrastructure that will take a f/oss project and user-test it, or set it up for user-testing (e.g. recorded user sessions) in a lightweight and easy way

paultag: like travis, but for user-testing

paultag: I’m sure people talk all day about it

paultag: but it’s usless unless you put it into practice, and use the issues and feedback created from it to generate usablity issues

1 Like
2

Do you mean something like usertesting.com, but using an opensource infrastructure?

I assume one reason that you might not have been able to find the SaaS is that it is usually a service. The software is only a part of it and does usually not make much sense itself – the most interesting feature is probably that the services also distribute users with suitable attributes to the tests.

So, such a software would need to have

  • Screen recording and audio recording
  • Annotations with instructions
  • simple feedback form or so

But also:

  • A data base of users
  • Some mechanism to draw a sample of users with suitable attributes (Age, experience, whatever)
  • Connection to the actual testing part above.
3

@jdittrich good question. A quick answer:

pdurbin: paultag: Do you mean something like usertesting.com, but using an opensource infrastructure?

paultag: pdurbin: sorta, there are tons of usertesting platforms, but, I was trying to explain something as easy as travis to put into a repo to enable running usertesting, including payment for people’s time and find a way to test new features before they get merged, as well as periodic retests of workflows. So, if you want to look at something like usertesting.com, something for before and after to help

paultag: mediate interactions

paultag: so have people express they’d be willing to test stuff, and a developer (who hopefully knows how to moderate) can jump on, session is recorded, and look across sessions for patterns

paultag: but like, hosting the test version, etc

paultag: anyway, whatever, just a thought

@jdittrich please do feel free to hop in #positivepython to chat about this more. I guess I’m just the messenger.