Automated (recorded) User Tests?

Hi, I know there were related discussions in the past, but I couldn’t find a usable outcome.

Is there any free/libre software platform for unmoderated user testing? A friend of mine works in a company that does small user tests with their custom proprietary software, and I’m looking for something related for Forgejo.

My idea boils down to the following:

  • a website where interested users can start a screenshare at any time
  • they can find a list of open test subjects (e.g. “Test these settings” or “Show us how you work with X”), or alternatively can just submit a general recording
  • optionally, there could be some notes per test subject such as instructions or hints
  • finally, the user starts the recording (including audio) while using the app, and stops it when done
  • the recording is saved to a disk and analyzed later

For the analysis, there is also some software for behavioural tracking IIRC (assigning activities to timeframes for later reuse, for example). And just watching the video and taking notes would likely work, too.

I spent my first 15+ hours doing user testing for Forgejo, mostly using BigBlueButton and OpenTalk. My two takeaways are that

  • it’s hard to observe the interesting workflows, because users don’t do tasks that require more time and thinking like publishing actual code
  • we don’t catch users in the moment they interact with something for the first time (e.g. the settings). I’d love to put a message there, proposing that they record their screen when interacting with part X for the first time
  • with volunteers, it’s really hard to scale this up, so some automation might help

Any options you know of? Some potential workarounds? Or motivation to hack something together?

2 Likes

I don’t know of any existing solution but it sounds like a worthwhile project.

At first I thought this is about direct alternatives to usertesting and the like. For these an open source alternative would not help much since their main service is the tester pool.

Having people record their screens seems pretty doable. I wonder if the most simple prototype would be a website consisting of:

  1. Instructions how to self-record a usability test (think-aloud etc.)
  2. Instructions for setting up a screen recording software
  3. A form to upload the video and to fill out a simple form. It appends the content to a ongoing text file.

The backend could be a parser for the text file, basically. No public sharing of recordings or comments, no login etc. It also has the advantage that people can easily NOT upload the video if they are embarassed or something got in the way etc.

1 Like

I’d love to put a message there, proposing that they record their screen when interacting with part X for the first time

That would be excellent! it would not infringe on privacy, would be a one-off and generate interesting data!

Hmm, if we give instructions on screen recording it might work as a first step. I wanted to avoid this, because it assumes at least a little technical knowledge, and would likely introduce some bias due to this barrier. Starting a screenshare is much more common knowledge nowadays.

I just stumbled across https://recordrtc.org/ and there is a demo of a simple screen recording. I suppose it would just need a way to submit this recording afterwards. However, I’m not fully sure if clientside recording would work out (depends on the length and filesize, probably). I would appreciate if I get lengthy material of full workflows where I can seek and skip … but these would likely not fit into the browser memory?

For me (~ full HD browser window), it’s about 10 MB / minute it seems. My dream of observing long-running actions is unlikely to become true, or it would require a lot of RAM. But maybe communicating this is okay.

The demo linked above allows me to save the video file using right-click context menu on the video itself.

This combined, I’m all in for creating a simple proof of concept that has instructions for self-recording the video file - having instructions for some local tools as well as such an online demo and using the “save video as” workaround. I might also take a look at saving the video directly, but I’m unfortunately not too familiar with frontend technology.

Small update: I “forgot” the need for simultaneous audio recording in the screenshare demo. I’d like to continue investigating this, but currently lack the time to dive deeper into this. Motivated coworkers still welcome to create a simple proof of concept (mainly looking for motivation, not specifically for technical knowledge).

Do you have any place where you would set this up, including a form to upload the videos? Could help designing that minimal solution, but I would not write the upload form as is (and maybe there are out-of-the-box upload solutions?)