Hi, I know there were related discussions in the past, but I couldn’t find a usable outcome.
Is there any free/libre software platform for unmoderated user testing? A friend of mine works in a company that does small user tests with their custom proprietary software, and I’m looking for something related for Forgejo.
My idea boils down to the following:
a website where interested users can start a screenshare at any time
they can find a list of open test subjects (e.g. “Test these settings” or “Show us how you work with X”), or alternatively can just submit a general recording
optionally, there could be some notes per test subject such as instructions or hints
finally, the user starts the recording (including audio) while using the app, and stops it when done
the recording is saved to a disk and analyzed later
For the analysis, there is also some software for behavioural tracking IIRC (assigning activities to timeframes for later reuse, for example). And just watching the video and taking notes would likely work, too.
I spent my first 15+ hours doing user testing for Forgejo, mostly using BigBlueButton and OpenTalk. My two takeaways are that
it’s hard to observe the interesting workflows, because users don’t do tasks that require more time and thinking like publishing actual code
we don’t catch users in the moment they interact with something for the first time (e.g. the settings). I’d love to put a message there, proposing that they record their screen when interacting with part X for the first time
with volunteers, it’s really hard to scale this up, so some automation might help
Any options you know of? Some potential workarounds? Or motivation to hack something together?
I don’t know of any existing solution but it sounds like a worthwhile project.
At first I thought this is about direct alternatives to usertesting and the like. For these an open source alternative would not help much since their main service is the tester pool.
Having people record their screens seems pretty doable. I wonder if the most simple prototype would be a website consisting of:
Instructions how to self-record a usability test (think-aloud etc.)
Instructions for setting up a screen recording software
A form to upload the video and to fill out a simple form. It appends the content to a ongoing text file.
The backend could be a parser for the text file, basically. No public sharing of recordings or comments, no login etc. It also has the advantage that people can easily NOT upload the video if they are embarassed or something got in the way etc.
Hmm, if we give instructions on screen recording it might work as a first step. I wanted to avoid this, because it assumes at least a little technical knowledge, and would likely introduce some bias due to this barrier. Starting a screenshare is much more common knowledge nowadays.
I just stumbled across https://recordrtc.org/ and there is a demo of a simple screen recording. I suppose it would just need a way to submit this recording afterwards. However, I’m not fully sure if clientside recording would work out (depends on the length and filesize, probably). I would appreciate if I get lengthy material of full workflows where I can seek and skip … but these would likely not fit into the browser memory?
For me (~ full HD browser window), it’s about 10 MB / minute it seems. My dream of observing long-running actions is unlikely to become true, or it would require a lot of RAM. But maybe communicating this is okay.
The demo linked above allows me to save the video file using right-click context menu on the video itself.
This combined, I’m all in for creating a simple proof of concept that has instructions for self-recording the video file - having instructions for some local tools as well as such an online demo and using the “save video as” workaround. I might also take a look at saving the video directly, but I’m unfortunately not too familiar with frontend technology.
Small update: I “forgot” the need for simultaneous audio recording in the screenshare demo. I’d like to continue investigating this, but currently lack the time to dive deeper into this. Motivated coworkers still welcome to create a simple proof of concept (mainly looking for motivation, not specifically for technical knowledge).
Do you have any place where you would set this up, including a form to upload the videos? Could help designing that minimal solution, but I would not write the upload form as is (and maybe there are out-of-the-box upload solutions?)
I considered using Nextcloud for submission of video files. But I struggle to find tools to create the ready-to-upload video files.
Even some client-side apps for screen recording don’t record the voice (or allow this but with some more steps).
I really wonder if there isn’t a pretty neat workaround somewhere. Most meeting software allow recording sessions, and they make it easy to enable audio and screen sharing. If it wasn’t so difficult to set this up, the easiest solution might be to host a video platform with recording features and give each user a random unattended meeting.
If you were to write a guide for local screen recording, do you have software in mind that is easy to use and records the voice, too?
Jibri recording for Jitsi might be an option. It would require a backend that automatically schedules a Jitsi call and redirects the participant and then starts the recording.
It would even allow user testers to enter the session if they have time, because they could watch for ongoing sessions, and keep them unmoderated otherwise.
The backend would need to prevent two users from entering the same session, and since jibri can only record one session at a time, it might even be necessary to limit the simultaneous sessions at all.
I have heard that Jitsi is easy to host (compared to other video conferencing platforms). I might try this out soon, but I am still asking around if someone has an idea for a better solution.
I assumed it is not hard to find but I was wrong. I fear “how to set up OBS” might be the best option and I do not like it as the most beginner friedly solution I can come up with.
Found this thread fascinating to read as my brain immediately went to how usertesting platforms ‘accidentally’ capture personal identifying information - e.g. names, timezones etc. and the large cost of holding data like screen recordings safely.
I know the org I work at, Superbloom.design have been thinking about a anonymity respecting user research open repository for a long time but we have considerable ethics, morals, data protection concerns before attempting that. Will keep watching this thread for developments
I see your concerns, and indeed I didn’t consider it in detail. During my user research, I was greeted with a (surprisingly) high amount of trust.
So far, my idea was
Select a project and topic (e.g. “Forgejo” and “Interacting with the settings”)
Agree to an initial disclaimer (“Your data (shared screen and microphone) will be recorded on our systems. You will be able to cancel and delete the recording at any time. Until you submit, it is only accessible to admins of the instance (Codeberg in this case).”)
You start the actual recording, and finish it eventually
You have another dialog to submit the actual recording, with some options
immediately delete the recording
submit the recording for review to the maintainers (e.g. “Forgejo maintainers”, can be different from only the administration of the service)
set a custom deletion timer (one week maximum, but allows for a shorter period; more time is more convenient for volunteers to review)
optionally submit an email address to receive the notes before they are published
After submission, you get a token to immediately delete the video later (if provided, the email address should also work)
if requested, it is possible to review the notes before they are published (I’m not sure if this is common practice in other projects, but we do this for Forgejo here)
What do you think? I would expect to get an informed consent for this procedure, but maybe I’m missing something important?
You have another dialog to submit the actual recording, with some options … you get a token to immediately delete the video later… if requested, it is possible to review the notes before they are published
Makes sense to me
You will be able to cancel and delete the recording at any time. Until you submit, it is only accessible to admins of the instance
The maintainers / user researchers of a project. In the early prototype, the lines are probably rather blurry, but in the future it would probably have some kind of more fine-grained access control.
Similar to how it is common practice to send confidential data to the “security team” of a project (also common via central platforms), you would trust the “user research team” to confidentially deal with your data.
If we build finer integration to match repositories or teams, it could even list the individual user accounts who would receive the recordings. But this is probably outside of the scope for an MVP.
