User research for an open data publishing service

Hi everyone,

We are a collective in the process of developing a web site to publish open data (please see our introduction in the lounge for more information). One of the participants of the collective is a UX designer who inspired us to do the right thing and begin with interviewing the intended audience, i.e. developers.

Distributing a data set is not very challenging: it consists of

  • the files containing the data,
  • the documentation explaining what they contain,
  • the license,
  • release notes explaining how to go about the data structure change, unless some efforts is made to ensure backward compatibility

I was surprised to discover that none of the existing open data provider implement all these basic features. It would have been so easy to mimic the UX of a well behaved provider :slight_smile:

The first interview of a developer (in french) working with open data is a good start to grasp the mental model of someone coping with the shortcomings of the current data providers. We plan on doing more of those (another one should be published this week), but this is difficult for me (and a few others in the collective) because we are not used to doing user research. Here are some challenges we are facing:

  • Are we overthinking this, given how simple the service is? There is a prototype with all the bits and pieces (data, documentation, release notes) but no effort whatsoever to make it usable.
  • Once we have a few interviews (maybe 5 from various backgrounds), should we try to get together and do an affinity mapping session?
  • Is it worth trying to shadow a user? I tend to think the interviews are enough because the service is simple.
  • The data sets we are working on are all in French, published by the French government, reason why the communications are also mostly in French. But these data sets are of interest to an international audience. There should be some version of the service that is available in English as well. Not all the documentation because that would be a huge undertaking. Where is the limit?
  • How to get in touch with a non-french speaking developer using data sets?

I hope that, by carefully creating a user centric web site for this deceptively simple data distribution service, we will acquire the necessary tools to improve the user experience of other Free Software & open data related initiatives that need a lot of love :scream:

Any advice you may have would be greatly appreciated !

1 Like

Hi there,

Here are some things that come to mind reading your message.

The first interview of a developer (in french) working with open data is a good start to grasp the mental model of someone coping with the shortcomings of the current data providers.

One of the main temptations when doing user research is to start drawing conclusions straight away. It is important to resist that temptation! Subsequent interviews may confirm the “mental model” you mention, or may show you something completely differnet, and it’s important to stay open to both possibilities. Try not to draw conclusions from a single interview :slight_smile:

this is difficult for me (and a few others in the collective) because we are not used to doing user research.

It is indeed difficult, particularly at the beginning. The good news is: things get better pretty quickly. Doing research with people is not rocket science, and quickly becomes easier the more you do it. You’ve already worked out a lot of the harder bits: you have created an interview guide, you have recruited an interviewee, you have managed to gather their consent to be interviewed, you have done the interview, you have transcribed the interview, anonymised the transcript and published it for all to see. So you have done a ton of work! With all that already in place, subsequent interviews will be much easier.

Are we overthinking this, given how simple the service is?

Certainly not. It always makes sense and always pays off to engage with the people you are designing for. You will learn a ton of stuff that will make your digital platform better in the short and in the long term.

There is a prototype with all the bits and pieces (data, documentation, release notes) but no effort whatsoever to make it usable.

Since you have this in place, you may consider using it as a prompt during your interviews, or do usability testing with it after the interviews are done. By the way, we are giving a free, practical workshop on how to do usability testing in Brussels at the end of January, just in case anybody from the collective would like attend. You can register at https://docs.google.com/forms/d/e/1FAIpQLSfCMcgZq9cG-scnVvkb7_6Z1eL1SszkPIratsdQuF0ikioDWw/viewform

Once we have a few interviews (maybe 5 from various backgrounds), should we try to get together and do an affinity mapping session?

See how you feel after the 5 interviews. If you feel you are hearing many of the same things (much commonality across all interviews), then you are ready to do some analysis. Otherwise, you will need to do some more interviews until you reach that point.

If it’s easy for some members of the collective to get together in the same physical space, an affinity diagram is a great way of working on analysis as a team. If you are not co-located, you can work on the analysis separately and in parallel, then discuss your individal analysis to draw final conclusions together. I’d be happy to provide some guidance on how to do this: just ping me when you are closer to start the analysis.

Is it worth trying to shadow a user? I tend to think the interviews are enough because the service is simple.

I would keep it simple for now. Interviews are a great way to start research because they are quite straigthforward to do. Wait to see what you learn from the interviews, then decide on next steps / research activities based on your findings.

The data sets we are working on are all in French, published by the French government, reason why the communications are also mostly in French. But these data sets are of interest to an international audience. There should be some version of the service that is available in English as well. Not all the documentation because that would be a huge undertaking. Where is the limit?

Given the potential effort involved, I would try to validate first that there is an international audience interested in the data. If there is, make the effort to translate at least the core elements of the platform. Make sure to let contributors know that you are looking for translation contributions, and put everything that needs to be translated in some kind of format people can use to submit translations (e.g. a Git repo on GitLab or similar). This is the great thing about openess: you don’t need to do everything yourselves. Interested people can help you translate the documentation!

How to get in touch with a non-french speaking developer using data sets?

Social media, forums, conferences, data-driven project mailing lists come to mind, but it really depends on who you are actually trying to reach. “Non-french speaking developers using data sets” is quite broad!

I happen to know a developer working on https://opendataservices.coop/ I could ask if he would be willing to speak to you, although I can’t guarantee anything :slight_smile:

Any other questions you have, feel free to ask!

Wow, thanks for taking the time to answer thoroughly, it’s a big help :slight_smile:

I’ll certainly do that. Is there any book / manual you would recommend to read in the meantime to learn about the analysis and how to draw conclusions that are not complete fantasies?

You are correct: we don’t know that for sure. Our best lead is probably to look for organizations that already re-publish / use open data published by the french government (parliament, laws etc.) in a non french speaking environment.

Great ! This is very kind of you.

Thanks for the invitation :tickets: @eraviart tells me he subscribed.

Is there any book / manual you would recommend to read in the meantime to learn about the analysis and how to draw conclusions that are not complete fantasies?

LOL. The most common way of doing this is some form of “thematic analysis”. In industry circles this type of analysis is often done using the affinity diagram technique. Here comes some stuff to read that might help:

The most often cited academic reference on how to do thematic analysis (sorry, this one will be a bit boring) is

https://www.tandfonline.com/doi/abs/10.1191/1478088706qp063oa

Great ! This is very kind of you.

I’ll ask him and let you know if he agrees. If I don’t say anything, it means he has decided not to engage at this time. He is a bit shy :slight_smile:

@eraviart tells me he subscribed.

Yes he has :slight_smile: I’ll be confirming his place today.

2 Likes

For the record here is a link to the PDF of the article, but the images are missing.

Thematic analysis is a method for identifying, analysing, and reporting patterns (themes) within
data.
…
It is not uncommon to read of themes „emerging‟ from the data (although this issue
is not limited to thematic analysis).
…
The language of „themes emerging‟: Can be misinterpreted to mean that themes „reside‟ in the data, and if we just look hard enough they will „emerge‟ like Venus on the half shell. If themes „reside‟ anywhere, they reside in our heads from our thinking about our data and creating links as we understand them.

This is exactly what I’m worried about :slight_smile:

Doing thematic analysis: a step-by-step guide
Phase 1: familiarising yourself with your data
Phase 2: generating initial codes Codes identify a feature of the data (semantic content or latent) that appears interesting to the analyst, and refer to “the most basic segment, or element, of the raw data or information that can be assessed in a meaningful way
regarding the phenomenon”

Phase 3: searching for themes sorting the different codes into potential themes, and collating all the relevant coded data extracts within the identified themes.
Phase 4: reviewing themes 1) read all the collated extracts for each theme, and consider whether they appear to form a coherent pattern, 2) consider the validity of individual themes in relation to the data set, but also whether your candidate thematic map „accurately‟ reflects the meanings evident in the data set as a whole
Phase 5: defining and naming themes
Phase 6: producing the report

That’s a very practical guide, thank you!

1 Like

Adding to what @belenbarrospena has written:

  • The book that Braun and Clarke have written on thematic analysis, Successful Qualitative Research, is also very good. In contrast the their paper you will get far more examples, which is very helpful in practice.
  • I have been a bit frustrated with data analysis instructions for design projects and wrote some instructions on it (as part of an open book)
  • “Is it worth trying to shadow a user?” While not a full on shadowing, I made good experiences with “can you show me how you…” kind of research. If it is about things on a computer, this can be even done remote.
1 Like

4 posts were split to a new topic: Middleware for Open Data & UX

@paulineleon hi! I’d feel a little remiss if I didn’t say that Dataverse (the open source project I work on for my day job) implements the basic features you mentioned (and more). Also, it is translated into French by partners in Canada. And there are five installations in France. I’d be happy to tell you all about it at http://chat.dataverse.org or you’re welcome to call in to our bi-weekly community calls to see if Datavese is a good fit.

Also, we have a design team that I can put you in touch with if that’s of interest. We are currently running a study using Validately and we’re actually looking for participants who have never used Dataverse before and I think you’d be an excellent candidate. We’re gathering feedback on mockups for an upcoming redesign of our dataset page.

For the record, the ongoing user research activity is documented in english (although most of the data and interviews are in French). Today we completed the affinity mapping that identified five emerging themes. The process and the result are available publicly.

Thanks for the pointer, I will look into it.

I’d by happy to participate, please let me know what to do :slight_smile:

1 Like

Great! Please go to https://dataverse.harvard.edu and click the “Feedback” button and then “Learn More” and fill in the form. The screens should look something like this:

1 Like

Thanks, I filled the form :slight_smile:

1 Like

@jdittrich we are at the last stage of our tiny user research project and ready to communicate our results. Would you happen to know if there is a good example of a relatively simple report we could get inspiration from and that (roughly) follows the structure you suggest in your book ?

Thanks again for writing this book, it is a great source of inspiration :slight_smile:

A draft of the user research report is available, in english. This is my first report and I probably made a few mistakes. I would be immensely grateful if anyone has time to proofread it and criticize. It is still a draft and I would not mind rewriting it entirely if it’s massively wrong :slight_smile:

This report on Wikidata use in cultural institutions might be helpful.

Notes:

  • Because I was asked to, I provided counts for how many people said what. I would usually not but such a focus on this)
  • I like that it is linked internally between the sections and to external ressources. This is something not in the book (yet?) but it helps in analysis (what relates to what?) and I hope it helps the interested reader, too.

Hope this helps a bit. Feel free to give me feedback, the communicate your results part of the book is a bit underdeveloped, I guess.

1 Like

@jdittrich thanks for the example report, it is very useful :slight_smile:

Am I right to assume the user research report should contain all the information a UX designer needs to build a prototype ? Using tools such as your quickMockup or pencil (Free Software tools is a requirement). If that’s what such a report is for, it would probably be useful for me to ask a UX designer if the report makes sense to them as it stands and fix it according to their cricitism. But I’m not sure this is the right approach.

Trying to move to the next step with all the required material :left_luggage:

1 Like

Sometimes – often user research goes more towards product or management first. But it depends on what your projects needs, so “can be understood and used for design ideas and evaluation by an UX designer” is a totally valid thing to strive for – although it is no sign of failure or problems if it needs some direct communication in addition and I would absolutely schedule this in.

Getting feedback from your report’s audience and iterating on it is a great idea and as “right” as it can be :slight_smile:

1 Like