A Story About a Crappy UX Study

The study was a brilliant piece of work. At least that’s what I thought.

Related posts:

The product manager and lead engineer were happy when I joined the team. When I talked with them about the study I planned, they seemed interested. We agreed on interviewing 40 people with the goal of identifying user needs and uncovering current product pain points.

The product manager wanted to use the results to help develop a detailed specifications document, which will guide the development team. I planned a study that involved four groups of participants – teenagers, students, high-tech employees, and senior citizens. I prepared a detailed discussion guide, then recruited and scheduled all 40 participants by myself. Some of the study sessions were held in our offices and some at users’ homes. The product manager and lead engineer did not observe or join any of the interviews. I didn’t care so much. I was so excited about this project.

When I was done, I sat down to analyze the huge amounts of data. It took me three weeks to complete, and in the end I proudly published a detailed report complete with screenshots, in-context pictures of users, video highlights, quotes, findings, smart insights, and recommendations.

The results collected dust.

I gave a presentation to the entire team, during which the lead engineer and some other team members argued that my data was flawed and that they thought we should develop things other than what I was suggesting. Someone said something about the users that I interviewed and that they were not the right audience. The product manager just sat there and didn’t say a word. In the following weeks, the product manager published a specification document and the team began developing the product. The document was not based on my study findings and recommendations – far from it. I heard from someone that the product manager interviewed some people, but I had no idea who, how many, or what questions were asked.

I felt really bad. Actually, a more accurate description is that I was very angry. How could they behave like that? How could this happen? Why did they not follow my recommendations? They were acting like typical product managers and engineers, I thought to myself. They just can’t develop empathy toward users. All they care about is what they think.

Takeaways

  • Treat any UX research as the team’s research, not yours;
  • If they feel it’s theirs as much as you feel it’s yours, you are on the right track;
  • Ensure the study either answers specific research questions, or helps cover a team knowledge gap, or helps drive a team decision;
  • Have the team decide which participants should be included in the study;
  • Schedule the study in a place and time that is convenient for your team.

Share your war stories in the comments below.

Getting Buy-in for UX Research workshop

Tomer Sharon will be giving a a half-day interactive workshop for researchers, designers, and anyone who practices UX research and wants to get more buy-in. The workshops will be held on September 21 at LeanUXDenver and on September 25 at The Web and Beyond.

Tomer Sharon

Tomer Sharon is a User Experience Researcher at Google New York and author of It's Our Research: Getting stakeholder buy-in for UX research projects. He is @tsharon on Twitter.

16 comments on this article

  1. Stewart Dean on

    This is often why user reviews happen in labs rather than in the users environment. Ideally we always talk to the users in the place they will be using the thing we’re developing but often effecting change and turning team members into user advocates is a bigger battle than finding out what our users do. This also extends through to the design process. One aspect of lean UX I embrace is the idea working collaboratively with a team, not just delivering the final verdict unto the poor non UX folks. A good UX person will be open to ideas and be able to turn these into a cohesive solution.

  2. Tim McCoy on

    The transformative powers of UX interviews are very strong but ONLY work first-hand. It’s vital to bring product managers, developers, sales and marketing folks, etc. along with you.

    Just exposing each member of a team to one interview will give your colleagues the chance to develop the empathy necessary to recognize the larger findings of the study.

    I once ran a study with a team that included a product manager who had been a user (so thought he understood user needs) and a lead developer with tons of rogue ideas that were interesting but didn’t match user need. Seeing them react to and process interview sessions was like watching the sky open after a rainstorm.

    Don’t take no for an answer when you ask for participation, because without it the only person who learns from the interviews is you.

  3. Andrew Mottaz on

    Nice article, and common experience. The only thing I have ever found effective to break this barrier is to have the engineers and product managers themselves in the meetings with end users. If all your data confirms what product management and engineering already thinks, its fine. If you are going to change their view on something, they have to hear it first hand. I’ve had engineers leave a user testing session and say ‘wow, I never would have believed that’. When they read a document or hear me expound, the response is ‘I don’t believe that’

  4. Jon Anderson on

    VERY common experience, sadly. I’ve found I’m much more productive when I conduct the testing as a way to vanquish behaviors that need to be done away with, or to support behaviors that might not seem feasible to the stakeholders. The few times that I’ve done comprehensive reports I’ve had a similar experience to yours. Your list of “Takeaways” is dead on..

  5. Ingjerd Jevnaker on

    I have had the exact same experience. I think that the key is that the UX people needs to be part of writing the specifications afterwards to ensure that user feedback and insight is taken on board in the actual project spec.

  6. Ingjerd Jevnaker on

    Sometimes it also helps to present the findings directly to the client or key business people and get their buy-in. They can help push the engineers in the right direction as well.

  7. Alex Sherman (@alex_sherman) on

    Not all product managers act like this. :)

  8. James on

    It seems like a pretty strange set-up where the Project Manager is producing the specification document. As UX designer, I see it as my job to take the research forward and turn it into a specification/brief that developers can work from. If I just presented my research to engineers I’d expect them to say ‘So what? What do you want me to do with that?’.

    I also couldn’t imagine people arguing the toss about research methods and participants. That’s my area. I don’t tell the Project Manager that I’m unhappy with the margin or the developer that their code is crappy.

    Overall, I’d say take the exact opposite away from your anecdote – it sounds like a classic case of too many cooks. I’d say you need less colloborative working and more people to sticking to what they’re good at.

  9. dan turner on

    I had a similar experience but in a different direction, in a way. Coming in on a project to adapt a quickly built web app for users in developing countries, I first put the existing site through a few rounds of standard user testing. I was looking forward to including the engineer who’d built and laid out the web app, but made sure to prep him about the usual “if you have a problem, it’s not you, it’s our design” talk we give to participants.

    My concern was that if we just brought him the empirical results showing user confusion, etc., he’d see the UX process as just a black box that outputted “you did it wrong”. But after getting to see how users — that is, the people who did not actually build the site — react to the site, he was really excited about UX and put in extra time and effort as we moved forward in the larger project.

  10. Erin young on

    I’m curious then, about a situation in which the team can’t be present. Perhaps they are handling others tasks or focused on other projects. Perhaps the expense of having ten people along for each interview is too great for a lean projwct to bear. Are you suggesting no research at all in that case?

  11. Rob Gillham on

    Very honest and revealing, thanks! Let’s face it, we’ve all been there…

  12. Alexis D on

    Interesting, and quite familiar. The flip side of involving product managers, owners, etc in the interviews, is again, they’re keen to be involved, but pick and choose which ones they come too. At the time of presenting the findings, the recommendations I made were either championed or knocked down based on the responses at the interview they attended. Still working on achieving that win-win situation… Thanks for your insight.

  13. Tomer Sharon on

    Thanks for all of these great comments.

    Erin: I’m not suggesting not to do research if nobody shows up. I’m saying it is more likely nobody would care about the results if that’s the case. One rare case is that the team trusts you so much that they don’t feel they need to attend. If you feel the same way, go for it. If that’s not the case, I highly recommend on having a serious, open conversation and try to agree that at least one-two team members attend sessions. Alternatively, you can schedule a workshop during which you show selected (longish) videos and have team members come to conclusions.

    If there’s a will, there’s a way. I think Bill Clinton once said it about the peace process in the Middle East. So far, no will hence no way. I hope our stakeholders have will.

  14. MsFlower on

    The product manager usually comes from all the walks. When the company don’t believe UX, usually most of them don’t, get to know the company and people better is the key before doing the research. In a real-life case, research results can be as dangerous as being against your shareholder’s idea and your colleague’s previous works, plus it’s not deliverable business. I bet people bitch about a researcher regarding their ‘invisible hard work’ as part of the office drama. So, research is good, timing is bad, cooperation needs more works.

  15. Cindy Chiuchiolo on

    I’m a usability recruiter, and part of my process is scheduling usability activities around the calendar of one developer (usually the lead dev). Also, one of the rules here is that if there isn’t at least one developer attending, the session doesn’t run. The theory behind me scheduling around one of their calendars is that at least one will show, but I know that making sure they attend still requires effort on the part of the usability specialists. I don’t know if that applies to even the smallest of phone interviews, but at least for a usability session with a prototype, there’s always a developer present here. Your story gives insight as to why we do things this way.

  16. James Skinner on

    Nothing quite as disheartening as a client who says “the users might have said that, but they’re wrong. I know what they want”. For me, the key is to spend a lot of time up front with the key stakeholders, and getting an impression of how genuinely they buy in to the whole process. If they don’t, bail.