Web Analytics and User Experience: An Interview with Louis Rosenfeld

UX Lisbon

Three day conference happening in the lovely Lisbon Portugal. The 2012 event will take place from 16 to 18 May.

See all posts

This May, Louis Rosenfeld will be speaking at UXLX in Lisbon, Portugal. As part of our media partnership with UXLX, Johnny Holland got the chance to interview Louis on a topic that is near and dear to his heart lately, Web Analytics and User Experience. We’d like to thank Louis once again for taking the time out of busy schedule for this interview. Hope you enjoy.

Johnny Holland: You’ve been spending a lot of time writing and presenting on Web Analytics and User Experience lately. From what you’ve learned so far, why do you think web analytics are being used more now than in the past?

Louis Rosenfeld: Design is a more strategic activity than ever before, and with more at stake, we’re all looking for evidence to help us make and validate our decisions.

Plus analytics tools are becoming cheaper and easier to use.  For example, there’s really no excuse for not at least trying out a tool like Google Analytics on your own personal site.  It’s free and, thanks to the great work of brilliant UX people like Jeff Veen, easy to use and understand.

How are teams commonly using analytics to measure the effectiveness of the user experience their product/service is providing?

Traditionally, teams have used basic analytics, like clickstream analysis, to measure conversions of all sorts, such as the percentage of customers that successfully make an online purchase, or the points at which prospective college students fail to complete an online application.  They’ve also compared behaviors among audience segments; for example, are international students having a harder time with that application than domestic students?.

Louis Rosenfeld

The challenge with traditional web analytics is that, while it’s a great way to determine what is happening when users interact with a product, it’s not that good at telling you why they do what they do.  Analytics will help you arrive at some great hypotheses–but for the most part, you’ll have to test them using other user research methods.  And it’s through those more qualitative methods–the one that UX people are more savvy with–that it becomes possible to learn why the new design is better, why the new content titling guidelines are worth following, and so on.

How can analytics help inform UX activities prior to performing user research?

Well, to be clear it is a form of user research, but sure there are many ways.  For example, I suggest reviewing frequent queries before determining what sort of task analysis you might do.  The “what” data of analytics helps you sharpen the “why” questions of qualitative research.

How can it be used following user research?

If you can segment your data by audiences that correspond to your personas, you can incorporate things like common queries and most-accessed documents into those personas.

How are teams misusing information gathered from analytics?

By not going beyond the reports.  By taking that what data–say, a factoid that states that “placing the button to the right of the address field increased conversion 13%”–as an important conclusion on its own, rather than exploring why that’s the case.  If we don’t go further, we only learn something about button positioning for some unique case, rather than something more generalizable about user behavior that can help us solve future design problems.

Is this why we see blog posts and articles on “Left aligned buttons work better for …”? How damaging can posts like that be?

Yes.  Those are very damaging when they’re presented as dogma and taken literally, rather than as points for discussion that, hopefully, lead to learning.  Thinking about the “what” is pretty pointless if you don’t explore the “why”.

Yes.  Those are very damaging when they’re presented as dogma and taken literally, rather than as points for discussion that, hopefully, lead to learning.  Thinking about the “what” is pretty pointless if you don’t explore the “why”.

Is the information being gained by analytics impacting boardroom decisions or business strategies at all? If so how?

I don’t spend a lot of time in boardrooms–for better or for worse–but I imagine that when you manage a large organization, even a non-profit, your main job is to supervise one or more tiers of middle managers.  That doesn’t scale well, so you’ll naturally resort to numbers to help track the performance of middle managers and the products they manage.  Those metrics may be poor, or they may be incomplete; either way, they’re likely to amplify what is already a dangerous approach to making important decisions.

Do you think it’s possible to become too married to the data that comes out of analytics? Where do you draw the line?

Yes, but that’s true of any form of research data, whether it comes from analytics or user testing or an ethnographic study.  Each, on its own, paints a woefully incomplete picture of reality.

But there are bigger risks with analytics data to keep in mind.  First, there is more of it, which will impress some people far more than it should–especially because some of it will be garbage data that should have been scrubbed in the first place.

Second, analytics apps provide us with canned, impressive-looking reports.  While these reports can be useful, they’re generic.  They don’t necessarily pertain to your users’ needs or your organization’s goals.  Analytics data becomes more useful when it helps answer one of your questions, but making it do that takes more effort than many organizations are willing to invest.

My favorite example here is Netflix.  They identify which movies are getting searched most frequently.  Of those, they identify the movies whose pages are getting the most visits.  Of those, they identify the movies which are getting added to customers’ queues least frequently.  That’s a report that’s hugely valuable for Netflix to study regularly; in fact, it’s not so much a report as the answer to a useful question:  “Which popular titles not being added to the queue?”  Either way, it certainly wasn’t anything their analytics application was going to provide automatically.

While writing your latest book (on Site Search Analytics), has anything special stood out to you on the subject?

I’m just shocked at how few people even bother to analyze what their site’s users are searching for–which is why I wanted to write this book and get the topic out there as a valid user research method.  This under-utilization of query data is due in part to ignorance–many of us don’t even know we can get at this data–and partly because it can be a little tricky to set up.

But there aren’t many better–or other–sources of such semantically-rich behavioral data.  In high volumes.  Without the taint of coming from a lab.  In effect, query data is our users telling us what content they want from our sites in their own words.  Really, they’re trying to have a conversation with us; are we listening and learning from it?

Search query data can not only to help us improve our search engine’s performance, but our content and our metadata as well.  So, if you’ve got a search engine, you’ve got query data somewhere.  Lots of it, likely.  Why wouldn’t you want to learn from it?

What can User Experience learn from Web Analytics? What can Web Analytics learn from User Experience?

This is one of those questions that could take a few pages to answer.  For sake of brevity, let me distill it this way:  Web Analytics is great at measuring and monitoring how well an organization is performing at meeting its goals (as expressed as Key Performance Indicators).  in other words, WA tells us about the world that we know.  UX methods, conversely, help us suss out patterns and outliers in data–they expose the world that we don’t know.  Each results in an incredibly valuable perspective, but the organizations that combine these will realize benefits far greater than the sums of their parts.

You describe the nature and personality of Web Analytics and User Experience as polar opposites. What if they had a baby together though? What would it look like? How would it behave?

That’d be one amazing (if somewhat schizophrenic) child.  He would pepper its parents with incessant “why?” questions, like all kids do, but just as many “what” questions as well.  He’d probably be quite odd-looking, but that’s another story.

UX Lisbon 2011

Louis Rosenfeld will be  speaking at UX Lx: User Experience Lisbon, one of Europe’s premier user experience events. The second annual UX Lx conference takes place May 11-13, 2011 in Lisbon, Portugal.

Brad Nunnally

Brad Nunnally is a User Experience Design Consultant at Perficient based in St. Louis, MO. Aside from writing, plotting UX world domination, and tweeting a whole bunch , he fills his time playing with his son and dog.

2 comments on this article

  1. Pingback: Web Analytics and UX | earlf.com - Waterloo Web Design & Development for Kitchener Waterloo Guelph Cambridge Ontario

  2. Pingback: » Web Analytics and User Experience: An Interview with Louis Rosenfeld Johnny Holland – It’s all about interaction » Blog Archive « Sales is an art and a science!