“To design is to have a ‘project’. Getting the design process moving is to expose and transform this ‘project’ in a conversation with those that it might eventually affect” (Buur, Binder, & Brandt, 2000).

In the early stages of design, rather than evaluate or validate specific user requirements or priorities, we are interested in exploring possibilities. As the opening quote suggests, we seek to engage with the various stakeholders the design project may eventually effect and gain an understanding of the unique design situation from their perspective. In Zimmerman et al.’s  (2004) framework for discovering and extracting knowledge during the design process, this is known as the Discovery phase of design. In this article we introduce Mobile Diaries as a field work method that can be utilised in the early stages of design to immerse into people’s everyday life.

This exploratory approach to self-reporting allows participants  to create and share a rich picture of their world, be they grandmothers, bankers, students, young parents or employees. In this article we describe Mobile Diaries, and provide examples of the kinds experiences they can enable.

A little background on self-reporting

In self-reporting, research participants are responsible for the data collection, allowing for the gathering of contextual data over-time and in situ, without the physical presence of researchers. Self-reporting can provide access into the private, personal and mobile aspects of people’s lives that are often difficult, or impossible, to access through traditional methods such as observation or interviews. The sustained personal reflection inherent in self-reporting makes available aspects that would otherwise remain tacit. So much of our lives are routinised and automatic, it is not until we are asked to document or consider certain activities that we are able to identify key junctures in our own understanding of a topic or a behaviour.

Self-reporting studies can take many different forms and the degree of formal structure is one of the things that differentiates approaches and determines the type of material collected. For example in the Electronic Sampling Method approach known as ESM (Larson & Csikszentmihalyi, 1983) or beeper studies, the participant is directed to systematically log specific things at specific times. In more open-ended approaches (such as cultural probes (Gaver, Dunne, & Pacenti, 1999) or visual diaries) data collection is only semi-structured around a particular topic. In this case participants are treated as active contributors and interpreters in the design process and select what, how and when to report. This encourages more playful and creative representations, important to an explorative and collaborative approach.

Over the last 10 years digital, online and mobile technologies have been incorporated into self-reporting methods in a range of ways (see end of article for some examples of other studies and platforms). These everyday tools can be easily integrated into people’s daily lives and support the generation of a range of different media forms such as video, images, text and audio.

Mobile Diaries

Mobile Diaries are a hybrid method that incorporate many of the creative and playful aspects of probes and emphasise the daily reflection of visual diaries. A range of different analog and digital technologies are used that allow participants to share and reflect on various dimensions of their day-to-day life.

A typical study

The exact design of the study (as always) is dependent on a number of factors such as:

• A definition of the problem space;
• The goals and objectives of the particular project;
• The theme of the study (e.g is it a personal project or focused on the workplace);
• Budget (how many people can we recruit and how, what kind of incentives might be required);
• The profile of the participants (e.g teenagers, adults or whole families); and
• Their current technology knowledge/competence and use (e.g how they might respond to the technology involved, how open are they to using new technologies).

Generally studies run from 1-3 weeks with between 1-10 participants. Topics explored depend on the study but could include, for example: sustainability in your everyday life; the role of mobile technologies in your life; or a ‘behind the scenes’ look at your job. Participants receive a ‘Mobile Diary Pack’ with various tools and instructions which direct the data collection around the particular design topic.

Mobile Diary Tools

A number of custom platforms have been developed to support online diaries, however, to date we have preferred to configure Mobile Diaries from existing platforms such as WordPress and Tumblr as this gives us greater flexibility over format, functionality and cost. Below we show the packs from a typical study (the list of available tools is growing and changing all the time, here we show typical ones we have used in the past).

During the study

Over the period of the study participants create collages, mind maps, videos and blog messages and send in mobile reports which appear on the blog. They also receive prompts, questions and reminders via the mobile phone and the blog.  The conversation is bi-directional: as we are receiving reports we are also responding with new questions or digging deeper into particular areas, and potentially redirecting the focus of the study as a result.

What do we see and learn?

The output of Mobile Diaries is a particularly provocative, experiential and sensorial insight into participant’s lives. As one of our clients described it “[we were able to] hear in people’s own words the challenges and learn about the context of sustainability in their day-to-day lives”. We share some examples below of the kinds of material generated and shared through this approach.

Life as it happens

Using their phone, participants capture images, text and audio and send this to the blog throughout the day. These reports give us a deeper appreciation of the activities that make up people’s daily lives and we are able to ‘see for ourselves’ actual examples and instances of things that might otherwise be anecdotal. Through these reports we can track events, locations, and a sense of participant’s emotions across the days and weeks. Over time, daily rhythms and habits emerge. The (near) real-time reporting increases the sense of immersion in people’s lives as we experience the activities ‘as they happen’ (Masten & Plowman, 2003).  This is complemented by more reflective accounts at the end of the day via the blog or with the video camera.

Personal Spaces and intimate stories

The use of video encourages in-depth descriptive accounts of events and surroundings from the participant’s perspective. The stills above are from one participant’s tour of their apartment building, which focused on areas relating to sustainability in the home. By giving participants video cameras and asking them to take us on a tour of their home we are able to explore and wander with the participant. This reveals some evocative spaces otherwise inaccessible to a design researcher; in addition to the explicit content being shared the video also conveys emotion and expression.

In another study focused on teenagers and their relationship to technology one participant gave us a tour of his shed, playing instruments as he told stories about the importance of this particular space to him.  These personal stories bring us closer to the participant’s world creating a sense of intimacy and proximity to the participants which is difficult to replicate in a one on one interview, discussion group or even during participant observations.

Inner thoughts and feelings, moments and metaphors

In addition to descriptions of external events and activities, Mobile Diary reports also capture emotions, feelings and inner thoughts. The examples above show emotional reactions and descriptions of personal feelings at particular moments in time. In some, the participant’s have used objects to serve as metaphors or symbols for representing emotions or ‘states’. This allows the participant to share inner thoughts and feelings that might have otherwise remained hidden. The open, honest and personal nature of these reports fosters empathy, often describing experiences we can relate to. That some are delivered in (near) real-time further increases the sense of connection; in that moment, we knew something of what the participant was thinking and feeling.

Action & Transformation

“Not only did it help us, the impact on participants and their way of thinking about sustainability was really interesting… the project really opened their eyes to their own patterns and behaviour and sparked some changes and definitely increased awareness” (feedback from client).

The process of self-reporting is an intervention designed to allow people to self-reflect and share aspects of their daily life; this process can also trigger participants to question their choices and everyday behaviours (Grinter & Eldridge, 2001).  The content of the Mobile Diary packs and the nature of the questions included can provoke new realisations and possibilities. For example, our self-reporting studies into sustainability in everyday life resulted in participants questioning personal behaviours and making changes in their lifestyles^[i]. For one participant, a discussion about sustainability with flatmates led to the installation of a composting unit in the household.

The interventionist nature of the method can be more intentionally activated through the inclusion of specific activities and questions within the packs. For example, in one study into sustainability we included sustainability challenges – new lifestyle habits -  that participants were asked to try and document throughout a week. These activities were particularly provocative at revealing emotional and infrastructural barriers to behaviour change.

And then…?

As the image below suggests, material generated from Mobile Diaries can be used in numerous ways. Mobile Diaries externalise aspects of people’s everyday lives through visual, tangible artefacts. These become shared resources that help us to understand current practices, provide a spring-board for ideation and allow us to envision how any future design might be taken up within the existing ecology of the participants life. For designers, the visual nature of the material allows for more active interpretation in ways not possible with written research reports.  For participants, the process of doing the Mobile Diaries means they are better equipped to reflect on and analyse their own practices, during follow up interviews and workshops (similar to primer tools), becoming active interpreters of the material and what it might mean for future designs.

The material generated through Mobile Diaries is not something to be reduced down into a traditional written report. The raw form of the material and the subjective picture it provides of the participants’ lives and world-view are essential to its immersive quality and its value for fostering empathy and connection with participants (Mattelmäki, 2005; Mattelmäki & Battarbee, 2002) ^[ii].

However, this does need to be balanced with normative business expectations of a ‘research outcome’, and the need to synthesise the data in a meaningful way for the client to then transmit to other stakeholders. Effective ways to share this tangible and personal material with those who were not directly involved is the subject of current research, e.g., (Sleeswijk Visser 2009). Our process includes the development of a multimedia ‘report’ that can support all the different formats of material generated. These reports introduce the participants through their own words and images, illuminate the themes that have emerged and identify some future possibilities to be considered. We have also found significant value in creating opportunities for co-interpretation of the material by clients and other designers, whilst this can be more time consuming, it is utlimately a more effective use of the material than simply ‘handing off the research’.

We have also found that the value of such methods is greater than their role as data collection activities. A personal connection is made with participants that can be of value well beyond the particular study. However, there is a tension between this and the day-to-day realities of client and agency practices and models which focus on deliverables and project phases, in between which there can be significant lags or breaks. There is still work to be done to articulate and communicate the value of such methods beyond their capacity to generate data ‘about people,’ and to embrace their ability to involve participants in a more ongoing and sustained way throughout the design process.

Mobile Diaries are a playful, immersive experience for the design team that allow us to discover something of the messy intricacies of participant’s daily life, valuable for both inspiring and grounding the design process as well as engaging directly with stakeholders. We’d love to connect with other designers employing similar approaches.

Acknowledgments

Big thanks to Chris Gaul for images & Will Evans for feedback and comments. Thanks also to Dr Toni Robertson and the @IDHuPLab at UTS, Digital Eskimo, Zumio, our clients & participants N.B Examples come from specific studies in which permission was granted for publication for the purposes of research. Our approach has been heavily informed by the ongoing research into generative methods inspired by Cultural Probes (Gaver, et al., 1999) and the work of Liz Sanders (www.maketools.com) as well as other research into self-reporting studies such as (Hulkko, Mattelmäki, Virtanen, & Keinonen, 2004; Masten & Plowman, 2003; Palen & Salzman, 2002).

References

Buur, J., Binder, T., & Brandt, E. (2000). Taking Video Beyond ‘Hard Data’ in User Centred Design. Participatory Design Conference PDC2000, New York, NY, USA.
Gaver, B., Dunne, T., & Pacenti, E. (1999). Design: Cultural Probes. Interactions, 21-29.
Grinter, R., & Eldridge, M. (2001). y do tngrs luv 2 txt msg. ECSCW, Amsterdam.
Hulkko, S., Mattelmäki, T., Virtanen, K., & Keinonen, T. (2004). Mobile Probes. NordiCHI 04, Tampere, Finland.
Larson, R., & Csikszentmihalyi, M. (1983). The Experience Sampling Method. In H. Reis (Ed.), Naturalistic approaches to studying social interaction: New Directions for Methodology of Social and Behavioral Science: Jossey-Bass, San Francisco.
Masten, D., & Plowman, T. (2003). Digital Ethnography: The next wave in understanding the consumer experience. Design Management Journal, 14(2), 75-81.
Mattelmäki, T. (2005). Applying probes – from inspirational notes to collaborative insights. CoDesign, 1(2), 83-102.
Mattelmäki, T., & Battarbee, K. (2002). Empathy Probes Paper presented at the PDC 2002, Malmö, Sweden.
Palen, L., & Salzman, M. (2002). Voice-mail diary studies for naturalistic data capture under mobile conditions.  CSCW, Louisiana, USA.
Sleeswijk Visser, F. (2009). Bringing the everyday life of people into design (PhD Thesis), Technische Universiteit Delft, Delft.
Zimmerman, J., Forlizzi, J., & Evenson, S. (2004)  “Taxonomy for Extracting Design Knowledge from Research Conducted During Design Cases.” Futureground 04, Melbourne, Australia.

Additional examples of other remote self-reporting techniques & studies & mobile/online tools

Digital Ethnography (pdf)

Digital Diaries

Digital Cultural Probes

Sticky Research (Video Diaries)

Revelation

7daysinmylife.com

[i] There is no way for us to tell how permanent these changes were, we can only be sure that particular practices were bought to people’s attention, and steps towards change were made.
[ii] The original Cultural Probes (Gaver 1999) were not designed to gather specific information, but rather to be a source of inspiration and empathy. Rather than being codified, transformed or translated into a report, probe material was designed to stand on its own as a rich visual resource for designers.

In-person user research used to be the only game in town, and as with most industry practices, its procedures were developed, refined, standardized, and then became entrenched in the corporate R&D product development cycle. Practically everything gets tested in a lab, hallway, or conference room nowadays: commercial web sites, professional and consumer software, even video games. But nowadays we’ve got remote usability testing.

This is a revised chapter from Rosenfeld Media's 'Remote Research'

Part of the appeal of formal lab research was that it provided a scientific-seeming basis for making decisions by using observational data, instead of someone’s error-prone gut instincts. Stakeholders appreciated the firm protocol and apparent reliability of properly managed lab research. But for all of us who have sat through formal studies with two-way mirrors, we know that there is a lot of bullshit that goes on—participants pretending to care, moderators pretending to understand, and stakeholders pretending to be open-minded. The appeal of what the kids call “guerilla testing”—informal testing, where you simply grab someone within shouting distance and ask them to use your interface—is clear. It’s easy, fast, and can produce great results. Lots of user research practitioners continue to use in-person methods because it’s what people have been doing for a long time.

There’s nothing specifically wrong with in-person research. But there is that whole Internet thing that’s been happening. It does have some unique properties we can take advantage of to do things that weren’t possible with old-school research. Like these things:

Insane Cost Savings
Usertesting.com is $39 per user. Compare that to flying to Chicago for three days to watch twelve people talk behind a two-way mirror, and that’s thousands of dollars in savings. Rolf Molich has been organizing the Comparative Usability Evaluation study (CUE) for eight years, where different usability methods and teams independently evaluate the same site. He knows something about comparing different research techniques, and makes the point that while there are advantages and disadvantages to a remote method like UserTesting.com, the “price/performance ratio was amazing” (that was before a price increase, but the cost is still quite low). Beyond travel expenses, other costs associated with in-person testing may be reduced or eliminated when you test remotely. Unless you’re doing guerrilla testing. With tools like Silverback, guerrilla in-person methods don’t have to cost much more than remote, but you are usually more limited by the audience. So in terms of cost comparison, let’s just say that remote testing will usually offer a big cost savings.

Time-aware research
Catching people in the middle of a task with a web or software intercept like ethnio (note: this is a product of ourselves) and calling them within a few seconds to share their screen and watch them use a tool remotely on their own timeline. It’s a degree of accuracy that never existed before. You could argue that it’s ethnographic in a way that is not possible with physical observation, but you could also spend your whole life arguing about that. Let’s not. Tools like Revelation and the Track Your Happiness project at Harvard use native timelines to gain insight. That’s a really big deal.

By now UX researchers are familiar with the importance of understanding the usage context of an interface—the physical environment where people are normally using an interface. Remote research opens the door to conducting research that also happens at the moment in people’s real lives when they’re performing a task of interest. This is possible because of live recruiting (the subject of Chapter 3 of the book), a method that allows you to instantly recruit people who are right in the middle of performing the task you’re interested in, using anything from the Web to text messages. Time-awareness in research makes all the difference in user motivation: it means that users are personally invested in what they’re doing because they’re doing it for their own reasons, not because you’re directing them to; they would have done it whether or not they were in your study.

Consider the difference between these two scenarios:

1. You’ve been recruited for some sort of computer study. The moderator shows you this online map Web app you’ve never heard of and asks you to use it to find some random place you’ve never heard of. This task is a little tricky, but since you’re sitting in this quiet lab and focusing—and you can’t collect your incentive check and leave until you finish—you figure it out eventually. Not so bad.
2. You’ve been planning a family vacation for months, but you’ve been busy at work so you procrastinated a bit on the planning, and now it’s the morning of the trip and you’re trying to quickly print out directions between finishing your packing and getting your kids packed. Your coworker told you about this MapTool Web site you’ve never used before, so you decide to give it a shot, and it’s not so bad—that is, until you get stuck because you can’t find the freaking button to print out the directions, and you’re supposed to leave in an hour, but you can’t until you print these damn directions, but your kids are jumping up and down on their suitcases and asking you where everything is. Why can’t they just make this stupid crap easy to use? Isn’t it obvious what’s wrong with it? Haven’t they ever seen a real person use it before?

Circumstances matter a lot in user research, and someone who’s using an interface in real life, for real purposes, is going to behave a lot differently—and give more accurate feedback—than someone who’s just being told to accomplish some little task to be able to collect an incentive check. Time-awareness is an important concept, so we’ll bring it up again throughout this book to demonstrate how the concept relates to different aspects of the remote research process (recruiting, moderating, and so on).

Circumstances matter a lot in user research, and someone who’s using an interface in real life, for real purposes, is going to behave a lot differently—and give more accurate feedback—than someone who’s just being told to accomplish some little task to be able to collect an incentive check.

Technological ecosystem
Some interfaces just don’t make any sense to test outside their intended usage environment. If you need the users to have their own photos and videos to use in a video editing tool, having them bring their laptop or media to a lab is an amazing hassle. Or, let’s say you’re testing a recipe Web site that guides users step-by-step through preparing a meal; it wouldn’t make much sense to take people out of their kitchen, where they’re unable to perform the task of interest. When this is the case, remote research is usually the most practical solution, unless the users also lack the necessary equipment. We also call this the participant’s “technological ecosystem” because it implies that their devices and computing environment have an impact on how they interact.

Democratization of user testing
That’s right, I said it. Democracy. As in, anyone in the world no matter how far removed from their potential audience can conduct user testing with less obstacles than before. After ten years of user research, 260 studies, and 3,000 participants at bolt | peters, we’ve noticed a trend lately that more people are doing their own research than ever before. And it’s great. There’s no reason to hire a specialist to observe real-world technology behavior. And that’s coming from a specialist.

Geographic Diversity
Even if you do have a lab, the users you want to talk to may not be able to get to it. This is actually the most common scenario: your interface, like most, is designed to be accessed and used all around the world, and you want to talk to users from around the world to get a range of perspectives. Will Chinese players like my video game? Is my online map widget intuitive even for users outside Silicon Valley? Big companies like Nokia and Microsoft are often able to conduct huge, ambitious research projects to address these questions, coordinating research projects in different labs around the world, flying researchers around in first class. If you don’t have the cash for an international longitudinal Gorillas-in-the-Mist project, then remote research is a no-brainer solution. If you can’t get to where your users are, test them remotely.

And Why Not?

Both in-person and remote UX research share the same broad purpose: to understand how people interact and behave with the interface you’ve made. There’s no need to set up a false opposition between the two approaches—one isn’t inherently better than the other. Despite the versatility of remote research, there are lots of reasons you might want to conduct an in-person study instead, most of which have to do with timing, security, equipment, or the type of interaction you want to have with participants.

Both in-person and remote UX research share the same broad purpose: to understand how people interact and behave with the interface you’ve made. There’s no need to set up a false opposition between the two approaches—one isn’t inherently better than the other.

Security
Security is often a concern for institutions like banks and hospitals, which deal in sensitive information, or companies concerned with guarding certain types of intellectual property. If you’re testing a top-secret prototype, you obviously don’t want to let people access something from their home computer, where it could be saved or screen-captured. On the other hand, you might also be doing a study on users who would be secretive about sharing what’s on their screen—government employees, doctors, or lab technicians, for instance. Either way, you’ll want to test users in a controlled lab environment to keep things confidential, especially if what you’re testing is so hush-hush that you’ve got to have your users sign a nondisclosure form.

Inability to use screen-sharing
You might also want to use a lab if your users are unable to share their screen over the Internet, for whatever reason. Some studies (of rural users, cybercafe patrons, etc.) may require you to talk to users who don’t have reliable high-speed Internet connections, who own computers too slow or unstable to use screen sharing services effectively, or who have operating systems incompatible with the screen sharing tools you’re using. These restrictions only apply to moderated studies, for which you need to see what’s on your users’ screens.

The need for special equipment
Depending on the interface you’re testing, you may require certain special software or physical equipment to run the study properly; this is most often the case with software that’s still under development. Getting users to install and configure tools to run elaborate software can be a pain (though that’s not unheard of), and requiring users to have certain equipment can make recruiting needlessly difficult.

The importance of seeing the user’s body
Some kinds of research will require you to study certain things about the user that are difficult to gather remotely. UX research has recently begun using eye-tracking studies, and for that kind of study, you’d need to bring the users to the eye-tracking device. Other studies might require you to attend to the participants’ physical movements, which may be difficult to capture with a stationary webcam.

Conclusion

You don’t necessarily have to choose between lab and remote methods. You can even conduct multiple studies on the same interface, using the findings from one study to add nuance to another. Probably excessive for the average study, but for really large-scale projects where you just want to gather every bit of information you can (a new version of a complex software program, an overhauled IA, etc.), being comprehensive can’t hurt.

You should have a good idea of whether or not remote research suits you. Give it a try—if it’s not your thing, you can always go back to lab testing. We won’t tell anyone.