“To design is to have a ‘project’. Getting the design process moving is to expose and transform this ‘project’ in a conversation with those that it might eventually affect” (Buur, Binder, & Brandt, 2000).

In the early stages of design, rather than evaluate or validate specific user requirements or priorities, we are interested in exploring possibilities. As the opening quote suggests, we seek to engage with the various stakeholders the design project may eventually effect and gain an understanding of the unique design situation from their perspective. In Zimmerman et al.’s  (2004) framework for discovering and extracting knowledge during the design process, this is known as the Discovery phase of design. In this article we introduce Mobile Diaries as a field work method that can be utilised in the early stages of design to immerse into people’s everyday life.

This exploratory approach to self-reporting allows participants  to create and share a rich picture of their world, be they grandmothers, bankers, students, young parents or employees. In this article we describe Mobile Diaries, and provide examples of the kinds experiences they can enable.

A little background on self-reporting

In self-reporting, research participants are responsible for the data collection, allowing for the gathering of contextual data over-time and in situ, without the physical presence of researchers. Self-reporting can provide access into the private, personal and mobile aspects of people’s lives that are often difficult, or impossible, to access through traditional methods such as observation or interviews. The sustained personal reflection inherent in self-reporting makes available aspects that would otherwise remain tacit. So much of our lives are routinised and automatic, it is not until we are asked to document or consider certain activities that we are able to identify key junctures in our own understanding of a topic or a behaviour.

Self-reporting studies can take many different forms and the degree of formal structure is one of the things that differentiates approaches and determines the type of material collected. For example in the Electronic Sampling Method approach known as ESM (Larson & Csikszentmihalyi, 1983) or beeper studies, the participant is directed to systematically log specific things at specific times. In more open-ended approaches (such as cultural probes (Gaver, Dunne, & Pacenti, 1999) or visual diaries) data collection is only semi-structured around a particular topic. In this case participants are treated as active contributors and interpreters in the design process and select what, how and when to report. This encourages more playful and creative representations, important to an explorative and collaborative approach.

Over the last 10 years digital, online and mobile technologies have been incorporated into self-reporting methods in a range of ways (see end of article for some examples of other studies and platforms). These everyday tools can be easily integrated into people’s daily lives and support the generation of a range of different media forms such as video, images, text and audio.

Mobile Diaries

Mobile Diaries are a hybrid method that incorporate many of the creative and playful aspects of probes and emphasise the daily reflection of visual diaries. A range of different analog and digital technologies are used that allow participants to share and reflect on various dimensions of their day-to-day life.

A typical study

The exact design of the study (as always) is dependent on a number of factors such as:

• A definition of the problem space;
• The goals and objectives of the particular project;
• The theme of the study (e.g is it a personal project or focused on the workplace);
• Budget (how many people can we recruit and how, what kind of incentives might be required);
• The profile of the participants (e.g teenagers, adults or whole families); and
• Their current technology knowledge/competence and use (e.g how they might respond to the technology involved, how open are they to using new technologies).

Generally studies run from 1-3 weeks with between 1-10 participants. Topics explored depend on the study but could include, for example: sustainability in your everyday life; the role of mobile technologies in your life; or a ‘behind the scenes’ look at your job. Participants receive a ‘Mobile Diary Pack’ with various tools and instructions which direct the data collection around the particular design topic.

Mobile Diary Tools

A number of custom platforms have been developed to support online diaries, however, to date we have preferred to configure Mobile Diaries from existing platforms such as WordPress and Tumblr as this gives us greater flexibility over format, functionality and cost. Below we show the packs from a typical study (the list of available tools is growing and changing all the time, here we show typical ones we have used in the past).

During the study

Over the period of the study participants create collages, mind maps, videos and blog messages and send in mobile reports which appear on the blog. They also receive prompts, questions and reminders via the mobile phone and the blog.  The conversation is bi-directional: as we are receiving reports we are also responding with new questions or digging deeper into particular areas, and potentially redirecting the focus of the study as a result.

What do we see and learn?

The output of Mobile Diaries is a particularly provocative, experiential and sensorial insight into participant’s lives. As one of our clients described it “[we were able to] hear in people’s own words the challenges and learn about the context of sustainability in their day-to-day lives”. We share some examples below of the kinds of material generated and shared through this approach.

Life as it happens

Using their phone, participants capture images, text and audio and send this to the blog throughout the day. These reports give us a deeper appreciation of the activities that make up people’s daily lives and we are able to ‘see for ourselves’ actual examples and instances of things that might otherwise be anecdotal. Through these reports we can track events, locations, and a sense of participant’s emotions across the days and weeks. Over time, daily rhythms and habits emerge. The (near) real-time reporting increases the sense of immersion in people’s lives as we experience the activities ‘as they happen’ (Masten & Plowman, 2003).  This is complemented by more reflective accounts at the end of the day via the blog or with the video camera.

Personal Spaces and intimate stories

The use of video encourages in-depth descriptive accounts of events and surroundings from the participant’s perspective. The stills above are from one participant’s tour of their apartment building, which focused on areas relating to sustainability in the home. By giving participants video cameras and asking them to take us on a tour of their home we are able to explore and wander with the participant. This reveals some evocative spaces otherwise inaccessible to a design researcher; in addition to the explicit content being shared the video also conveys emotion and expression.

In another study focused on teenagers and their relationship to technology one participant gave us a tour of his shed, playing instruments as he told stories about the importance of this particular space to him.  These personal stories bring us closer to the participant’s world creating a sense of intimacy and proximity to the participants which is difficult to replicate in a one on one interview, discussion group or even during participant observations.

Inner thoughts and feelings, moments and metaphors

In addition to descriptions of external events and activities, Mobile Diary reports also capture emotions, feelings and inner thoughts. The examples above show emotional reactions and descriptions of personal feelings at particular moments in time. In some, the participant’s have used objects to serve as metaphors or symbols for representing emotions or ‘states’. This allows the participant to share inner thoughts and feelings that might have otherwise remained hidden. The open, honest and personal nature of these reports fosters empathy, often describing experiences we can relate to. That some are delivered in (near) real-time further increases the sense of connection; in that moment, we knew something of what the participant was thinking and feeling.

Action & Transformation

“Not only did it help us, the impact on participants and their way of thinking about sustainability was really interesting… the project really opened their eyes to their own patterns and behaviour and sparked some changes and definitely increased awareness” (feedback from client).

The process of self-reporting is an intervention designed to allow people to self-reflect and share aspects of their daily life; this process can also trigger participants to question their choices and everyday behaviours (Grinter & Eldridge, 2001).  The content of the Mobile Diary packs and the nature of the questions included can provoke new realisations and possibilities. For example, our self-reporting studies into sustainability in everyday life resulted in participants questioning personal behaviours and making changes in their lifestyles^[i]. For one participant, a discussion about sustainability with flatmates led to the installation of a composting unit in the household.

The interventionist nature of the method can be more intentionally activated through the inclusion of specific activities and questions within the packs. For example, in one study into sustainability we included sustainability challenges – new lifestyle habits -  that participants were asked to try and document throughout a week. These activities were particularly provocative at revealing emotional and infrastructural barriers to behaviour change.

And then…?

As the image below suggests, material generated from Mobile Diaries can be used in numerous ways. Mobile Diaries externalise aspects of people’s everyday lives through visual, tangible artefacts. These become shared resources that help us to understand current practices, provide a spring-board for ideation and allow us to envision how any future design might be taken up within the existing ecology of the participants life. For designers, the visual nature of the material allows for more active interpretation in ways not possible with written research reports.  For participants, the process of doing the Mobile Diaries means they are better equipped to reflect on and analyse their own practices, during follow up interviews and workshops (similar to primer tools), becoming active interpreters of the material and what it might mean for future designs.

The material generated through Mobile Diaries is not something to be reduced down into a traditional written report. The raw form of the material and the subjective picture it provides of the participants’ lives and world-view are essential to its immersive quality and its value for fostering empathy and connection with participants (Mattelmäki, 2005; Mattelmäki & Battarbee, 2002) ^[ii].

However, this does need to be balanced with normative business expectations of a ‘research outcome’, and the need to synthesise the data in a meaningful way for the client to then transmit to other stakeholders. Effective ways to share this tangible and personal material with those who were not directly involved is the subject of current research, e.g., (Sleeswijk Visser 2009). Our process includes the development of a multimedia ‘report’ that can support all the different formats of material generated. These reports introduce the participants through their own words and images, illuminate the themes that have emerged and identify some future possibilities to be considered. We have also found significant value in creating opportunities for co-interpretation of the material by clients and other designers, whilst this can be more time consuming, it is utlimately a more effective use of the material than simply ‘handing off the research’.

We have also found that the value of such methods is greater than their role as data collection activities. A personal connection is made with participants that can be of value well beyond the particular study. However, there is a tension between this and the day-to-day realities of client and agency practices and models which focus on deliverables and project phases, in between which there can be significant lags or breaks. There is still work to be done to articulate and communicate the value of such methods beyond their capacity to generate data ‘about people,’ and to embrace their ability to involve participants in a more ongoing and sustained way throughout the design process.

Mobile Diaries are a playful, immersive experience for the design team that allow us to discover something of the messy intricacies of participant’s daily life, valuable for both inspiring and grounding the design process as well as engaging directly with stakeholders. We’d love to connect with other designers employing similar approaches.

Acknowledgments

Big thanks to Chris Gaul for images & Will Evans for feedback and comments. Thanks also to Dr Toni Robertson and the @IDHuPLab at UTS, Digital Eskimo, Zumio, our clients & participants N.B Examples come from specific studies in which permission was granted for publication for the purposes of research. Our approach has been heavily informed by the ongoing research into generative methods inspired by Cultural Probes (Gaver, et al., 1999) and the work of Liz Sanders (www.maketools.com) as well as other research into self-reporting studies such as (Hulkko, Mattelmäki, Virtanen, & Keinonen, 2004; Masten & Plowman, 2003; Palen & Salzman, 2002).

References

Buur, J., Binder, T., & Brandt, E. (2000). Taking Video Beyond ‘Hard Data’ in User Centred Design. Participatory Design Conference PDC2000, New York, NY, USA.
Gaver, B., Dunne, T., & Pacenti, E. (1999). Design: Cultural Probes. Interactions, 21-29.
Grinter, R., & Eldridge, M. (2001). y do tngrs luv 2 txt msg. ECSCW, Amsterdam.
Hulkko, S., Mattelmäki, T., Virtanen, K., & Keinonen, T. (2004). Mobile Probes. NordiCHI 04, Tampere, Finland.
Larson, R., & Csikszentmihalyi, M. (1983). The Experience Sampling Method. In H. Reis (Ed.), Naturalistic approaches to studying social interaction: New Directions for Methodology of Social and Behavioral Science: Jossey-Bass, San Francisco.
Masten, D., & Plowman, T. (2003). Digital Ethnography: The next wave in understanding the consumer experience. Design Management Journal, 14(2), 75-81.
Mattelmäki, T. (2005). Applying probes – from inspirational notes to collaborative insights. CoDesign, 1(2), 83-102.
Mattelmäki, T., & Battarbee, K. (2002). Empathy Probes Paper presented at the PDC 2002, Malmö, Sweden.
Palen, L., & Salzman, M. (2002). Voice-mail diary studies for naturalistic data capture under mobile conditions.  CSCW, Louisiana, USA.
Sleeswijk Visser, F. (2009). Bringing the everyday life of people into design (PhD Thesis), Technische Universiteit Delft, Delft.
Zimmerman, J., Forlizzi, J., & Evenson, S. (2004)  “Taxonomy for Extracting Design Knowledge from Research Conducted During Design Cases.” Futureground 04, Melbourne, Australia.

Additional examples of other remote self-reporting techniques & studies & mobile/online tools

Digital Ethnography (pdf)

Digital Diaries

Digital Cultural Probes

Sticky Research (Video Diaries)

Revelation

7daysinmylife.com

[i] There is no way for us to tell how permanent these changes were, we can only be sure that particular practices were bought to people’s attention, and steps towards change were made.
[ii] The original Cultural Probes (Gaver 1999) were not designed to gather specific information, but rather to be a source of inspiration and empathy. Rather than being codified, transformed or translated into a report, probe material was designed to stand on its own as a rich visual resource for designers.

Transformation is the act of taking a set of values from a dataset, processing them in some way (depending on the aims of the research) and arriving at a new set of values with the goal of revealing some aspect of the data from a new perspective.

(This article is the fourth part in the Deconstructing Analysis Techniques series.)

This technique is characterised by the fact that the values are changed; that someone looking at the new values will be unable to work backwards to the original values; and that for each original data point there is a single, new data point.

In mathematical parlance, (and you can skip this part if you like) the difference between a manipulation technique and a transformation technique is that manipulated data sets are congruent with the original, whereas transformed data only maintains cardinality (i.e. the same number of elements).

So, what does that all mean? We’re talk here about analysis methods like:

• Scaling – taking one set of data and massaging them to fit a distribution or ‘shape’ of values.
• Moving averages – taking a number of consecutive values and averaging them as way of ‘smoothing’ the last value in the series.
• Weighted averages – calculate an average value where more importance – ‘weight’ – is given to some values.
• Weighted indexes – calculate an indexed score (against a baseline) where more importance – ‘weight’ is given to some values.
• Seasonal adjustments – an adjustment made to a data point to account for cyclical peaks and troughs to highlight the ‘real’ shift
• Differences – a method of looking at the changes between one value and the next.

Now, initially, most of these methods may feel pretty technical, quantitative and removed from standard design research analysis. However, they form a powerful collection of analysis methods that will better equip you in undertaking design research. They also represent fairly low-level mathematical/quantitative methods and are available in a standard spreadsheet program. More importantly, used properly, these methods – and transformation techniques generally – open up new avenues for understanding the people who will use the services and products we design.

Used properly, these methods – and transformation techniques generally – open up new avenues for understanding the people who will use the services and products we design.”

Scaling

In “Deconstructing Analysis Techniques” we used the example of fitting test scores to a pre-determined probability distribution – scaling – as the example for Transformation techniques.

When we measure a population characteristic – such as height, or a test score – we create a sample set of data for that characteristic (unless we are measuring the entire population). There are times when the raw distribution (the frequency of occurrence for each value in our data) of results is not what we’re after. We may wish to compare the shape and attributes of two separate samples – two groups of test participants, for example – and so we transform the two sets of data so that they share a common mean (the average value for the data set).

Usually this is done to bring both sets of data to what is known as a ‘normalized’ distribution with a mean of 0. Of course, in our test/exam result example, we want to adjust the scores so that the class as a whole receives a pre-determined number of A, B, C, D & F. What we’re doing here is to adjust the overall shape of the data. (In these cases a plot of the raw data will look different to the scaled data.) When graphed the scaled data will look roughly bell-shaped, with the middle – or ‘hump’ – representing average performance, and the two thin tails representing high-performance (at the top end) and failure (at the bottom end).

Moving Averages

A moving average is used to smooth out day-to-day fluctuations with time series data. It is, literally, the average of the previous x days’ worth of data. A good example would be the number of page views received by a site. Each day the data will jump up and down, creating a sense of “noise” that makes analysis difficult, and, when a small number of observations are looked at in isolation, can create a false impression. A moving average is useful in time-series or longitudinal studies where we measure the value of a characteristic for a single object (person, server, site etc) over time.

One rather well-publicised and important example of this is the series of global temperature readings that have been used by both sides of the climate change debate. Skeptics of global warming point to a recent period of observations (2002 – 2007)   which show a decline in global average temperatures. When the same data is looked at using a moving average, smoothing out the peaks and troughs, a clear upward movement is seen.

The choice of time period to use when calculating a moving average is based on the specific circumstances of the data. However, common sense is usually all that’s required. For example, when looking at Web traffic, a moving average calculated over 7 days is sufficient to counter spikes that occur during a given week. You might also calculate a moving average over a month if fluctuations occur over a longer cycle.

Weighted Average

Weighted averages aim to address one of the criticisms of a moving average – and other types of averages – that being all values in the average are treated equally. It is often the case that one observation is more significant or important that another.

Let’s say for example we’re measuring the time to complete a task in a user evaluation session. We have representatives from each of our personas (or other audience segments): 2 primary personas, 3 secondary personas, and one tertiary persona. In this case, the performance of the two primary persona representatives is far more significant than that of the tertiary participant.

When we calculate the mean time-to-complete value, we can weight the results so as to reflect the relative importance of each participant. We may assign (and the exact values will vary for you) a weighting as follows:
Primary: multiply by 9
Secondary: multiply by 3
Tertiary: no multiplier

What we’re essentially saying is that our secondary personas are three times more important than our tertiary persona; and that our primary persona are three times more important than our secondary. We could just as easily use a factor of 2 (instead of 3) leading to values of 4, 2 & 1 in the example above; what matters is that we use weighted averages to adjust the dataset to account for the relative importance of some measurable data set by some exogenous variable.

Weighted Index

An indexed value is one measured in terms of some baseline figure. The aim is to convey movement around a starting point when there is no way to specify a zero.

An example of an index might be a satisfaction score. Since satisfaction a largely subjective measure, there is no way to define a zero point. Instead we typically measure a ‘pre’ figure and map that over time. Common values for an index are zero and 100. The choice is arbitrary and is typically chosen for clarity in communication.

Indexes are often calculated as an aggregate of a number of measurements. But it is also the case that we sometimes need to treat the data we receive from one group as being more important than another. This is where a weighted index comes in handy. A weighted index  – like our weighted average – treats different values as more or less important.

So, if it is common practice to design a product or service to better meet the needs of our primary audience segments; it also makes sense for our satisfaction index to put more stock in the satisfaction of our primary segments. We do this by applying a weighting (some multiplier) to each piece of data collected based on its relative importance.

We could easily do the same with responses to a question like “Would you recommend this service to a friend?”

This technique provides us a with a convenient way to build positive bias – towards the needs of our important audience segments – directly into our research methods.

Seasonal Adjustments

Some of the things we observe in design research are subject to cyclical variations. We may not, however, want to include a change in our data due to “seasonal” fluctuations, instead wanting to identify “real” changes (in frequency of use, for example).

In order to look at the real changes in our observed data we need to account for the seasonal variability first.

A familiar example might be to look at the number of page views or unique visits received by a site. We might see a big lift in traffic between Sunday & Monday; and a big drop between Friday & Saturday. In order to tell whether an observed drop in traffic on some Saturday is “normal”, we need to look at the regular pattern of changes and “adjust” the Saturday figure.

One way to do this is to calculate the average drop in traffic over time (between Friday & Saturday) and then apply this to the current observation for Friday. This as a predictor or estimator for the current Saturday, which we can then compare against the actual observed data. The average difference acts as our seasonal adjustment.

The Adaptive Path Aurora concept uses a scenario where a farmer shows that their farm will still have rain, using seasonal adjustment. See video

Differences

There are times when what we’re interested in knowing is not the raw value of an observation but the change between one observation and the next.

The calculation (transformation) is simple: for each pair of observations, subtract one from the other. Of more interest is why we would want to know such a thing.

Consider a test of a new design in which we test first the time to complete a task with the current design; and then the same task with a new design. Across all participants in the test the raw observations (i.e. time to complete) is far less interesting than the change in that time as a result of the new design. (Note that we may wish to express that change as a percentage rather than a raw value.)

We can use the same technique to highlight the variability of some observation over time. For example, we may be tracking the number of connections or ‘friends’ a person has in some social network to understand the relationship between the current number of connections and the rate at which new connection requests come in. To identify the number of new connection we simply calculate the difference between successive observations.

Summary

Although primarily applied to quantitative data, transformation techniques are useful in a wide range of design research activities beyond the quantitative.

Transformation of our research data can act as a way of reducing noise and bringing into sharp relief characteristics of the underlying user behaviour. The act of transforming removes us from the raw, original data, but in doing so we can gain the opportunity to uncover meaningful insights hidden from us otherwise.