Good design is hard to do. The very nature of human centred design is confronting, challenging and often uncomfortable. Every project builds up a collection of challenges along the way, which can pose significant risk to the project’s success, and if we don’t tackle them head on they can be detrimental for everyone involved. How can  we share and learn from each other’s challenges?

At Neoteny, we refer to the collection of challenges in a project as its ‘challenge pile’, given they’re exactly that; a pile of issues, constraints and problems. We keep track of the challenge piles using walls in our studio for each project. Some are collections of post it notes, others are photographs, drawings, diagrams, scribbles or hand written notes. Each week as part of our work in-progress meeting (team jam), we take stock of each project’s challenge pile.

We ask ourselves the following for every challenge:

• How did this challenge come about?
• Briefly establish the current reality, including:
  • What has it cost the project? Not necessarily in financial terms, what has been the cost to our momentum, resources, client expectations etc.
  • What is the potential impact? In what areas?
  • Could it have been avoided? How or why not?
• How have we handed it thus far? As a group, explore options for how we handle it moving forward.
• Agree on proposed solutions or new approaches and secure buy-in from everyone involved.
• We’ve found that this structure helps us stay out of the drama whilst understanding the drivers for each challenge, and then focus on solutions. This makes the process much more collaborative and productive, we aren’t sitting at our desks sweating over something we could probably work through together in a few minutes.

We recently had a team jam, and here’s what came out of our challenge pile review:

1. Customer needs and business requirements collide.

This project is in its early stages. The client came to us with a new product that they wanted to develop, the first step was to research the product feasibility and desirability in the market.After conducting research aimed at validating the customer need for a new product, we found that what the customer needed and what the client planned to launch, were two very different things.

We’re currently in discussions with the client to try to shift the project objectives and focus, to meet real customer needs. As a group we decided not to proceed to stage two unless we could get their buy in on a revised approach.

Question for readers: What would you do in this situation?

2. Budget streams are unclear for future phases of work.

We’ve been involved in projects in the past that have unclear funding streams for future work. This is especially common in large corporates, where steering committees assign funds based on a comprehensive business case analyses including return on investment predictions. These can’t necessarily be defined without first doing some work. The problem with this structure is that you have a team of stakeholders that can only see as far as the next steering committee meeting. This makes a design project with a strategic foundation i.e.  something that’s designed with the whole in mind, very difficult to manage.

This particular case was flagged early because we’ve seen the warning signs before. The signs included hearing things like:

• “If we build this…”, highlights the fact that the stakeholder doesn’t believe this project will make it to implementation.
• “We need to show results by June…”, if you ask why, you’ll probably hear something like “that’s our next steering committee check point”.
• “We won’t be able to build that”, if you ask why, you’ll probably hear something like “because the next budget release won’t be anywhere near that much”.

In the past, this issue has created a divide between the client or project stakeholder group and the design team. Whilst the stakeholder group is focused on securing the next round of funding to ensure that this phase can move to implementation, the design team is focused on exploring and exposing every possible opportunity for solving the design problem.

We’re currently working with senior management to ensure we have their buy-in throughout this project. In our experience, we’ve found that if the person signing the cheques is on board with the approach, the whole stakeholder group is much more relaxed and inclined to get their hands dirty in design.

Q: Have you experienced this before, and if so, how did you get around it?

3. Stakeholder groups have varying ideas of the project objectives

Have you ever been in a project meeting and realised that the client team doesn’t agree on the project’s objectives? This is an awful moment for a designer. It’s the moment when you move from designer to mediator. Playing mediator with your clients is generally not a lot of fun and not how you want to be spending your energy.

The design team typically work with clients to reach a shared set of project objectives. If you find yourself in a situation where you think this has happened but it isn’t the case, then it needs to be dealt with immediately. This agreement needs to be made before design work can start. Of course, these objectives may shift and be adjusted as part of the design process, but the aim is for adjustments to be made as a whole, not as a fragmented set of perspectives from different stakeholders.

We’re currently experiencing this on a scoping project we’re working on. It came about in a workshop, where up until that point, the team seemed aligned. We handled it by stressing the need for a shared project vision and refusing to move forward without one. We managed to facilitate developing a shared set of objectives, prioritising them and we’re currently working with the client to ensure that every stakeholder is in agreement on the vision for the project.

Without this shared vision, we put the success of the project at risk because no one is clear on what success will look like. We’re currently working through ways to communicate this in a more explicit way to our clients before we start on their projects.

Q: Perhaps it’s about signing off on the project vision, would that make people more accountable?

4. Mystery stakeholder stomps on the project.

Does this scene sound familiar? The design team is working away, the client is happy and excited, they’re getting involved and spending time designing with us. Then BAM! Along comes the mystery stakeholder who has significant influence, but just “doesn’t like blue”. In most cases, the mystery stakeholder is a fairly senior member of the client team who hasn’t been along for the ride and is looking at the design solution without any understanding of the brief, the agreed approach, the challenges or the project’s constraints. This situation can be crippling. Challenges like this can impact resources, motivation, relationships, momentum, time and budget. You could argue that it’s the design team’s fault for not ensuring that all stakeholders were engaged, the project owner’s fault for not engaging the full spectrum of players, or the mystery stakeholder’s fault for stepping in with the ‘I’m gonna leave my mark on this project regardless of how you got here’ kind of attitude.

We’ve started to enforce what we call a stakeholder roll call. At the start of every project, and within our terms and conditions we gather a list of stakeholders, their roles and responsibilities and have the project owner sign off on this list. The full list of stakeholders are required to sign off on all milestones and agreed deliverables.

We acknowledge that the stakeholders may change, but the terms allow for this situation and protect the progress we would have made in the project up to that point. The success of this approach remains to be seen, though what it does achieve is a level of accountability agreed up front for the potential impact of those ‘stomping moments’.

Q: How do you protect your projects from random stakeholder stomping? How have you dealt with this situation in the past?

Where To From Here?

As you’d expect, there’s a ‘magical box’ of learnings and insights created by each challenge pile. It’s what we choose to do with the magic that makes the obstacles and the heartache worthwhile. I’m sure we’ll learn a hell of a lot more as our company matures, but here are some of the more salient ones we’d like to share with you:

• There’s not always something ‘to do’, there’s something ‘to know’. There are situations we can’t ‘solve’ in the context of the project we’re working on. But being aware of the specific challenges and carefully managing expectations accordingly can be a very effective approach, one which better supports our potential success.
• As a company (and perhaps as an industry) let’s be more reflective. That doesn’t mean we have to wade into the drama or analyse it ad nauseam, but we do need to nip things in the bud, be honest with ourselves and the team, be open about the potential impact that the shifts might have, and involve everyone.
• Getting the players involved as the challenges arise. Rather than keeping our ‘dark passengers’ under our hat, and suffering in relative silence, all with a smile on our face, let’s face the challenges together! Clients and project stakeholders are often quite pleased when you invite them to be part of the solution. Any shifts to the project approach are also much more likely to fly if we’ve got buy-in from everyone involved.
• Sharing the good the bad and the ugly with our peers. Let’s foster a culture where we share both our triumphs and our failures, rather than keeping the latter closely guarded. As a collective mind, I’m sure we can come up with some inspired, insightful ways to circumvent and also completely avoid some of these challenges.

We believe that we can get better at this thing called ‘design’ if, as an industry, we can make the most of lessons we learn from these challenges. After all, they enable us to be more resourceful, they give us an opportunity to be more creative, to build stronger teams and deeper relationships.

So, what do you think?

Michelle is one of the speakers at UX Australia 2010, taking place 25-27 August 2010 in Melbourne, Australia. The conference has sold out, but workshops are still available, or you can go on the waiting list. See the UX Australia site for details.

“To design is to have a ‘project’. Getting the design process moving is to expose and transform this ‘project’ in a conversation with those that it might eventually affect” (Buur, Binder, & Brandt, 2000).

In the early stages of design, rather than evaluate or validate specific user requirements or priorities, we are interested in exploring possibilities. As the opening quote suggests, we seek to engage with the various stakeholders the design project may eventually effect and gain an understanding of the unique design situation from their perspective. In Zimmerman et al.’s  (2004) framework for discovering and extracting knowledge during the design process, this is known as the Discovery phase of design. In this article we introduce Mobile Diaries as a field work method that can be utilised in the early stages of design to immerse into people’s everyday life.

This exploratory approach to self-reporting allows participants  to create and share a rich picture of their world, be they grandmothers, bankers, students, young parents or employees. In this article we describe Mobile Diaries, and provide examples of the kinds experiences they can enable.

A little background on self-reporting

In self-reporting, research participants are responsible for the data collection, allowing for the gathering of contextual data over-time and in situ, without the physical presence of researchers. Self-reporting can provide access into the private, personal and mobile aspects of people’s lives that are often difficult, or impossible, to access through traditional methods such as observation or interviews. The sustained personal reflection inherent in self-reporting makes available aspects that would otherwise remain tacit. So much of our lives are routinised and automatic, it is not until we are asked to document or consider certain activities that we are able to identify key junctures in our own understanding of a topic or a behaviour.

Self-reporting studies can take many different forms and the degree of formal structure is one of the things that differentiates approaches and determines the type of material collected. For example in the Electronic Sampling Method approach known as ESM (Larson & Csikszentmihalyi, 1983) or beeper studies, the participant is directed to systematically log specific things at specific times. In more open-ended approaches (such as cultural probes (Gaver, Dunne, & Pacenti, 1999) or visual diaries) data collection is only semi-structured around a particular topic. In this case participants are treated as active contributors and interpreters in the design process and select what, how and when to report. This encourages more playful and creative representations, important to an explorative and collaborative approach.

Over the last 10 years digital, online and mobile technologies have been incorporated into self-reporting methods in a range of ways (see end of article for some examples of other studies and platforms). These everyday tools can be easily integrated into people’s daily lives and support the generation of a range of different media forms such as video, images, text and audio.

Mobile Diaries

Mobile Diaries are a hybrid method that incorporate many of the creative and playful aspects of probes and emphasise the daily reflection of visual diaries. A range of different analog and digital technologies are used that allow participants to share and reflect on various dimensions of their day-to-day life.

A typical study

The exact design of the study (as always) is dependent on a number of factors such as:

• A definition of the problem space;
• The goals and objectives of the particular project;
• The theme of the study (e.g is it a personal project or focused on the workplace);
• Budget (how many people can we recruit and how, what kind of incentives might be required);
• The profile of the participants (e.g teenagers, adults or whole families); and
• Their current technology knowledge/competence and use (e.g how they might respond to the technology involved, how open are they to using new technologies).

Generally studies run from 1-3 weeks with between 1-10 participants. Topics explored depend on the study but could include, for example: sustainability in your everyday life; the role of mobile technologies in your life; or a ‘behind the scenes’ look at your job. Participants receive a ‘Mobile Diary Pack’ with various tools and instructions which direct the data collection around the particular design topic.

Mobile Diary Tools

A number of custom platforms have been developed to support online diaries, however, to date we have preferred to configure Mobile Diaries from existing platforms such as WordPress and Tumblr as this gives us greater flexibility over format, functionality and cost. Below we show the packs from a typical study (the list of available tools is growing and changing all the time, here we show typical ones we have used in the past).

During the study

Over the period of the study participants create collages, mind maps, videos and blog messages and send in mobile reports which appear on the blog. They also receive prompts, questions and reminders via the mobile phone and the blog.  The conversation is bi-directional: as we are receiving reports we are also responding with new questions or digging deeper into particular areas, and potentially redirecting the focus of the study as a result.

What do we see and learn?

The output of Mobile Diaries is a particularly provocative, experiential and sensorial insight into participant’s lives. As one of our clients described it “[we were able to] hear in people’s own words the challenges and learn about the context of sustainability in their day-to-day lives”. We share some examples below of the kinds of material generated and shared through this approach.

Life as it happens

Using their phone, participants capture images, text and audio and send this to the blog throughout the day. These reports give us a deeper appreciation of the activities that make up people’s daily lives and we are able to ‘see for ourselves’ actual examples and instances of things that might otherwise be anecdotal. Through these reports we can track events, locations, and a sense of participant’s emotions across the days and weeks. Over time, daily rhythms and habits emerge. The (near) real-time reporting increases the sense of immersion in people’s lives as we experience the activities ‘as they happen’ (Masten & Plowman, 2003).  This is complemented by more reflective accounts at the end of the day via the blog or with the video camera.

Personal Spaces and intimate stories

The use of video encourages in-depth descriptive accounts of events and surroundings from the participant’s perspective. The stills above are from one participant’s tour of their apartment building, which focused on areas relating to sustainability in the home. By giving participants video cameras and asking them to take us on a tour of their home we are able to explore and wander with the participant. This reveals some evocative spaces otherwise inaccessible to a design researcher; in addition to the explicit content being shared the video also conveys emotion and expression.

In another study focused on teenagers and their relationship to technology one participant gave us a tour of his shed, playing instruments as he told stories about the importance of this particular space to him.  These personal stories bring us closer to the participant’s world creating a sense of intimacy and proximity to the participants which is difficult to replicate in a one on one interview, discussion group or even during participant observations.

Inner thoughts and feelings, moments and metaphors

In addition to descriptions of external events and activities, Mobile Diary reports also capture emotions, feelings and inner thoughts. The examples above show emotional reactions and descriptions of personal feelings at particular moments in time. In some, the participant’s have used objects to serve as metaphors or symbols for representing emotions or ‘states’. This allows the participant to share inner thoughts and feelings that might have otherwise remained hidden. The open, honest and personal nature of these reports fosters empathy, often describing experiences we can relate to. That some are delivered in (near) real-time further increases the sense of connection; in that moment, we knew something of what the participant was thinking and feeling.

Action & Transformation

“Not only did it help us, the impact on participants and their way of thinking about sustainability was really interesting… the project really opened their eyes to their own patterns and behaviour and sparked some changes and definitely increased awareness” (feedback from client).

The process of self-reporting is an intervention designed to allow people to self-reflect and share aspects of their daily life; this process can also trigger participants to question their choices and everyday behaviours (Grinter & Eldridge, 2001).  The content of the Mobile Diary packs and the nature of the questions included can provoke new realisations and possibilities. For example, our self-reporting studies into sustainability in everyday life resulted in participants questioning personal behaviours and making changes in their lifestyles^[i]. For one participant, a discussion about sustainability with flatmates led to the installation of a composting unit in the household.

The interventionist nature of the method can be more intentionally activated through the inclusion of specific activities and questions within the packs. For example, in one study into sustainability we included sustainability challenges – new lifestyle habits -  that participants were asked to try and document throughout a week. These activities were particularly provocative at revealing emotional and infrastructural barriers to behaviour change.

And then…?

As the image below suggests, material generated from Mobile Diaries can be used in numerous ways. Mobile Diaries externalise aspects of people’s everyday lives through visual, tangible artefacts. These become shared resources that help us to understand current practices, provide a spring-board for ideation and allow us to envision how any future design might be taken up within the existing ecology of the participants life. For designers, the visual nature of the material allows for more active interpretation in ways not possible with written research reports.  For participants, the process of doing the Mobile Diaries means they are better equipped to reflect on and analyse their own practices, during follow up interviews and workshops (similar to primer tools), becoming active interpreters of the material and what it might mean for future designs.

The material generated through Mobile Diaries is not something to be reduced down into a traditional written report. The raw form of the material and the subjective picture it provides of the participants’ lives and world-view are essential to its immersive quality and its value for fostering empathy and connection with participants (Mattelmäki, 2005; Mattelmäki & Battarbee, 2002) ^[ii].

However, this does need to be balanced with normative business expectations of a ‘research outcome’, and the need to synthesise the data in a meaningful way for the client to then transmit to other stakeholders. Effective ways to share this tangible and personal material with those who were not directly involved is the subject of current research, e.g., (Sleeswijk Visser 2009). Our process includes the development of a multimedia ‘report’ that can support all the different formats of material generated. These reports introduce the participants through their own words and images, illuminate the themes that have emerged and identify some future possibilities to be considered. We have also found significant value in creating opportunities for co-interpretation of the material by clients and other designers, whilst this can be more time consuming, it is utlimately a more effective use of the material than simply ‘handing off the research’.

We have also found that the value of such methods is greater than their role as data collection activities. A personal connection is made with participants that can be of value well beyond the particular study. However, there is a tension between this and the day-to-day realities of client and agency practices and models which focus on deliverables and project phases, in between which there can be significant lags or breaks. There is still work to be done to articulate and communicate the value of such methods beyond their capacity to generate data ‘about people,’ and to embrace their ability to involve participants in a more ongoing and sustained way throughout the design process.

Mobile Diaries are a playful, immersive experience for the design team that allow us to discover something of the messy intricacies of participant’s daily life, valuable for both inspiring and grounding the design process as well as engaging directly with stakeholders. We’d love to connect with other designers employing similar approaches.

Acknowledgments

Big thanks to Chris Gaul for images & Will Evans for feedback and comments. Thanks also to Dr Toni Robertson and the @IDHuPLab at UTS, Digital Eskimo, Zumio, our clients & participants N.B Examples come from specific studies in which permission was granted for publication for the purposes of research. Our approach has been heavily informed by the ongoing research into generative methods inspired by Cultural Probes (Gaver, et al., 1999) and the work of Liz Sanders (www.maketools.com) as well as other research into self-reporting studies such as (Hulkko, Mattelmäki, Virtanen, & Keinonen, 2004; Masten & Plowman, 2003; Palen & Salzman, 2002).

References

Buur, J., Binder, T., & Brandt, E. (2000). Taking Video Beyond ‘Hard Data’ in User Centred Design. Participatory Design Conference PDC2000, New York, NY, USA.
Gaver, B., Dunne, T., & Pacenti, E. (1999). Design: Cultural Probes. Interactions, 21-29.
Grinter, R., & Eldridge, M. (2001). y do tngrs luv 2 txt msg. ECSCW, Amsterdam.
Hulkko, S., Mattelmäki, T., Virtanen, K., & Keinonen, T. (2004). Mobile Probes. NordiCHI 04, Tampere, Finland.
Larson, R., & Csikszentmihalyi, M. (1983). The Experience Sampling Method. In H. Reis (Ed.), Naturalistic approaches to studying social interaction: New Directions for Methodology of Social and Behavioral Science: Jossey-Bass, San Francisco.
Masten, D., & Plowman, T. (2003). Digital Ethnography: The next wave in understanding the consumer experience. Design Management Journal, 14(2), 75-81.
Mattelmäki, T. (2005). Applying probes – from inspirational notes to collaborative insights. CoDesign, 1(2), 83-102.
Mattelmäki, T., & Battarbee, K. (2002). Empathy Probes Paper presented at the PDC 2002, Malmö, Sweden.
Palen, L., & Salzman, M. (2002). Voice-mail diary studies for naturalistic data capture under mobile conditions.  CSCW, Louisiana, USA.
Sleeswijk Visser, F. (2009). Bringing the everyday life of people into design (PhD Thesis), Technische Universiteit Delft, Delft.
Zimmerman, J., Forlizzi, J., & Evenson, S. (2004)  “Taxonomy for Extracting Design Knowledge from Research Conducted During Design Cases.” Futureground 04, Melbourne, Australia.

Additional examples of other remote self-reporting techniques & studies & mobile/online tools

Digital Ethnography (pdf)

Digital Diaries

Digital Cultural Probes

Sticky Research (Video Diaries)

Revelation

7daysinmylife.com

[i] There is no way for us to tell how permanent these changes were, we can only be sure that particular practices were bought to people’s attention, and steps towards change were made.
[ii] The original Cultural Probes (Gaver 1999) were not designed to gather specific information, but rather to be a source of inspiration and empathy. Rather than being codified, transformed or translated into a report, probe material was designed to stand on its own as a rich visual resource for designers.

Style guides are a great way to ensure user experience consistency when developing an application and a way to communicate user experience standards across an organization. They can be application specific, platform specific, and may encompass enterprise-wide standards. A style guide can help make the development of user interfaces more efficient and help ensure good user interface design practices.

Types of Style Guides

Style guides for applications usually contain specific instructions on how to design and develop an application’s UI. In some instances, code snippets may also be provided to simplify development of the application.

Example of an application style guide specifying a web page template grid

Enterprise-wide style guides may include standards specific to an organization. These style guides may overlap with company branding style guides that are often defined by marketing departments. They can outline a variety of company-wide items such as standard colors, typography, logos and language.

Example of branding color definitions found in a corporate style guide

Platform specific guidelines are often tailored to a specific platform, such as desktop, web, or mobile. These style guides often give particular guidance on how to design for that platform, such as control and content guidelines.

Example of a link control definitions in a website style guide

The style guide you choose to create can be any combination of these three types. It’s up to you to figure out what makes the most sense for you.

Style Guides Are Not UI Specifications

Style guides are different from user interface specifications:

• A specification document details the functionality of a UI design for developers building an application. It is usually more descriptive and is often accompanied by wireframes that act as blueprints for the design. In contrast, a style guide is often a general outline of the elements of a UI design.
• Style guides have a longer shelf-life than specifications documents that are often tied to a project life-cycle. When an application is first created, some elements of the initial specification document might turn into the application style guide for long-term reference.
• Elements of a style guide may be referred to from a specification. For example, the functionality of a web application enhancement would be captured in a specifications document; but the operation of standard UI controls found throughout the website would be outlined in the website style guide and referred to by the specifications document.

How to be Successful

Over the years, I have had a chance to create a variety of style guides. The format and purpose of these style guides were variable and were suited to the task at hand: such as details of the grid layout, colors and typography used in a website and a general guidebook providing guidance on the use of website user interface controls and how to write web-based content. From these experiences, I’ve learned a lot about what makes a style guide a success and I’d like to share some of these tips with you.

1. Keep the audience in mind

Style guides can be written for numerous audiences (e.g. other user experience practitioners, developers, graphic designers, business analysts, etc.) and the content should be structured to match the audience. Graphic designers would benefit from knowing the colors used in a website elements and programmers may desire knowing the code used to create a control.

2. Plan for success

When planning a style guide, seriously consider what would make your style guide successful in your organization. Would it be ensuring your company understands how to better design usable applications? Or would it be ensuring the large-scale website you are creating has a consistent user experience? Or would it be something else?

3. Keep it alive

Documents produced in traditional document formats can become stale and quickly become outdated. Successful style guides are produced in a manner that supports easy maintenance and supports a living document.

4. Define a review process

Create a process that supports modification and review of the style guide to actively ensure style guide maintenance and buy-in. You may want to have a person or group of people responsible for periodically updating a style guide.

5. Think of the platform differences

Style guides can be platform specific or neutral. Design guidelines can be different depending on the platform (i.e. Windows vs. Mac, iPhone vs. Blackberry). Consider how you want to support communicating any platform differences when creating your style guide.

6. Socialize the document in your organization

The use of your style guide should be communicated throughout all levels of your organization to ensure everyone knows the existence of the guide, understands how to use the guide, and actively works to use and maintain the guide. The more people about your style guide, the more successful you and your style guide will be.

7. Clearly define mandatory and flexible standards

User interface design is part art and part science and user interface paradigms shift quickly. Ensure that your style guides support new platforms and creative ideas by specifying what standards are mandatory and what are flexible. For example, you may want to ensure certain usability rules are strictly adhered to throughout your applications (i.e. “Sans serif fonts must be used for text that will be read on a screen”), but be more flexible in other areas (i.e. “Radio buttons should be used when a user is asked to select one item from a list of items.”)

8. Make the style guide as scannable and searchable as possible

Style guides can be very dense and contain a lot of detailed information. Search and browsing capabilities will make it easier for people to find what they are looking for in your style guide. In addition, use as many visual examples as possible to support quick scanning of style guide elements.

9. Provide real world examples

Successful styles guides often show one or more examples from real applications for illustration. If you are writing a style guide for a specific application, use examples from that application to demonstrate your point. If you are writing a style guide for a large organization with many applications, ensure that your examples encompass all of the applications you are describing.

I hope you find these tips helpful when you are creating your own style guides. Do you have any additional tips you would like to share?

—–
Header image by nathanborror / /CC by 2.0

Transformation is the act of taking a set of values from a dataset, processing them in some way (depending on the aims of the research) and arriving at a new set of values with the goal of revealing some aspect of the data from a new perspective.

(This article is the fourth part in the Deconstructing Analysis Techniques series.)

This technique is characterised by the fact that the values are changed; that someone looking at the new values will be unable to work backwards to the original values; and that for each original data point there is a single, new data point.

In mathematical parlance, (and you can skip this part if you like) the difference between a manipulation technique and a transformation technique is that manipulated data sets are congruent with the original, whereas transformed data only maintains cardinality (i.e. the same number of elements).

So, what does that all mean? We’re talk here about analysis methods like:

• Scaling – taking one set of data and massaging them to fit a distribution or ‘shape’ of values.
• Moving averages – taking a number of consecutive values and averaging them as way of ‘smoothing’ the last value in the series.
• Weighted averages – calculate an average value where more importance – ‘weight’ – is given to some values.
• Weighted indexes – calculate an indexed score (against a baseline) where more importance – ‘weight’ is given to some values.
• Seasonal adjustments – an adjustment made to a data point to account for cyclical peaks and troughs to highlight the ‘real’ shift
• Differences – a method of looking at the changes between one value and the next.

Now, initially, most of these methods may feel pretty technical, quantitative and removed from standard design research analysis. However, they form a powerful collection of analysis methods that will better equip you in undertaking design research. They also represent fairly low-level mathematical/quantitative methods and are available in a standard spreadsheet program. More importantly, used properly, these methods – and transformation techniques generally – open up new avenues for understanding the people who will use the services and products we design.

Used properly, these methods – and transformation techniques generally – open up new avenues for understanding the people who will use the services and products we design.”

Scaling

In “Deconstructing Analysis Techniques” we used the example of fitting test scores to a pre-determined probability distribution – scaling – as the example for Transformation techniques.

When we measure a population characteristic – such as height, or a test score – we create a sample set of data for that characteristic (unless we are measuring the entire population). There are times when the raw distribution (the frequency of occurrence for each value in our data) of results is not what we’re after. We may wish to compare the shape and attributes of two separate samples – two groups of test participants, for example – and so we transform the two sets of data so that they share a common mean (the average value for the data set).

Usually this is done to bring both sets of data to what is known as a ‘normalized’ distribution with a mean of 0. Of course, in our test/exam result example, we want to adjust the scores so that the class as a whole receives a pre-determined number of A, B, C, D & F. What we’re doing here is to adjust the overall shape of the data. (In these cases a plot of the raw data will look different to the scaled data.) When graphed the scaled data will look roughly bell-shaped, with the middle – or ‘hump’ – representing average performance, and the two thin tails representing high-performance (at the top end) and failure (at the bottom end).

Moving Averages

A moving average is used to smooth out day-to-day fluctuations with time series data. It is, literally, the average of the previous x days’ worth of data. A good example would be the number of page views received by a site. Each day the data will jump up and down, creating a sense of “noise” that makes analysis difficult, and, when a small number of observations are looked at in isolation, can create a false impression. A moving average is useful in time-series or longitudinal studies where we measure the value of a characteristic for a single object (person, server, site etc) over time.

One rather well-publicised and important example of this is the series of global temperature readings that have been used by both sides of the climate change debate. Skeptics of global warming point to a recent period of observations (2002 – 2007)   which show a decline in global average temperatures. When the same data is looked at using a moving average, smoothing out the peaks and troughs, a clear upward movement is seen.

The choice of time period to use when calculating a moving average is based on the specific circumstances of the data. However, common sense is usually all that’s required. For example, when looking at Web traffic, a moving average calculated over 7 days is sufficient to counter spikes that occur during a given week. You might also calculate a moving average over a month if fluctuations occur over a longer cycle.

Weighted Average

Weighted averages aim to address one of the criticisms of a moving average – and other types of averages – that being all values in the average are treated equally. It is often the case that one observation is more significant or important that another.

Let’s say for example we’re measuring the time to complete a task in a user evaluation session. We have representatives from each of our personas (or other audience segments): 2 primary personas, 3 secondary personas, and one tertiary persona. In this case, the performance of the two primary persona representatives is far more significant than that of the tertiary participant.

When we calculate the mean time-to-complete value, we can weight the results so as to reflect the relative importance of each participant. We may assign (and the exact values will vary for you) a weighting as follows:
Primary: multiply by 9
Secondary: multiply by 3
Tertiary: no multiplier

What we’re essentially saying is that our secondary personas are three times more important than our tertiary persona; and that our primary persona are three times more important than our secondary. We could just as easily use a factor of 2 (instead of 3) leading to values of 4, 2 & 1 in the example above; what matters is that we use weighted averages to adjust the dataset to account for the relative importance of some measurable data set by some exogenous variable.

Weighted Index

An indexed value is one measured in terms of some baseline figure. The aim is to convey movement around a starting point when there is no way to specify a zero.

An example of an index might be a satisfaction score. Since satisfaction a largely subjective measure, there is no way to define a zero point. Instead we typically measure a ‘pre’ figure and map that over time. Common values for an index are zero and 100. The choice is arbitrary and is typically chosen for clarity in communication.

Indexes are often calculated as an aggregate of a number of measurements. But it is also the case that we sometimes need to treat the data we receive from one group as being more important than another. This is where a weighted index comes in handy. A weighted index  – like our weighted average – treats different values as more or less important.

So, if it is common practice to design a product or service to better meet the needs of our primary audience segments; it also makes sense for our satisfaction index to put more stock in the satisfaction of our primary segments. We do this by applying a weighting (some multiplier) to each piece of data collected based on its relative importance.

We could easily do the same with responses to a question like “Would you recommend this service to a friend?”

This technique provides us a with a convenient way to build positive bias – towards the needs of our important audience segments – directly into our research methods.

Seasonal Adjustments

Some of the things we observe in design research are subject to cyclical variations. We may not, however, want to include a change in our data due to “seasonal” fluctuations, instead wanting to identify “real” changes (in frequency of use, for example).

In order to look at the real changes in our observed data we need to account for the seasonal variability first.

A familiar example might be to look at the number of page views or unique visits received by a site. We might see a big lift in traffic between Sunday & Monday; and a big drop between Friday & Saturday. In order to tell whether an observed drop in traffic on some Saturday is “normal”, we need to look at the regular pattern of changes and “adjust” the Saturday figure.

One way to do this is to calculate the average drop in traffic over time (between Friday & Saturday) and then apply this to the current observation for Friday. This as a predictor or estimator for the current Saturday, which we can then compare against the actual observed data. The average difference acts as our seasonal adjustment.

The Adaptive Path Aurora concept uses a scenario where a farmer shows that their farm will still have rain, using seasonal adjustment. See video

Differences

There are times when what we’re interested in knowing is not the raw value of an observation but the change between one observation and the next.

The calculation (transformation) is simple: for each pair of observations, subtract one from the other. Of more interest is why we would want to know such a thing.

Consider a test of a new design in which we test first the time to complete a task with the current design; and then the same task with a new design. Across all participants in the test the raw observations (i.e. time to complete) is far less interesting than the change in that time as a result of the new design. (Note that we may wish to express that change as a percentage rather than a raw value.)

We can use the same technique to highlight the variability of some observation over time. For example, we may be tracking the number of connections or ‘friends’ a person has in some social network to understand the relationship between the current number of connections and the rate at which new connection requests come in. To identify the number of new connection we simply calculate the difference between successive observations.

Summary

Although primarily applied to quantitative data, transformation techniques are useful in a wide range of design research activities beyond the quantitative.

Transformation of our research data can act as a way of reducing noise and bringing into sharp relief characteristics of the underlying user behaviour. The act of transforming removes us from the raw, original data, but in doing so we can gain the opportunity to uncover meaningful insights hidden from us otherwise.

“We won’t be doing any user testing for this project. There’s just no budget for it, and we don’t have time.” Hear this often? We do, particularly since we work in Ireland – a small country with similarly small companies and budgets. However, we believe user research is too important to give up. So instead, we have to run tests quickly and cheaply for our clients to accept the cost – and we have to clearly show how it brings value. Because of this, we’ve developed a toolbox of quick, cheap UX research techniques.

In this article, we’ll talk about one technique known as fast prototyping, and how we effectively used it in a recent project for Vodafone Ireland. We’ll also be talking about this subject at EuroIA 09 in Denmark.

Background: Vodafone Ireland

The Vodafone Ireland webiste

Our task was to redesign the Vodafone Ireland information architecture (IA), which the company had quite simply outgrown. Traditional business areas dominated the website and left little breathing room for new business streams in which the company was investing heavily. Our challenge was to provide Vodafone’s users with a clear map of the site and its products in a quickly shifting market.
We investigated traditional IA user testing tools such as card sorting, but we felt given the complexity of the site, it would give us limited results. A prototyping solution was a much more effective use of resources.

Why Prototype?

Card sorting is great for defining natural associations among a selection of words and objects, and really helpful in deconstructing concepts into logical groupings. But we felt that it often places undue weight on the unusual cases that straddle concept groupings, and not enough on the main paths that most users pursue when navigating a site.

True, prototyping doesn’t provide the volume of statistical data you’d get from a large group of card sorters. You have to invest more time in fewer people, but the qualitative and comparative data can be a really rich resource.

At Vodafone, our focus was on efficient findability – getting users to their destination easily. So we had to test not just our product groupings, but the usability of our principal purchase flows.
A prototype combined with specific use cases or scenarios was perfect for this.

Making the Prototypes

We have a small development team of 3 full-time front-end programmers who spend their days (and often nights) writing production-quality code. For us, assigning developers’ time to prototyping is a huge expense. But for our interaction designers – who aren’t all expert coders – creating prototypes in HTML is too time-consuming (even if 37Signals say otherwise). Therefore, we needed a prototyping technique that was simple and didn’t involve a lot of coding.

While there are many methods around, (for example, Phil Frelinger of Xero champions working with Flash, others have even suggested Powerpoint or Keynote), we found that for static pages, image map prototypes worked best, and for forms integrating image map prototypes with HTML.

Static Pages – Image Map Prototypes

Most wireframing tools (Omnigraffle, Axure or Visio) allow you to assign actions to shapes, such as opening a new wireframe or displaying a layer on click. The wireframes can then be exported to HTML, with clickable areas represented by image maps.
We’ve found this is a great way to prototype static web pages and have used them to test:

• Information architecture
• Navigation
• Labeling
• Content

What’s best about them is that they’re fast. A prototype created in HTML takes about 2 days of development and 1 day of interaction design. An equivalent image map prototype, starting from existing wireframes, can be created by an interaction designer in just 3 to 5 hours.

Forms – Image Map + HTML

Unfortunately, image maps can’t be used to test forms. Forms must be filled and submitted. They must validate input. They must provide error and confirmation messages. This means form prototypes must be built in HTML.

To minimise development work, we integrate image map prototypes with forms built in HTML, CSS and JavaScript. (See the below video for how to make one).

In our projects, we use prototypes to test process flows (e.g. online purchases and applications) – from the homepage to the confirmation message. We use image maps for all static pages, and build the forms in HTML, CSS and a JavaScript framework. We then link the image maps and the application forms together, and it works seamlessly.

We can built these prototypes in just one day, with a half-day each of development and interaction design.

Testing With Prototypes

The Vodafone IA project involved 2 rounds of user testing.

• Round 1 was 2 days with a total of 12 people for just under an hour each. We tested 2 low fidelity prototypes against the existing website.
• Round 2 involved just one combined prototype for verification purposes and looked a little harder at some of the decision pages, content and design decisions that needed firming up.

We used scenarios when testing to provide depth of context and richness of content. We inserted live videos and advertisements into the prototypes, and used posters and storytelling to get the user to take on a role and reenact scenarios. It led to very effective, real life, real world conversations.

How prototypes provided context

The Vodafone site had enormous similarities between its product subsections. Mobile phones and home phones all had plans, costs and bundle offers, and could be used to access the internet. Because card sorting approached IA analysis from the bottom up, we would have had to prefix each choice with a content heading like “mobile plans” or “mobile costs” in order to provide context. Using prototyping to test from the top down, we were able to avoid this and base our analysis on the user’s contextual position once they’d clicked on a particular section.

Testing page designs and IA at once

The previous design of the site included a number of decision pages to minimise reliance on an IA. These served a purpose within the site structure, but we found they had usability issues – and analytics confirmed our suspicions.

Still, we saw these pages as an integral element of the IA itself, so making them work was a key objective. Prototyping also allowed us to explore a number of design concepts for these pages. For example, there was discussion in the project team over the best approach to display IA layouts. When we looked at other global Vodafone sites, we saw two distinct approaches:

1. A single top level IA exposed
2. A top level IA with a sub level IA exposed

Prototyping allowed us to feasibly examine both possibilities and unearth the most efficient solution.

Conclusion

In cases such as the Vodafone Ireland project, we found that fast prototyping techniques opened up possibilities for innovative, low-cost and iterative approaches that engaged users in task and scenario-based exercises during the design process.

Fast prototyping gets us close to a real-life scenario of a website, but affords us the flexibility to test different structures and designs very early on in a project – before any costly development work has begun.

For more on other cheap UX tools, see our workshop at EuroIA, “Cheap and efficient tools: how to engage users in IA design when there is no budget and there is no time“.

Acknowledgments

We would like to thank Vodafone Ireland for allowing us to feature their projects.

Everyone reading this knows what a brainstorm is (I hope). I want to review a bit about the process I’ve used to find success in brainstorms, why they are helpful, and how they can be more successful in a corporate environment. If you work in an agency, this may not be as helpful, but might offer an interesting perspective. For those in the corporate environment, I hope this will help give you new ideas and erode old beliefs.

At a basic level, if done well, running brainstorms can show leadership and faster ROI for a team. If you can help the team create and develop ideas in a short session instead of weeks sitting alone, it makes you pretty valuable asset. Brainstorming is also great for engaging the full team and solicit their ideas. By “the full team”, I don’t just mean Design or UX, it’s also development, test (or QA), Project Managers… everyone. It’s important that designers get rid of the idea that we’re the only creative people on a team. Everyone is creative, some people are simply more creative, and others show it in different ways. People think in all different ways, but a good idea is the same no matter who says it.

Win them over

After accepting the fact that everyone can be creative, you can start to collaborate with people more productively. In addition to the great ideas you can get, you’ll also win over people. Many times developers or QA aren’t asked for their ideas, so the simple act of asking can get them excited and involved. Plus they will most likely be more dedicated and work harder when they’ve got the feeling they were involved in the early proces of the product.

Many times developers or QA aren’t asked for their ideas, so the simple act of asking can get them excited and involved

Let me assure everyone reading that I’m not telling you to allow a developer or someone else to make a design decision, but to simply get their ideas. In the corporate environment, another fallacy to wear away is that the designer always has to come up with the idea. This puts a lot of pressure on designers, but if you believe good and great ideas are hidden anywhere on the team, all you need to do is find them. Once you find all the ideas, the job of a designer is to tell a great story, make it simple, usable, and culturally relevant. That process and thinking is where the real design work comes in. Good ideas can be cheap, telling a great story takes a lot of time and thinking… but let’s get back to the ideas.

Involve people who want to be involved

When you do brainstorms, don’t involve people who don’t want to be involved. I’ve run a few sessions where people didn’t want to be involved or didn’t think they could contribute and sit quietly. Since I’m not in an agency, it’s not on my shoulders to try and make everyone look good in a meeting. So when someone isn’t interested, I leave them out, but let them know as an FYI in case they change their mind. A session with one negative person can quickly take the whole session down and drain the energy in the room.

It’s also important to get people to feel inspired. On the last project I worked on, we all went to see Wall-e to kick off the project. In addition to that we reviewed sites, objects, or products we all enjoyed and thought were cool. Somewhat of a warm up exercise to get us thinking broad so we didn’t stay in a software mindset. I’ve seen IDEO, Frog, and a few other companies put toys and object on the table during the brainstorm. Whatever the team finds helpful; if it works, go for it!

On the last project I worked on, we all went to see Wall-e to kick off the project.

Types of brainstorming

There are two style of brainstorming I most often use for corporate teams. One is what I call Improv Brainstorming, and it pulls from… Improv comedy. The second is a simple round robin approach, which I’ve seen many times. For any brainstorm, these are general rules of engagement I have. Several are pulled from IDEO.

• Be visual
• Defer judgment
• Stay focused
• One person talking at a time (I’ve heard people dispute this, but it works for me)
• Defined agenda

For improv brainstorming I use these rules and processes:

• Start with a single idea
• Build on that idea
• Offshoot other ideas or just start with a new concept
• Use a moderator to help guide ideas and conversation
• Have ideas in your back pocket to restart things in case they get stalled

For round robin brainstorming I set it up in the following way:

• Everyone has N minutes to sketch a few ideas on their own
• Everyone presents their ideas
• Team votes on a few core ideas
• Another individual round for N minutes dedicated to those ideas
• Present again / repeat

Hopefully all of that will give you one idea to use in your work place. I’m simple, so if you walk away with just a single idea, that’s enough for me. Someone once told me that it’s not what you know that’s important, it’s what you don’t. If that’s true, brainstorming certainly is a way to get to the important stuff fast.

Image by Jakob Botter

Deconstruction is one of the most frequently used and fundamental analysis techniques in our toolkit. It is used as both a preparatory technique to get research data ready for use in other ways; and a powerful technique in its own right as a method of isolating, exposing, and testing assumptions deeply embedded in our mental models.

One example of deconstruction is turning an interview transcript into a series of separate comments or answers to questions. Deconstruction is often used simply to prepare data for other analytic processes such as manipulation or summarization, or even abstraction.

Note: this article builds on the first part of the series: Deconstructing Analysis Techniques published in February.

Examples of Deconstruction

• Chemical analysis – mass spectrometry: a technique for determining the elemental makeup of a substance or molecule.
• Philosophy/literary criticism: a technique of isolating and testing ideas contained within a work of philosophy or literature
• Systems analysis: identifying root causes through the identification of individual system ‘actors’ and their interactions
• Quality control: unit testing functional components of an application, requires first identifying those components (typically by recourse to the specification)
• User interviews: identifying individual concepts or ideas
• Card-sorting: working with card-pairs
• Task analysis: breaking down complex activities into individual tasks and their components.

There are a wide range of examples of the way in which deconstruction occurs, but our aim is always to reach a definite ‘atomic’ state (where the atom is defined by our research objectives). It should be noted that there are typically more things going on than merely breaking down the data. In the case of chemical analysis, level of elements or compounds are measured; in the case of a stakeholder or user interview, the individual words, phrases or ideas may be tallied, grouped, manipulated or otherwise worked with to form some new insight.

Deconstruction can – and often is – built into the design of the research. We see this in online card-sorting, for example, where data is stored from the outset as card-pairs. Survey results are another example of data where pre-deconstruction is built into the research.

Why, though, this urge to break data down into smaller and smaller pieces?

Smaller, more granular data provides for greater flexibility in the other analysis techniques we need to undertake. By separating ideas or objects out into their own data elements we can have greater control over how elements are treated and positioned with respect to other elements.

For example, splitting a Name element into separate First Name and Surname elements allows us to treat these two components independently, and ask a broader range of questions – such as: “What are the most common first names?”

Smaller, more granular data provides for greater flexibility in the other analysis techniques we need to undertake.

It requires extra effort to break data down and then to store it in more granular form. It also takes effort to request and record extra data during the research process itself. So, whatever level of data granularity we use should be for specific reasons, and to address specific research questions.

Deconstruction represents a powerful analytic technique in its own right. By isolating concepts and ideas, and exposing them to scrutiny on their own, deconstruction highlights the existence of untested assumptions and ‘sacred cows’.

In this sense, deconstruction is often used to analyze problems or situations to which we need to formulate a response. This use of deconstruction allows us to test the reality of perceived constraints: by isolating each constraint to the design, and looking at the conditions under which they may hold true, new possibilities can open up that may otherwise not have been possible or feasible.

Dangers in Deconstruction

There are dangers in deconstruction that are worth mentioning here. At the end of the day our work should lead to something substantively new. This can be difficult if we lose sight of the macro-level problem in pursuit of an understanding of the data in finer and finer detail.

Secondly, in studying the fine detail of our data we can miss seeing the patterns in our data that help drive insights and accelerate the transition to design concepts. At the same time, some patterns only become visible or apparent when we reach a level of granularity appropriate for the data.

Deconstruction can also generate noise in our data which obscures our sense-making abilities. This noise may be the result of data overload – simply having too much information to allow for processing; or it may be that small-scale, natural random variations are masking higher-level trends or patterns. In these cases, the use of summation and aggregation techniques might be an appropriate contrast to the deconstruction technique.

Deconstruction in practice

Deconstruction can often be used in very close association with other analytic techniques. For example, we may break data down into more granular form to facilitate manipulation of that data as a means of inspection or ‘eye-balling’.

In the same vein, we may manipulate or transform our data to allow us to zero in on a particular characteristic – deconstruction in the critique sense of the term.

It may help at this point to look at some examples to help illustrate the different uses of deconstruction as an analytic technique:

User Interviews
A typical interview scenario involves asking participants a series of questions (usually open-ended; sometimes based around topics rather than using a strict question set) and recording the responses. Recording may be through the use of written notes, audio recording, video taping, interviewer/observer notes; and may include a combination of the above.

To begin drawing connections and identifying themes between interviews we need to break down – or deconstruct – the interviews to the level of individual ideas or concepts, feelings, thoughts etc. The medium we use to record each of these ‘objects’ is not important: a spreadsheet might be used just as effectively as Post-It notes or index cards.

Once the data is in this more granular form we can carry out further analysis on the interviews. We may, for example, want to look at the prevalence of positive versus negative feedback.

Note, however, that the need for deconstruction is entirely dependent on the questions we are trying to answer through our research. For example, if our intent was to formulate an impression of the overall level of satisfaction for each interview subject, the deconstruction would be an entirely unnecessary task.

Diagnosing Causes
When faced with a failure in a complex system – such as the inability of users to complete a multi-step process, or the appearance of a previously unplanned-for edge case – it is typically quite difficult to diagnose the cause of the failure (in the absence of error handling designed specifically with this in mind). In order to identify the root cause of the failure we undertake a deconstruction exercise to help isolate the components of the systems.

Designing a Car: Highlighting untested assumptions
If we were to begin designing a car we might begin with a brain-storming session and list out all of the components or features that are needed. That list might include items such as “wheels”, “engine”, “fuel”, “doors”, “seats” and a whole range of others. We can now look at each of these features and ask why it’s there, and what it says about our notion of the solution.

For example, ‘fuel’ presupposes a form of combustion engine which, increasingly, may not be relevant. More importantly, ‘fuel’ highlights a range of assumptions – mostly tacit – derived from our mental model of the object ‘car’.

Once these assumptions are exposed we can begin to question their validity in the context of the problem – instead of pre-defining a solution in the statement of the problem. Such questioning, enabled through deconstruction, opens up a broader perspective on the design of a solution.

Conclusion

Deconstruction serves a dual role in our analysis work: as both a preparatory technique to get research data ready for use in other ways; and a method of isolating, exposing, and testing assumptions deeply embedded in our mental models.

The technique is not without its drawbacks: more granular data requires effort to gather and record, store, and analyze. It can also generate ‘noise’ in the data, which can obscure instead of illuminate.

Understanding the role of deconstruction in analysis can help us to better target it’s application to the solution of specific research questions.

Photos by s1mone (card sorting), andercismo (magnifying glass), smiling da vinci (interview)

Analysis is that oft-glossed over, but extremely important step in the research process that sits between observation (data gathering) and our design insights or recommendations. In many respects, analysis is crucial to realizing the value of our research since good analysis can salvage something from bad research, but the converse is not so true. This is where the literature tends to fall a little silent, jumping over the analysis techniques straight to a discussion of how best to document and communicate the findings from analysis. This article seeks to begin to redress that imbalance by breaking down the analysis black box into its major sub-techniques.

On a recent project I needed to collect and analyze the content management templates in use across a large enterprise Intranet. We were looking to inventory the diversity of templates in use; whether they existed outside or within the enterprise content management system; what changes might be made to the ‘official’ template set to reduce the overall number of templates, and to prepare for the migration of all content to a new design a few months down the track. I looked around at the literature for information architecture and Web design generally and found quite a few references to content inventories and content analysis, but nothing on analyzing templates.

I set about designing the analysis task from scratch: looking at what we wanted to get out of the analysis; and looking at what tools and techniques would most effectively allow us to get there. In so doing, it struck me that there is very little information published about the process of analysis that would equip practitioners with a toolkit to construct their own analytical techniques. So User Experience literature and all of its component domains focuses on techniques for user research and testing, it’s surprising to realize that the coverage often skips over the process of analysis, since this is where much of the value of our research is realized.

Techniques of Analysis

We can start to pull back the curtain on analysis by looking at the techniques that go into the process:

• Deconstruction: breaking observations down into component pieces. This is the classical definition of analysis.
• Manipulation: re-sorting, rearranging and otherwise moving your research data, without fundamentally changing it. This is used both as a preparatory technique – i.e. as a precursor to some other activity – or as a means of exploring the data as an analytic tool in its own right.
• Transformation: Processing the data to arrive at some new representation of the observations. Unlike manipulation, transformation has the effect of changing the data.
• Summarization: collating similar observations together and treating them collectively. This is a standard technique in many quantitative analysis methods.
• Aggregation: closely related to summarization, this technique draws together data from multiple sources. Such collections typically represent a “higher-level” view made up from the underlying individual data sets. Aggregate data is used frequently in quantitative analysis.
• Generalization: taking specific data from our observations and creating general statements or rules.
• Abstraction: the process of stripping out the particulars – information that relates to a specific example – so that more general characteristics come to the fore.
• Synthesis: The process of drawing together concepts, ideas, objects and other qualitative data in new configurations, or to create something entirely new.

Let’s take a look at each of these techniques in detail and discuss some of the ways in which each technique can be applied.

Deconstruction

Breaking observations down into component pieces. This is the classical definition of analysis.

Breaking down research data into its component parts is a standard technique for analysis. One example of deconstruction is turning an interview transcript into a series of separate comments or answers to questions. Deconstruction is often used simply to prepare data for other analytic processes such as manipulation or summarization, or even abstraction.

The aim of deconstruction is to decouple each component so as to allow inspection of each in its own right. In other disciplines this process is used as a device for critical thinking, bypassing the potentially misleading image conveyed by the whole. In so doing deconstruction can be a powerful tool for exposing unquestioned assumptions about our users’ mental models or the business priorities of the client organization.

Looking at our template analysis example, one of our first analysis tasks was to deconstruct the templates into their components. Like most of the technique we took a very low-tech approach to the task, blocking out the individual components with a pencil. In our case, the deconstruction made easier a lot of the subsequent analysis work.It was a minor, but significant, step in the overall process.

Manipulation

Re-sorting, rearranging and otherwise moving your research data, without fundamentally changing it. This is used both as a preparatory technique or as a means of exploring the data as an analytic tool in its own right.

The ability to “play with the data” is a critical capability in analysis. We utilize this technique in many situations: searching for patterns or trends in our observations; or as another preparatory stage for further analysis. For example, sorting data in some way – alphabetic, chronological, complexity or numerical – is an a form of manipulation.

The ability to easily manipulate data is one of the key determinants for the tools we use in our analysis work. Spreadsheets are an excellent tool for manipulating data; but as we see in our template analysis task, the use of a more tangible form – such as our index cards – can be just as effective: if not more so in some cases.

When data recorded in a format that resists fluid manipulation and exploration people can stumble when moving from observation & data collection into analysis. It is important to plan this task into the research design so that it is not overlooked. You could find yourself with a costly and time-consuming data-entry process  if it is forgotten in the planning stages.

Transformation

Processing the data to arrive at some new representation of the observations. Unlike manipulation, transformation has the effect of changing the data.

Transforming research data is the process of taking our research data and turning it into something else. For example, you may recall from your schooling days the practice of “scaling” results from an assessment task (exam, essay etc) so they fit a certain distribution, so you end up with (for example) 10% A, 15% B, 25% C, 25% D etc

Another example might be to convert raw data into a logarithmic form to reduce the impact of extreme values – or to demonstrate power laws in the data.

Summarization

Collating similar observations together and treating them collectively. This is a standard technique in many quantitative analysis methods.

The goal of summarizing data is to generate an additional set of data, typically more succinct, that encapsulates the raw data in some way. This may be a short sentence that captures the essential point from several minutes of an interview transcript: “participant finds site search unwieldy, confusing and difficult to use”.

We can also summarize the data quantitatively using summary or descriptive statistics such as frequencies, means, and standard deviations. Unlike the process of abstraction, where specificity is sacrificed for the sake of clarity; or aggregation, where several data sets are “rolled up”; summarization seeks to characterize the underlying data.

Once again, spreadsheets are a very useful tool, especially when dealing with quantitative data. But they can be similarly useful when handling other data types. An equally useful medium for capturing summaries (once you have them) – particularly of qualitative data – is the PostIt or sticky note. This medium is also highly suited to manipulation and exploration of the resulting data. One advantage sticky notes have over a spreadsheet is that you can arrange and re-arrange them in two dimensions, so you can further manipulate and explore the summaries.

Index cards share many of the same advantages as sticky notes. They can be an excellent tool for capturing and working with summaries. They have the added advantage of being relatively robust and can therefore sustain a greater degree of handling.

Aggregation

Closely related to summarization, this technique draws together data from multiple sources. Such collections typically represent a “higher-level” view made up from the underlying individual data sets. Aggregate data is used frequently in quantitative analysis.

As discussed previously, aggregation is similar to, but distinct from summarization. In one respect aggregation is simply the process of bringing together data from a variety of sources and adding it together. In an analytic context it also carries with it the connotation of combining those sources together into something new.

A good example to highlight aggregation in action is the creation of a (fictional) customer satisfaction index (CSI). Our CSI will use data from:

• An annual customer survey;
• The number of product returns received; and
• The ratio of new to repeat customers.

We combine data from each of these sources and arrive at some single figure – based on some form of calculation (we’ll save the ‘how’ of that for another time). That single figure – which we can track year-to-year – is our aggregate. Unlike a summary, which characterizes a single piece of data, you can see that our aggregate is a composite value.

Generalization

Taking specific data from our observations and creating general statements or rules.

Taking the results of some specific research task and drawing general inferences about the broader population is one of the most common, but perhaps the least understood analytical technique. Generalization draws a great deal of its strength from the discipline of statistics, and the particular techniques of statistical inference.

In many respects generalization is similar to abstraction in that it reflects a move from the specific to the general or essential. It is a way of describing the common characteristics of the objects reflected in the data.

An example of generalization might be: “security is important to our users” based on an analysis of user interviews.

Abstraction

The process of stripping out the particulars – information that relates to a specific example – so that more general characteristics come to the fore.

The process of abstraction involves the progressive removal of specific data retaining just the essential information needed to communicate particular characteristics of an object. For example, “professional” is a more abstract form of “Doctor” or “Lawyer”; “graphic” is a more abstract form for “photograph”, “logo”, “illustration” or “chart”.

A wireframe is an abstract representation of a page design; the template thumbnails on our index cards are an abstract representation of the templates.

Abstract representations can be very useful because they remove a lot of visual noise from the analysis process. What we’re left with is a “high-level” depiction devoid of specific detail; highlight focused on just those elements which are relevant to the discussion.

Synthesis

The process of drawing together concepts, ideas, objects and other qualitative data in new configurations, or to create something entirely new.

Combining multiple elements together to create a new, complex ‘thing’ is what the technique of synthesis is all about. Similar in some respects to aggregation, synthesis typically deals with non-numeric data.

Synthesis is often undertaken towards the end of an analytic process as the reverse of deconstruction. So where we might begin by breaking down data into its component parts and examining them; we often end by recombining those components in new ways. Note, however, that synthesis can also form part of an exploration and is one of the fundamental tools of the trade for UX strategy work.

If deconstruction allows us to critically examine assumptions by isolating individual components, synthesis allows us to explore new configurations for the whole.

But what about…

In discussing this article with other people we identified three other techniques that we either weren’t sure belonged as analytic techniques, or we couldn’t decide if they were already covered by the techniques discussed above. We believe they’re all very important to the analysis process. They are:

• Reflection: thinking, pondering, contemplating. To the outside observer it looks a lot like staring into space, but your mind is going over and over and over all the detail of your observations, data, diagrams, and other research materials. It’s the part you can’t put a time limit on, and can make or break your subsequent work. You might call it “soaking it all in”, or “immersing myself in the data”. This technique is incredibly valuable to me in my own work and I’m not sure I’d be as effective if I didn’t include it.
• Visualization: this technique is about giving the data a visual dimension. Instead of lists of items, or rows of numbers in a spreadsheet, a chart or graph or some form of illustration. A good visualization can help expose patterns or gaps much more clearly than the raw data.
• ‘Number-crunching’: this feels like it needs to be drawn out as a separate activity from data manipulation, transformation, or summarization, but I also recognise that this level of distinction may just be peculiar to me. This refers to all of the heavy-duty quantitative analysis work like clustering analysis, or regression, calculating correlation co-efficients and the like.

Conclusion

Working with research data and observations is often treated as a black box in design literature. Designers find themselves faced with the daunting task of analysing research data, but lack clear approaches to that task. Understanding the major techniques used in analysis work can remove some of the uncertainty and provide a clear way in to the work.

There still exists a very large gap in the literature on analysis and analytic techniques, but I hope that this discussion of the major components of analysis will go some way towards filling that void. The next time you’re undertaking some analysis work, try and identify these major techniques, and see if there are any others we can add to the list.

I’d like to say a very big thank you to the people who helped clarify and refine both my thinking on this topic, and the expression of that thinking in this article: Will Evans, Livia Labate, Donna Spencer and Daniel Szuc; Christian Crumlish, Michael Leis and Kaleem Khan.

Graphics by Jeroen van Geel (and he’s pretty proud of them .