How to combine multiple research methods: Practical Triangulation

Related posts:


All research methods have their pros and cons, the problem comes when you rely on just one method. I’m often disappointed when UX and IxD practitioners describe the research they do, and it’s obviously very one dimensional. They only do surveys, for example. Or they only do usability testing at the end of the project (it’s quite alarming but this practice does continue).

This problem isn’t restricted to UX and IxD of course, our marketing brethren might do likewise, referring only to Roy Morgan for insight, for example.

Each of these techniques can be incredibly useful for giving insight into a particular aspect of what you’re studying, but relying solely on one is a big mistake.

Introducing triangulation

This is where the concept of “triangulation” comes into its own. Also known as “mixed method” research, triangulation is the act of combining several research methods to study one thing. They overlap each other somewhat, being complimentary at times, contrary at others. This has the effect of balancing each method out and giving a richer and hopefully truer account.

[Triangulation is an] attempt to map out, or explain more fully, the richness and complexity of human behavior by studying it from more than one standpoint
- Cohen and Manion

The effect is rather like the use of telescope arrays, such as that shown in the photo above, which make use of many small telescopes spread over a large area to simulate the effect of one very large telescope. Not only does the array have the power of one very large telescope but it is more nimble, more practical and has the ability to cross-check itself. Because each individual telescope’s view overlaps that of its neighbours, the accuracy of each telescope can be validated to a certain extent by the others. This last property is a key benefit of triangulation in research, and one which we’re going to explore further.

Let’s start with a real life example of triangulation. Most of my own research projects of late, have included:

  • Analytics
  • Stakeholder interviews
  • Interviews (face-to-face or phone as needs be)
  • Cultural probe (aka diary study)
  • Focus groups or workshops
  • Secondary research (including an examination of market research data)
  • Quantitative survey (to help validate findings with a much larger sample size)
  • Usability testing (of existing product or early concepts)

Lots of different views, lots of data! Each method is used in a way which is appropriate for it, and when combined they allow a degree of cross checking. The above list would be applied to quite a large project; a small project might just include a competitor review, heuristic review and maybe a few interviews. Horses for courses, really.

Triangulation to minimize bias

Specifically, the problem with relying on just one method is to do with bias. There are several types of bias encountered in research, particularly the qualitative design research we use in the field of UX and IxD. And triangulation can help with most of them.

  • Measurement bias – Measurement bias is caused by the way in which you collect data. Probably the most common form of this is the effect of the setting in which you conduct your research, for example, peer pressure on focus group participants. Triangulation allows you to combine individual and group research methods to help reduce this bias. Related to this is “response bias” in which participants tend to tell you what you want to hear. Again, a triangulated approach means you can combine self-reported and observational research methods to help balance out the problem.
  • Sampling bias – Put simply, sampling bias is when you don’t cover all of the population you’re studying (omission bias) or you cover only some parts because it’s more convenient (inclusion bias). Some research methods make it easier to cover certain parts of the population, for example using phone interviews for interstate participants can be a good substitute for the face-to-face interviews you do with local participants. Similarly online surveys or cultural probes might make it easier for you to include geographically distant participants. Triangulation combines the different strengths of these methods to ensure you getting sufficient coverage.
  • Procedural bias – Procedural bias occurs when participants are put under some kind of pressure to provide information. For example, doing “vox pop” style interrupt polls might catch the participants unaware and thus affect their answers. Similarly, an online exit survey might make the participant rush their answers to finish the survey quickly. Triangulation allows us to combine short engagements with longer engagements where participants have more time to give considered responses.

You can never rule bias or preconceptions but you should be cognizant of their presence and potential impact. In fact, failing to recognize bias is itself known as “design bias” (which also includes failing to disclose assumptions and possible bias when reporting your findings). Particularly with qualitative research, it is considered best practice to acknowledge bias and preconceptions. This is what Anthropologists and social scientists refer to as “reflexivity”.

In this regard, a bit of navel gazing is very important; self reflection and awareness of the limitations of your methods help you assess possible bias and take them into account when analysing data. Is my own sense of fashion affecting my perception of the fashionistas I am studying? Does my own view on smoking/drinking/gambling impact my research into marketing those products and services?

Tips for triangulation

Mix it up

Combine different techniques that balance each other out: quantitative vs qualitative, individual vs group, face-to-face vs remote, self-reported vs facilitated, short engagement vs long engagement etc. This is central to the idea of triangulation.

The right tool for the right job

It’s important to know what each is good for and ask the right questions in each. For example focus diary studies on what people do and think at a pertinent time, when you’re not around. Don’t ask them questions in the diary that you can ask directly (while still keeping the context). Likewise, don’t assess the usability of a website in a focus group.

Two heads are better than one

A kind of triangulation can also be achieved by having two (or more) people on the project. This helps immensely in terms of making observations, taking notes, analysis and “sensemaking”. Because unless the two researchers are very similar, they are likely to have quite different perspectives on what they are seeing and hearing, thus giving them different theoretical platforms from which to interpret and analyse. As well as simply allowing them to capture more data, the researchers balance each other out.

Layer upon layer

Yet another way of achieving a kind of triangulation is to conduct your research in successive layers of detail. Start off with a very broad piece of investigative research to identify top level issues and to provide better scope for the next layer. That next layer would be more detailed and focus on a smaller area than the first level. And so on and so forth.

Setup a feedback loop

Feedback findings into later methods to help validate or flesh out issues that have already popped up. Continuing the above example, during a focus group you might explore an issue that only one interviewee mentioned and see what the rest of the group thinks. Similarly, you can adapt your interview approach as you go, feeding back what you learn into the questions you ask. My favourite example of this would be using a quantitative method with a large sample size, such as a survey, to validate findings from earlier research using a relatively small sample size.

Be reflexive, grasshopper

Look for where preconceptions might exist, in yourself and your colleagues, and work out how you might be able to minimise their impact. As discussed above, using multiple researchers will make this easier, but if you’re working alone, keeping a journal or diary for yourself can work well in this regard. It forces you to examine what you’re doing and how you felt at the time—your emotional state on any given day actually makes a big difference to how you conduct research. And make sure you state a summary of this in your findings, along with any bias that you feel may have had an effect on the research.

Re-visit participants

Visiting the same people at multiple times throughout the research can give good results. For example, you might invite interviewees back for a focus group, allowing you to compare and contrast their views with other similar participants. This longer engagement with these individuals allows you to see how their goals, attitudes and behaviors change over time.

It’s often said that any research is better than no research, and largely this is true, but you need to take account of the fact that your methods may have limitations, such as bias. Triangulation is a very useful means of capturing more detail, but also of minimizing the effects of bias and ensuring a balanced research study, no matter how big or small that study may be.

Note: Patrick will be running a full day workshop on design research methods for UX practitioners at UX Australia 2009 – a 3-day user experience design conference, with inspiring and practical presentations , covering a range of topics about how to design great experiences for people. It will be held on 26-28 August 2009, in Canberra (Australia).

Top image by watchsmart

Patrick Kennedy

Patrick Kennedy is a user experience architect and design researcher based in Sydney, Australia. Besides assisting clients to better understand their audience and design for them, he speaks, writes, blogs and generally blurts lots of somewhat interesting stuff.

11 comments on this article

  1. Pingback: Twitter Trackbacks for Johnny Holland - It’s all about interaction » Blog Archive » How to combine multiple research methods: [johnnyholland.org] on Topsy.com

  2. Pingback: Research triangulation article on Johnny Holland | Pat's Point of View

  3. Pingback: Johnny Holland - It’s all about interaction » Blog Archive » How to combine multiple research methods: Practical Triangulation

  4. Pingback: Laura Carlson (lauracarlson) 's status on Monday, 24-Aug-09 23:23:58 UTC - Identi.ca

  5. Thats good. As a customer experience researcher, I like online, task-based research for a variety of reasons, most importantly the insights I gain from the users are rich, articulate, honest, and sometimes very surprising.

  6. Thanks “user experience research”. What kind of tools do you use for your online, task-based research?

  7. Pingback: Twitted by robtannen

  8. Pingback: Fábio Caparica » del.icio.us entre 23.08.2009 e 31.08.2009

  9. Enjoyable read and nice meeting at http://www.uxaustralia.com.au

    Suggest researchers also need a point to reassess a research plan, if you feel that patterns are already pointing consistently one way. Is it time to reassess? Are the insights making sense? Is it time to re-direct the questions? Is the business flexible enough to cope with a change to the research plan? (sometimes they are not)

    rgds,
    Dan

  10. Pingback: Posterous as an online cultural probe (user research diary) | USiT

  11. Pingback: Selecting methods [part 1] | Urban Phosphorus Ballet