Good design is hard to do. The very nature of human centred design is confronting, challenging and often uncomfortable. Every project builds up a collection of challenges along the way, which can pose significant risk to the project’s success, and if we don’t tackle them head on they can be detrimental for everyone involved. How can  we share and learn from each other’s challenges?

At Neoteny, we refer to the collection of challenges in a project as its ‘challenge pile’, given they’re exactly that; a pile of issues, constraints and problems. We keep track of the challenge piles using walls in our studio for each project. Some are collections of post it notes, others are photographs, drawings, diagrams, scribbles or hand written notes. Each week as part of our work in-progress meeting (team jam), we take stock of each project’s challenge pile.

We ask ourselves the following for every challenge:

• How did this challenge come about?
• Briefly establish the current reality, including:
  • What has it cost the project? Not necessarily in financial terms, what has been the cost to our momentum, resources, client expectations etc.
  • What is the potential impact? In what areas?
  • Could it have been avoided? How or why not?
• How have we handed it thus far? As a group, explore options for how we handle it moving forward.
• Agree on proposed solutions or new approaches and secure buy-in from everyone involved.
• We’ve found that this structure helps us stay out of the drama whilst understanding the drivers for each challenge, and then focus on solutions. This makes the process much more collaborative and productive, we aren’t sitting at our desks sweating over something we could probably work through together in a few minutes.

We recently had a team jam, and here’s what came out of our challenge pile review:

1. Customer needs and business requirements collide.

This project is in its early stages. The client came to us with a new product that they wanted to develop, the first step was to research the product feasibility and desirability in the market.After conducting research aimed at validating the customer need for a new product, we found that what the customer needed and what the client planned to launch, were two very different things.

We’re currently in discussions with the client to try to shift the project objectives and focus, to meet real customer needs. As a group we decided not to proceed to stage two unless we could get their buy in on a revised approach.

Question for readers: What would you do in this situation?

2. Budget streams are unclear for future phases of work.

We’ve been involved in projects in the past that have unclear funding streams for future work. This is especially common in large corporates, where steering committees assign funds based on a comprehensive business case analyses including return on investment predictions. These can’t necessarily be defined without first doing some work. The problem with this structure is that you have a team of stakeholders that can only see as far as the next steering committee meeting. This makes a design project with a strategic foundation i.e.  something that’s designed with the whole in mind, very difficult to manage.

This particular case was flagged early because we’ve seen the warning signs before. The signs included hearing things like:

• “If we build this…”, highlights the fact that the stakeholder doesn’t believe this project will make it to implementation.
• “We need to show results by June…”, if you ask why, you’ll probably hear something like “that’s our next steering committee check point”.
• “We won’t be able to build that”, if you ask why, you’ll probably hear something like “because the next budget release won’t be anywhere near that much”.

In the past, this issue has created a divide between the client or project stakeholder group and the design team. Whilst the stakeholder group is focused on securing the next round of funding to ensure that this phase can move to implementation, the design team is focused on exploring and exposing every possible opportunity for solving the design problem.

We’re currently working with senior management to ensure we have their buy-in throughout this project. In our experience, we’ve found that if the person signing the cheques is on board with the approach, the whole stakeholder group is much more relaxed and inclined to get their hands dirty in design.

Q: Have you experienced this before, and if so, how did you get around it?

3. Stakeholder groups have varying ideas of the project objectives

Have you ever been in a project meeting and realised that the client team doesn’t agree on the project’s objectives? This is an awful moment for a designer. It’s the moment when you move from designer to mediator. Playing mediator with your clients is generally not a lot of fun and not how you want to be spending your energy.

The design team typically work with clients to reach a shared set of project objectives. If you find yourself in a situation where you think this has happened but it isn’t the case, then it needs to be dealt with immediately. This agreement needs to be made before design work can start. Of course, these objectives may shift and be adjusted as part of the design process, but the aim is for adjustments to be made as a whole, not as a fragmented set of perspectives from different stakeholders.

We’re currently experiencing this on a scoping project we’re working on. It came about in a workshop, where up until that point, the team seemed aligned. We handled it by stressing the need for a shared project vision and refusing to move forward without one. We managed to facilitate developing a shared set of objectives, prioritising them and we’re currently working with the client to ensure that every stakeholder is in agreement on the vision for the project.

Without this shared vision, we put the success of the project at risk because no one is clear on what success will look like. We’re currently working through ways to communicate this in a more explicit way to our clients before we start on their projects.

Q: Perhaps it’s about signing off on the project vision, would that make people more accountable?

4. Mystery stakeholder stomps on the project.

Does this scene sound familiar? The design team is working away, the client is happy and excited, they’re getting involved and spending time designing with us. Then BAM! Along comes the mystery stakeholder who has significant influence, but just “doesn’t like blue”. In most cases, the mystery stakeholder is a fairly senior member of the client team who hasn’t been along for the ride and is looking at the design solution without any understanding of the brief, the agreed approach, the challenges or the project’s constraints. This situation can be crippling. Challenges like this can impact resources, motivation, relationships, momentum, time and budget. You could argue that it’s the design team’s fault for not ensuring that all stakeholders were engaged, the project owner’s fault for not engaging the full spectrum of players, or the mystery stakeholder’s fault for stepping in with the ‘I’m gonna leave my mark on this project regardless of how you got here’ kind of attitude.

We’ve started to enforce what we call a stakeholder roll call. At the start of every project, and within our terms and conditions we gather a list of stakeholders, their roles and responsibilities and have the project owner sign off on this list. The full list of stakeholders are required to sign off on all milestones and agreed deliverables.

We acknowledge that the stakeholders may change, but the terms allow for this situation and protect the progress we would have made in the project up to that point. The success of this approach remains to be seen, though what it does achieve is a level of accountability agreed up front for the potential impact of those ‘stomping moments’.

Q: How do you protect your projects from random stakeholder stomping? How have you dealt with this situation in the past?

Where To From Here?

As you’d expect, there’s a ‘magical box’ of learnings and insights created by each challenge pile. It’s what we choose to do with the magic that makes the obstacles and the heartache worthwhile. I’m sure we’ll learn a hell of a lot more as our company matures, but here are some of the more salient ones we’d like to share with you:

• There’s not always something ‘to do’, there’s something ‘to know’. There are situations we can’t ‘solve’ in the context of the project we’re working on. But being aware of the specific challenges and carefully managing expectations accordingly can be a very effective approach, one which better supports our potential success.
• As a company (and perhaps as an industry) let’s be more reflective. That doesn’t mean we have to wade into the drama or analyse it ad nauseam, but we do need to nip things in the bud, be honest with ourselves and the team, be open about the potential impact that the shifts might have, and involve everyone.
• Getting the players involved as the challenges arise. Rather than keeping our ‘dark passengers’ under our hat, and suffering in relative silence, all with a smile on our face, let’s face the challenges together! Clients and project stakeholders are often quite pleased when you invite them to be part of the solution. Any shifts to the project approach are also much more likely to fly if we’ve got buy-in from everyone involved.
• Sharing the good the bad and the ugly with our peers. Let’s foster a culture where we share both our triumphs and our failures, rather than keeping the latter closely guarded. As a collective mind, I’m sure we can come up with some inspired, insightful ways to circumvent and also completely avoid some of these challenges.

We believe that we can get better at this thing called ‘design’ if, as an industry, we can make the most of lessons we learn from these challenges. After all, they enable us to be more resourceful, they give us an opportunity to be more creative, to build stronger teams and deeper relationships.

So, what do you think?

Michelle is one of the speakers at UX Australia 2010, taking place 25-27 August 2010 in Melbourne, Australia. The conference has sold out, but workshops are still available, or you can go on the waiting list. See the UX Australia site for details.

Here in Drupal7 User Experience Project land we’ve been moving from ‘iteration zero’ to the actual production iterations. In iteration zero we’ve been doing a lot of our strategic thinking and documenting, but now it is time to start producing output that the developers who are working with us on this project can turn into something that will be contributed to the Drupal7 Project.

There is a real cadence to the project, and although there is no time in the schedule for us to take a breather, between you and I, it has been impossible for us not to do so (just a little), before heading back into the fray. I’ve noticed this effect a few times in agile projects and I think that I’m going to try to encourage project managers to allow for a little breather at this point in future projects I work on.

I thought I’d take a moment to share with you some of the other shifts that start to happen as we move from Iteration Zero into the Production Iterations.

Communication Framework: From Abstract to Concrete

As I’ve mentioned in the past, a big part of the time we spend on this project is spent either communicating with the community about the work we’re doing, our process and our ideas, or trying to work out a better way to communicate with the community.

One of the biggest challenges of the Iteration Zero stage in the project is that it is, by and large, a series of quite abstract and strategic discussions.

It is really easy to forget that many people find abstract and strategic discussions really difficult. I think there are particular types of brains that embrace the abstract better than others, but experience in this project phase is also very helpful.

In Iteration Zero there is often a lot of writing and talking and not much making/showing – this can create a very challenging environment for project participants. It is pretty easy for people to have vastly different interpretations of the same concept and it can be difficult to make sure that everyone is on the same page with the higher level strategy for the design and product. I’ve experienced this recently not only with the Drupal project, but with a few other projects I’m involved in.

Abstract discussions can be difficult to grok due to their predominantly conceptual nature.

It is pretty easy for people to have vastly different interpretations of the same concept and it can be difficult to make sure that everyone is on the same page with the higher level strategy for the design and product.

I can’t begin to tell you how many times I have explained and re-explained the very same concept, each time thinking that it’s been communicated clearly, only to discover that we still have at least two very different ideas about how something is going to work. There are at least two reasons for this: firstly I have to take responsibility for communicating – if the message isn’t being received I have to re-evaluate either what or how I’m communicating. We also have a second and somewhat unique problem when communicating with the Drupal community and that is that they have a tremendously strong mental model of How Things Work In Drupal. Every time an idea is presented the community almost invariably tries to map it directly to their mental model of How Things Work In Drupal – this is natural and what we *do* with mental models, but when the concept we’re suggesting actually breaks the model, we can run into trouble. It just doesn’t compute! It becomes abstract, difficult to understand, as we have to try to find ways to make concepts more concrete.

Iteration Zero can be a stressful time as a result of this abstraction – people aren’t really certain that they know what you’re talking about, but you’re also asking them to make decisions that will be really significant in shaping the product they’ll be getting at the end of the project.

I think it’s pretty common for people to be fairly fraught towards the end of Iteration Zero.

Thank goodness it is also around this time that something excellent happens – things start to become a little more concrete. There are still a bunch of abstract concepts that need to be agreed on, but as designers we’re also starting to get our heads around exactly how things will fit together and we can start to communicate that.

This is around about the time that we had a fundamental overhaul of the way we’ve been communicating with the Drupal community and interested onlookers on this project.

Towards the end of Iteration Zero we were starting to get a little down about the some of the feedback we were getting on the D7UX project – people were saying that they didn’t want to get involved because it was too intimidating for people who didn’t have UX experience and expertise, that they didn’t think it would actually happen or be a success, that they felt that the discussion was too disjointed and widespread.

It was clear to us that we needed to change the way we were engaging with the community to help them help us. Essentially, we needed to change the structure of the conversation from it’s abstract Iteration Zero format to a more concrete format appropriate to the production iterations, and, we suspected, to a format that most of our participants would find much more comfortable.

Over the course of a day, we created a ‘Project Framework’ on the D7UX site by breaking down the project into it’s main component parts and providing a wireframe, description and outline of ‘what we’re thinking’ for each part. Threaded comments allow people to give their thoughts on each component as it evolves over time.

Allowing people to participate in a place that is most comfortable to them is a key part of our communications strategy. We wanted to continue with this strategy even as we move into this new phase of the project, but also to aggregate the discussions into one place and to facilitate this we created a system of tags for the project components and put together a series of Yahoo Pipes to pull tagged content together. We added a link to these pipes on each of the component pages in the framework.

It was a pretty big overhaul and quite a time consuming process, but almost immediately we noticed a significant difference in the way that people were communicating with us on the project – the interactions became much more focussed and productive and felt a whole lot more positive, and that trend seems to have continued. It also makes it much easier for us to be more conversational with the community in the project – thanks to the simple addition of threaded comments and also the aggregation of the main part of the conversation into one place.

Overall, we’re really pleased with the effect that changing the format of the conversation from abstract to concrete has had on the project to date and the effort involved has already been rewarded.

The Challenge of System Design with User Stories

Another major challenge that we’re butting up against at the moment is to try to make a system design fit into an agile environment.

I’m a big fan of agile methodologies and have had a long term interest in finding better ways for UX practitioners to engage in agile methods. Unfortunately, there is no denying that pushing a design project like this one into agile iterations is tricky.

The way that our user stories are being developed at the moment is that the project manager from the developer agency (Acquia) is writing user stories then pushing them over to us to check that they are right and for us to adjust and re-order as required. To date, we have mostly let them sit in a large spreadsheet whilst we focus on the design strategy (iteration zero) and try to ignore the need for user stories.

We’ve done quite a bit of work on developing an Audience Matrix that allows us to take quite sophisticated ‘views’ of the design from multiple audience perspectives, but to translate this into user stories is untenably complex. The alternative to date has been overly simplistic. We are struggling to find a way to make good use of our audience modeling work to date without breaking agile.

Another issue that we’re butting up against is the nature of system design and templating in an agile environment. There are sets of design elements or template components that would ideally be designed in components then re-used throughout the project – for this project examples of these would be the admin header, the overlay window and the edit-in-place interaction model. Describing these using user stories is incredibly clumsy and inappropriate.

Once these elements are built and we start looking at user pathways that make use of them for particular tasks and outcomes then user stories will come into their own, but it seems that in the same way that developers need a piece of time to set up their development environments and databases without requiring user stories being used, designers need some time to get the ‘design environment’ set up without requiring user stories.

Again, this is something that I’ve come across on a number of agile project I’ve worked on but I’ve not seen any allowance for this way of working in Agile UX project methodologies.

If you’ve had similar challenges and some ideas or solutions then I’ve love to hear from you!

Update on Crowdsourcing Usability Testing

In my last update I was telling you about the Crowdsourcing Usability effort we had launched. Since then we’ve seen that WordPress have launched a similar campaign and they managed to get coverage in the New York Times no less, so we will be watching their progress with interest. Exciting times!

Launching Microprojects

Want to dip your toe in Open Source Design? Help out with D7UX? Well, here’s a great way to give it a try – sign up to help out with one of our microprojects! You need to commit just 12 hours over 3 weeks, but you’ll get a feel for what it’s like to design with one of the most vibrant and clever communities you could ever come across. Be warned, it’s challenging but potentially very addictive!

22-year old David Karp may not yet have the profile of web 2.0 celebrities such as Biz Stone or Kevin Rose, but his keynote at Auckland’s Web09 suggests that it may only be a matter of time. His high speed, high impact talk about what he has learned running mashup blog platform Tumblr – established in 2007 and currently with close to 1 million publishers on it – was well worth the price of the conference ticket.

Aided with supersized text slides, Karp suggested that the main principles for creating online communities is through understanding engagement, use, negativity, change, and feedback. The Web09 audience listened closely.

Engagement

It turns out that Karp understood users and engagement – “the thing that makes people hit refresh” – well before Tumblr. Having loved America’s Funniest Home Videos and the “promise” of potential celebrity, Karp mixed this idea with the US “Adult Swim” format in 2005 to create the world’s first cartoon video podcast, Frederator Studio’s Channel Frederator (it had 1,000 submissions in its first week and is still running today). Tumblr has a similar concept in that people can easily upload different types of content, browse, and share it (while retaining their profile as having found or created it).

Use

He explained that use has to be watched (people decided that the tumblr star icon was bookmarks rather than favourites), but can be tended through knowing when to seed “create your own test case” (tumblr put good examples of tumblr logs on the main page so that people would replicate it) and when to prune (hopefully done only once, for example Vimeo banning uploads of video game play).

Negativity

Karp also wasn’t afraid to deviate from the straight and narrow to get things done. He suggested the best way to deal with negativity of malicious users (‘greifers’) was by hellbanning (“ Don’t let people know they’ve been blocked. Remove it for everyone but them”).  This raised a few laughs – though he also pointed out that the Twitter concept of follow/unfollow (asynchronous relationships) is a similar idea. In a wider sense, he blamed a lot of the vitriolic comments on blogs and Youtube on the community-unfriendly format: “You’re impeding their voice, making them third class citizens hidden at the bottom of the page in a blind comment box. Instead, he suggested using the UI to “Make your users 2nd class citizens, not 3rd class” by making their comments more prominent, ruling out anonymous posts, framing what people are to write in fields – “It’s hard to use a question field to say ‘F**k you’”  – generally giving them more responsibility (while still retaining control).

Change

Many of his tips for change (e.g. “don’t test … leak it”) were also appropriately irrational responses to human irrationality (“you’re changing the place people live in … Don’t launch for the sake of it”). Most controversial (see the tweets) was to “take a note from slumlords … make things worse before they get better”:  i.e. degrade the experience (run slower, dim colours) before an upgrade. Great idea, potential PR disaster – use if you dare, I guess.

Feedback

His final suggestions about feedback (i.e. questions) was that rather than filtering them by request and volume, it was far more useful to assign problem to user type:

Doing this allowed him to understand the relative importance of the problem in relation to the user, and from that how to prioritise (i.e. beginners were an immediate concern as they might leave if they were frustrated, but others also needed to have reasons to stay with the network).

The best testament to Karp’s talk was to see the majority of laptop screens in the room on tumblr by the end of the presentation. App aside, Karp proved himself to be an engaging, illuminating and knowledgeable speaker, whom I would highly recommend seeing speak should you get the chance (hear an interview with him while he was in Auckland).

Why is it that when people think of involving a community in design their minds immediately turn to surveys and polls? Enough already. Involving a community in your design process doesn’t mean making the community make the design decisions… this is why we’re dreaming up new ways of engaging a community, qualitatively, in the design process for the Drupal 7 User Experience Project.

What’s so bad about a poll?

I think there are two key reasons why voting should generally be avoided in community design practice:

1. Your community is (probably not) made up of designers

what colour would you like? Which icons do you prefer? Do you want your navigation here, here or here? Which of these three visual design treatments do you like best?

When you’re out doing design research in person, you don’t ask people where they’d prefer their navigation. Rather you use conversation and observation to gain insight into who your users are, what they need to do with and any other information you need in order to make good design decisions. Why should it be any different online?

Humans, at the best of times, are pretty bad at telling you how they’re going to use things and what configuration will best suit them. We know that, don’t we? So what makes us think a poll is a good idea? A
nd what do you, as a designer, do with data that says that 67% of the people surveyed prefer green? You don’t know anything about *why* these people like green, or in what context they actually hate green. This data contains no real insight.

2. Voting is like fast food
cheap

It is easy and initially feels good, but it leaves you wanting more

A
 vote is super quick and easy to set up and similarly it is quick and easy to participate – you can sit back and watch the numbers roll in. ‘Look!’ you can shout, ‘see how many people are involved in our project!’

It’s easy to be seduced by the numbers. But ultimately a few quality interactions with your community are worth more than 1.000 empty gestures. Just as when you compare numbers in a qualitative study and a quantitive study, it’s not always a matter of more is best. Ultimately everyone is left wanting more. The designer wants more insight and the participant more engagement.

We need to be more thoughtful, more creative and willing to work a little harder in designing ways for communities to engage in an open design process. With that in mind, here’s a few activities we’ve been trying out on the D7UX project.

Crowdsourcing Wireframes

Extracting Existing Community Knowledge

This is an exercise that we first did on the Drupal.org project and are now running again for D7UX. Essentially we invite people to draw up a wireframe (in whatever medium they prefer) of a part of Drupal they think needs improvement. This time around we also asked people to make a quick screencast walk through to give us some more context and understanding of both the problem and the solution. Participants post their work either to their own blogs and give us a link or to the Flickr and Youtube groups we have for the project.

There is a reasonable amount of effort involved in participating in this activity, so we tend not to be flooded with responses. This is good, for a number of reasons:

• The data is quite tough to analyse.
• The people who do participate really do care about the project and the problem and have given it some real consideration. Each submitted artifact then becomes a discussion point for the rest of the community to gather around.

Working with a developer-led community like Drupal we do tend to get our ‘wireframes’ in pretty high fidelity. In fact, a wireframe for us is everything from a pencil sketch to a live implementation. Especially on a project like D7UX where there is a ‘back story’ to everything and almost every idea has already been considered and discussed at length. It is a great way to draw some of these conversations out earlier in the project than might otherwise be the case.

Crowdsourcing Usability Testing

Capturing Scale, Sharing Our Process

Continuing in the crowdsourcing vein, we have also been trying to capture the scale and diversity of the Drupal community by recruiting willing participants to help us do some user research on the project. We first attempted this exercise during the Drupal.org project but it didn’t really take off. Despite lots of interest, no actual tests emerged. So, it seemed like a good idea but we needed to tweak the process a little to make it more accessible and easier to engage. So far we’ve run one round of CrowdTesting for D7UX and happily, we’ve had about a dozen people around the world participate. And the information we’ve been collecting has been put to great use already.

What did we do differently this time? Rather than just putting the concept out there and saying ‘go for it!’, we have scheduled a series of dates throughout the project when testing will be undertaken. I’ve also been a lot more prescriptive about what needs to be tested and how to test it this time around. I provided a downloadable document with the materials (we’re testing paper prototypes at the moment) and a script to follow, plus some interviewing tips for newbies.

Participants conduct a short test and post the results to an online questionnaire. We also encourage participants to video their test and post the video to our Youtube group (or elsewhere if they prefer). The posted video is particularly helpful in allowing us to analyse and contextualise reported findings.

What are the benefits of this activity? Not only do we gather more research data from all around the world (and yes, we are still coming to terms with the challenge of languages other than English), but we are also making user research much more accessible to people who might otherwise never had exposure to it. We are really hoping to continue to promote the practice of user observation throughout this project. And we hope to see real user observations used as evidence as we work with the community to debate the best approaches to various aspects of the interface.

‘Pimp Your Admin’

Extracting Existing Community Knowledge

When we first started on the D7UX project, we had a feeling that lots of people out there were ‘hacking the interface’ of their Drupal admin to make it more usable for their clients. We knew this, of course, because we’d seen a few instances of it. It struck us that this would be a great way to do some ‘requirements gathering’: to take a look at as many ‘hacked interfaces’ as possible and see if there were similarities in the changes that people were making to the admin interface to make it more human friendly.

And so we came up with ‘Pimp Your Admin’ in which we invite people to walk us through the customisations they’ve made (or that they regularly make) to the admin interface. We launched this at a conference, Drupalcon DC, where we sat down with people and did the screen recording and interviews in person. But this is also an online activity and members of the community from around the world have since prepared screencasts sharing their own personal interface modifications. This has been interesting both for us and for the community – you don’t really get to see other people’s admin interfaces very often!

As we suspected, there are definite ‘themes’ in the changes made to the interface and even at this early stage we know that several of these will be mirrored in our proposed design. We’re continuing to capture this material throughout the project too.

Having the opportunity to work within this community context is an amazing challenge and opportunity, we’re really trying hard to be as thoughtful about the way we engage with the community as we are with the design work we produce but it is a constantly evolving process of trial and error and refinements.

Want to see where we’ve gotten with it so far? Be sure to check out our recently released Initial Concepts and Directions and be sure to let me know what you think!

Top image by yoroy