Over the last few years, social computing has been relegated to asynchronous websites like FaceBook and Twitter, where users connect with many people and their collective information is harvested for the larger group. However these are still largely individually actions, not synchronous… yet we call it “social”. I would like to expand that definition.

Think of a computer where you sit down and work simultaneously with your friend. Imagine you and your friends all playing a game or solving a problem at the same time on the same computer. This has been one of the key aspects of our vision at Microsoft Surface. As I’ve presented over the last year on Surface, I’ve talked a lot about the aspects of social computing. Initially when I started on the team I was skeptical of social computing as Surface defines it: multiple people working around a single computer. Since that time, I have seen some of the great innovations and situations that Surface style social computing allows. With its vision system (instead of capacitive or infra-red), Surface can recognize 52 simultaneous points of touch, as well as physical objects, making it a computer for a truly social environment.

Social computing is described as the intersection between social behavior and computing systems, and often in somewhat ambiguous computational terms. I question how much of what happens on social sites like FaceBook [et al] is really social (I don’t often come to work sharing a list of 25 personal oddities about myself). The only real social aspect is that you’re sharing items with other people in an easy way across geographic divides. Although the web seems like a macrocosm of that definition. I’m not sure why things like instant messaging are not considered social computing, but they are more social than most sites bearing that label.

I would like to implore readers to expand their definition of the term social computing and realize it can apply to many more situations than it currently is; those being actual social situations. I would describe that as what Surface is aspiring to be, the first true social computer. It provides context and use for multiple people, on all sides. Although true social computing can be done with a single computer and two or more people, it may not be optimal. Below are a few of the ways I’ve come to think about social computer usage:

• Driver as a presenter: this happens when you’re showing someone a YouTube video
• Driver (w/ an influencer or back seat driver): this happens when you’re searching the internet for someone and someone is telling you what do type in
• Turn taking: passing a laptop back and forth to share information
• Simultaneous: both playing a game on Microsoft Surface. I’ll call this synchronous social activities. Very different from the three above it

Of course none of what I describe here is the current way we define social computing, which is why I’m asking people to expand their thinking. Perhaps there is another word to describe these situations? Whatever happens, it’s become clear to me that the computer cannot simply stay as the personal device it has been and designers should begin to think about social proximity activities and behaviors. As technology becomes more pervasive and cultures become more acquiesced to computers, there will be a need and desire to continue and expand the social aspects.

As an aside, while I bring up Surface several times in my posts, please don’t take that for blindly selling the technology. I am very aware of its flaws and issues, and part of my opportunity at Microsoft is to make those better. For those interested, here is my talk from MIX09 on Surface and touch computing where I discuss both my love and discomforts on those topics.

In the wake of the ever increasing popularity of gestural interfaces, Dan Saffer wrote his newest book: Designing Gestural Interfaces. In this publication by O’Reilly he tries to give us some valuable insights. But did he succeed?

Gestural interfaces are not new, but they are more popular than ever. Everywhere around us you see new ones popping up. And before everybody is coming up with their own interpretation and translation of what they should be and do, we must evaluate what has happened in the past and how we must deal with the future. Dan Saffer’s book is a great starting point for this exercise.

Let’s go through the book chapter by chapter.

The basics

The first chapter is an introduction to the subject. It gives a good and basic explanation. One interesting part of this chapter is ‘matching the gesture to the behavior’. This approach states that the best designs are those that ‘dissolve in behavior’. Here the product must become part of what the user is doing fluently. This is what good gestures should be, combining peoples natural behavior with a related action.

The human body

In the second chapter Saffer focuses on the human body, something a lot of designers often forget. The importance of understanding the mechanics of a human body is really fundamental in order to design good gestural interfaces. What is you body capable of doing? And what not? And we should not forget ergonomics. This is important even though there is no mouse. Saffer goes into this subject rather well, describing possible limitations and pointing us on some simple, but valuable facts (10% of adults are left-handed).

Patterns for touch, interactive & free form

After the first two chapters the book dives into possible patterns for gestural interfaces. It’s an important step Saffer tries to take here: trying to define an international set of patterns. Several companies are developing gestural interfaces and are creating their own patterns, independent of each other. Some, like Apple, are even trying to patent some patterns… so it’s really important to start describing a set that is recognized by users and freely usable by any company. It would be a great step if the patterns Saffer describes would be available online, open for discussion and growth.

Documenting gestural interfaces

Documenting static websites in the 90s was pretty straight forward. But with growing interactivity and gestural interfaces it is a challenge for designers to capture the way an environment or device interacts with a user. The fifth chapter of Designing Gestural Interfaces looks into different ways to document your project. The one I like the most is drawing storyboards, encapsulating not only the interface but also the context and how the user behaves in it. It’s a great way to test for yourself if the design works, but also a superb way of presenting the concept to clients.

Prototyping

When you’re done with the sketching you want to prototype an interface. In the sixth chapter of the book you get some low- and high-fidelity ways of prototyping. Amongst the low-fidelity is the creation of paper prototypes, which I really like. It’s an easy and really fast way of making a ‘working’ version of your product. You can give it to a colleague and let him play around with it. But even playing around with it yourself is really helpful, since you will ‘feel’ and see what it does. Of course paper prototyping is not always the best solution, since gestural interfaces can also be about bigger objects or even spaces. But these are also easily prototyped. Saffer gives some nice examples of these.

Communicating

For me chapter 7 “Communicating Interactive Gestures” was one of the more interesting ones. It learned me the true importance of communicating that there are interactive gestures or not. Imagine that there are some public spaces that have gestural interfaces, and some don’t… but it’s not communicated well. This will cause real stress, since people won’t know if, when, how and what they will trigger an event. It’s really important to be clear about this, stating when there is an interaction possible… what it triggers and how. But in order to achieve this we will have to find an international language, explaining this to us. It could be something as simple as an RSS icon, but it has to be something clear.

Future

In the last chapter Saffer tries to sketch his vision for the future. Unfortunately he plays on the safe side and only describes the main trends going on in the field. So when you follow the important sites in the IxD and UX field, you are covered and can skip this chapter.

Conclusion

Before I started reading this book I thought it would dive deep into the world of gestural interfaces. But I should have known better… Since this is the first serious attempt to capture the subject it is logical that it needs to introduce us to a lot of new things. And although only the first chapter is called ‘Introducing Interactive Gestures’ I have to say that the entire book is an introduction to the subject. This doesn’t mean it doesn’t give helpful information, but it never goes into the subject deep enough to totaly cover it. This makes Designing Gestural Interfaces a great book for people new to the subject, like students. Let’s take this book as the fundament and see some new ones building upon it, going more in-depth.

Book Details
Designing Gestural Interfaces
author: Dan Saffer
publisher: O’Reilly
details: 247 pages, paperback

In my first article I laid some ground work discussing touch as part of an ecosystem and the matrix of inputs available for people to interact with systems. In this article we’ll look at making things a little more complicated by adding in different technologies, form factors, and the cardio effect of touch.

Even when narrowing the input method solely to touch, designers are still faced with a complex matrix of issues that they may not yet be aware of. Let’s consider orientation the first variable in this equation. Are you designing for a horizontal, vertical, or tilted system? In the media today, such as CNN, we mostly see vertical touch walls. A few years ago you could barely find companies developing these, and now the field is fast filling up with companies using different technologies, sizes, price points, and interaction techniques to develop their systems. Until touch walls are really commoditized, I don’t see them moving much beyond novelty and “cool” factor. Not that I’m complaining, anything that gets touch more into the public eye is good business for me. Although as designers we must not become jaded to the fact that most people have not used touch walls… or touch computers beyond an airport kiosk or ATM, despite what we may think.

Continuing on vertical systems, I’m not sure how many readers here have tried one, but while they are cool for show-off factor, there is one key piece of information people forget. Holding your arms up to use a vertical touch system makes you tired. It’s not easy, and not much fun after a few minutes. Hence why none of these technologies are made for serious personal use at this point. Of course the other aspect besides the cardio work out, is accuracy. When holding your arms out, people tend not to be super accurate because they must fight gravity. A horizontal system can produce better accuracy in that respect [although there is still the fat-finger concern], but may have issues from wrists or elbows acting as accidental inputs. Extrapolating some basic starting points, use Fitts’s law, make things large, try to determine accidental contacts, and utilize transient tasks for vertical orientations.

All that in the simple aspect of vertical and horizontal, and we haven’t even begun yet. In addition to orientation, designers must start to think about technology and form. With Windows 7 introducing touch, designers will now see touch becoming a more common interaction commodity on personal computers. Looking at the mouse and keyboard, the positive aspect is they’re standardized technologies. A mouse is always represented with a cursor, the same way every time. The keyboard has the same keys and same layout [within a given culture]. Despite different technologies and hardware manufacturers, there are constants in these tools. Unfortunately touch, manipulation, and gestures do not get the same ease of consistency. Consider the chart below.

This is a small portion of what is available on the market for touch PC’s. Since there isn’t a standard way to provide touch input, different companies are presenting their methods in new and separate proprietary ways. This means tough problems for designers of touch and gesture working across various systems; in other words, anyone designing a touch application for Windows 7. In addition to the basics, like the number of simultaneous touch inputs systems can accept, the difference in technologies means all touch inputs are not created equal. A capacitive system, like the iPhone, relies on energy generated from a person’s body, it needs a person to make a touch. An infrared camera systems, like the HP Touchsmart, relies on having an object break a “blanket” on infrared beams. This means an object (not necessarily a person) can create a touch contact point. Additionally, varying numbers of simultaneous touch inputs means gestural interfaces can be more difficult to design and develop. Adding to these issues, people who design cross platform applications must consider the ability for direct touch on Windows 7 and indirect track pad gesture interactions on an Apple MacBook.

I wish I could offer great suggestions or solutions for how to tackle these problems, but we’re searching for the answers as well. For now I’ll just have the leave with the thought that designing systems for touch will get harder before they gets easier, and I look forward to it. New challenges, new interactions, and adapting the worlds behavior to a new type of input. We have some of the toughest problems around. This isn’t the mouse, this isn’t the keyboard, this isn’t controlled. It’s design in the wild, sometimes out of our control… how will we all work to solve it?

For additional discussion and insight on these topics you can view my interview for the MSDN Channel 9 site.

Initially we only had a keyboard for the command line and text entry. Then the mouse appeared for navigating two dimensional plains of UI. Now the field of computing has a new input toy to play with; our hands. Touch, multi-touch and gestural computing, also known as Natural User Interface (NUI) has become the newest input craze. Excitement around this has even spurred comments predicting the demise of the mouse in the next 3-5 years1. Computer designers (and engineers) have become engrossed with the ability to touch the screen with multiple fingers and control software by waving their arms. However in this excitement, have designers overlooked how to properly engage users and use multi touch to create useful, innovative, and interesting experiences? Perhaps touch and gesture are simply the new shiny objects in the room, soon to be discarded for the next new thing. In my next few articles for Johnny Holland Magazine I’ll look at some of the details of touch and gesture computing and what I’ve learned as a practitioner in the field.

Before I dig in, I want to plug Designing Gestural Interfaces, by Dan Saffer. The book is a great starting guide and reference for anyone looking to get engaged in this field. I’d suggest grabbing a copy if you’re new to the ranks of touch and gesture design.

Touch is but one slice of the pie

Let’s start the journey here. As a designer on Microsoft Surface, we’re uncovering and discovering things as we go. In my work I’ve come to learn quickly that touch, gesture, and NUI are not right for everything. As obvious as this sounds, it’s often overlooked. They should be considered part of an input ecosystem. Each type of input below has unique attributes that make it good for certain types of interactions between users and systems. This is not a comprehensive list, but here are some of the most common input and interaction methods.
•      Keyboard
•      Mouse
•      Stylus
•      Voice
•      Single Point Touch
•      Multi point touch
•      Gesture

Each of these methods have pros or cons associated with them. Text input is a perfect example of a task that touch is rather inadequate for. There is no haptic feedback upon pressing the keys, and there isn’t tactile feedback to touch type. Touch also falls short in applications that require precision, such as Adobe Photoshop or Microsoft Office Excel. A mouse would be able to cover ground quicker across the screen and not make the user reach back and forth, as well as more precise in its actions. However when people begin their design of touch, they forget all this, and seemingly everything else.

When not done properly, touch and gesture can appear as a step backwards..

A belief I’ve heard is touch can be so compelling, people will forget the inadequacies, when in reality, it only serves to shine a light on the downfalls of touch. When not done properly, touch and gesture can appear as a step backwards. The (design) problem takes a back seat to the “innovation” of touch. My advice for any designer approached by a client in need of a touch system (holding pictures of Tom Cruise in Minority Report) is make sure to evaluate the problem first. Make sure the interaction fits the needs. Again, the key point is to consider touch as part of an input eco-system, and not view it always as the sole method of device interaction. Not all input methods are equal.

This early thinking has led me to squarely declare that tap is not the new click, which is something I’ve heard thrown around, and anyone who believes so lacks an understanding and respect for how to approach different problems and searching for the best method of interaction between a user and a system.

Systematic approach of gesture integration

Most systems utilizing touch are purely touch based with no addition methods of interaction. This leads to touch being sequestered from other interactions, thus making it more of a user burden to learn. When a new behavior is introduced into a working knowledge system, it can be easier to absorb. In their recent laptops, Apple has taken an approach of incorporating touch into their behavior and input systems by using the track pad. In doing so they have managed to introduce and teach people touch and gesture behaviors in a method users already accept (the track pad). In addition, they are beginning to train people to move between input modes, from track pad mouse, to gesture, to keyboard, depending on the task. These types of associations allow for a better learning and input experience. On the flip side, the gesture actions are secondary to the main system, so they can be ignored fairly easily. It will be interesting to see if this makes gesture and touch easier to adopt, or if people will disregard it.

Top image by pinksherbet