I am in constant pursuit of the “clay” of interaction design (IxD). Even if that clay is intangible, if we are to consider ourselves a true design discipline there must be something that we are manipulating. Once we understand what it is that we are manipulating we will be better able to communicate to all our stakeholders the intentions of what it is the interaction designer designs. One possible property of said “clay” may be motion or movement.For almost all interactions we place our body in motion. Even speaking requires muscles to move in order to work. There has been a ton of work done on motion as an aesthetic quality towards an audience, even if that audience is just perceived. What I’m interested in is motion as an aesthetic regardless of perceived or real audience. The question I ask is if certain movements just feel better than others at an aesthetic level and further that perception is manipulated by other interacting factors.

The Foundations: A Recap

A couple of years ago I started positing that there are foundational elements to IxD. If we are to discuss material and medium in IxD there must be properties that we can use to describe and differentiate and even qualify what it is.

Why I pursue foundations as a concept is strongly influenced by Roweena Reed Kostellow (founder of the Pratt Institute’s Industrial Design Department) and her six foundations for three-dimensional design: line, luminance & color, space, volume, negative space, and texture. It isn’t just that these foundations exist for their own sake. They are a basis for two important requirements for the education and practice of design—educating craft and a basis for criticism.

Another growing influence is the work of Bill Verplank. His three areas of concern for the interaction designer are articulated beautifully in his video taped lecture he gives in Designing Interactions for his former colleague Bill Moggridge. In it he suggests that the Interaction Designer is concerned with three things that all start with “How do you …”. It’s worth the watch.

What is unclear to me from Bill’s explanation is how do I manipulate things to achieve the outcomes that he describes. Further it seems that he is only discussing the end result or point of interface that people interact with. This did not seem to map against my idea of what interaction design is. For me interaction design supports the interface by defining both the desired behavior of a product or service and the desired behavior of the people who will interact with that system.

So, with this in mind I’ve been working out a collection of foundations that I believe make up the “clay” of how to do just that. I have three original elements: Time, Metaphor, and Abstraction.

• Time is in many ways the most multi-facetted of the three. It breaks down into the following attributes: pacing, rest, duration, frequency, attention. These properties all combine to create a relative sense of time amongst people using the system, the same way that one experiences anything.
• Metaphor is related to what Richard Buchanan calls the “Poetics of Design”. It is the way we need to use analogy as the bridge between the intangible complexities that are forged through digital technologies (and other complex intangible and abstracted systems such as services) and the tangible world where our senses and cognitive abilities evolved to embody.
• Abstraction is really a value property. It relates to combined physical and cognitive activities that takes place to initiate an activity and when it is perceived to have been occurred.

The rest of this article though is about a new type of foundation that I alluded to when I presented at Interaction 09 on motion or movement.

Background on Motion

We are using a larger variety of motions with our primary computing devices than ever before. The devices are in motion like when we shake an iPhone to initiate an undo, or we are in motion & our devices can sense the movements we make. The previous tap which mapped almost exclusively to a mouse-click has been extended with new gestures like pinch, flick and swipe. Like the ubiquitous mouse-click there are a variety of contexts where these gestures are used changing their meaning, and emotional contexts. Mouse down, move, mouse up is commonly called “drag & drop”. How we combine movements within specific contexts can effect how we interpret their interpersonal meaning and the feelings we have associated with them.

One aspect of motion and movement comes from dance and martial arts. I love to dance and I used to practice both Tae Kwon Do and capoeira (two fairly different martial arts). Dance and martial arts requires a practitioner to be fairly aware of how they move in the world. Yes, you can say this is about balance and agility, but it is also about understanding what brings about balance and agility. It also forces you to understand your place in the world physically compared to everything around you. To me, this spatial awareness is to motion the equivalent that attention is to time.

spatial awareness is to motion what attention is to time

I spent more focused attention on my practice of capoeira as an adult. In doing so I realized quickly that how I felt emotionally doing a movement directly correlated to whether or not the movement itself was successful. On watching capoeira I noticed similarly as an audience member that beauty occurred within the success of those playing (you play capoeira instead of fighting it because of its history as a covert mechanism to learn how to defend yourself within the context of being a slave in Brazil.)

Compare the act of moving a file from one container to another with the act of panning a map. In this example the motion is almost the same but there is a clear difference that effects the aesthetic quality. The level of precision required for panning a map is substantially less than that of file-folder management depending on the level of graphic resolution and other factors related to Fitt’s Law. The motion of panning can in fact have a comparable flick like quality to it, especially when the user knows they are several lengths of motion away from their desired target. Targets themselves are usually approximations as well. Applying Fitt’s Law to this activity, an approximate target has a cognitive equivalent of just being a fairly larger target than an absolute target.

The Case for both good & bad motion design: Twitter for iOS

What got me to return to thinking about motion almost two years later was my own impressions using the newly released Twitter for iPad app and comparing those to my other iPad and iPhone apps I use. Specifically, there are new gestures introduced by the designer of both Tweetie for iPhone (now Twitter for iPhone) and Twitter for iPad, Loren Brichter formerly of Atebits.

On Tweetie, Loren brought to the iPhone world a whole new gesture. Playing on the existing metaphors of gravity & friction in other iPhone gestural interfaces, he used the existing playful springiness at the end of a list as a spring-loaded trigger to call for a refresh of the results of that same list.

The now-standard spring refresh on iPhone

This first gestural innovation was so successful that a host of other applications have taken it on as their primary means of refreshing a result list. For me the adoption of the new gesture so permeated my standard use of my iPhone that I now expect this gesture to be available in every app that I use. That is a pretty successful independent major UI paradigm to design.

When I opened up the new Twitter for iPad app, I was ready for some goodness because of all the hype I read before I downloaded. It is very well designed and is completely different from its iPhone sister. It takes advantage of the unique properties of the iPad. (For those not familiar with the app, the motions are all shown below).

The new iPad app puts the details of a single tweet in a right column, but instead of putting an “X” icon or other “button” to close or collapse the detail view, Loren invented a new gesture/action combo where the user swipes (a common gesture for deletion) to literally push aside the right column, which disappears for portrait view and squeezes it and clips it in landscape view. In so doing he both creates a new motion gesture and uses that new gesture as a means of reducing abstraction through what appears to be a tangible equivalent of pushing aside a pile of paper on your desk. (Yes, it is also an abstract metaphor and also has attributes of time associated with it.)

Clipping Columns

Understanding Aesthetics of Motion

Using all these apps I began to get new critical thinking that I could apply to the foundations I mentioned above. Whether it is the original flick-scroll that Apple designed with the launch of the iPhone, or the spring-refresh, or the swipe-dismiss there is a commonality for how the gestures are engaged. The movements share a lack of control and/or precision. This has as much to do with the size of the targets as it does with the complete lack of target for ending. These free-ending gestures work because of their ease, but also because of the extended range of motion creates an aesthetic quality to them that more precise and controlled gestures do not. In turn they add to the overall aesthetic quality of the interface around feelings of play & personal satisfaction.

I’ve also noticed some other key areas when using my iPad that have, compared to my iPhone, triggered similar emotional responses due to gestural differences. In general, scale of motion adds a lot aesthetically. As in dancing, extensions are just more beautiful.

The area that I find really different is in typing and general tapping. When I compare the typing experience on my iPhone to my iPad I notice the difference greatly. To really feel it open an iPhone app that requires data entry. Normally though we type on an iPhone with the single finger peck or by thumbing. I’m a big thumber. Even when in the correct form factor (and I’m pretty good at thumbing on my iPhone) the feeling of being more constrained & swaddled is there when compared to the openness & bounce you feel when typing on an iPad.

I’ve equated this feeling to the scene in Star Trek Generations when Data, with his new emotion chip, is singing while tapping away on his glass console screen. I’ve felt this so strongly that I’ve even been searching for a Star Trek console wall paper. I’m also constantly singing Data’s refrain when using my iPad, “Life forms. Tiny little life forms. Where are you? Da da da Da!” (Star Trek: Generations. 1994).

The added scale of space allows one to almost feel like they are dancing with their two hands on a glass dance floor.

When looking at any system of evaluation it isn’t only important to look at what works, but also understand what doesn’t work. My example here also comes from the iPad Twitter app. It has 2 other gestures that are applied to new outcomes. Both are related to revealing something in a new context without any visual cues that it is there. Like the swipe to reveal actions in the iPhone app.

The first of these is is two-finger gesture. With two fingers target a touch holding it down and swipe down. If there is a conversation related to the targeted tweet then it will reveal itself.

iPad Twitter Replies—two finger swipe

The other one also requires two fingers. It uses the the reverse pinch to reveal the detailed view of the tweet.

Pinch Open

Without going into why we need these gestures (I kinda feel they are “easter eggs” more than really usable functionality), they both have properties that lead to their lower performance or evaluation.

First, because they are two-fingered gestures it is less likely that a person will discover these behaviors accidentally. People do not use two fingers regularly accept in specific contexts that are well understood like zooming. For example, I was recently struggling to figure out how to scroll an inset frame without scrolling the surrounding container. It never occurred to me that I should use 2 fingers to scroll. When hearing that, I thought, “that’s messed up” and I tested it on 5 avid iPad users who all failed to figure it out as well and all complained that they were having the same problem.

The second problem is more about the reverse pinch activity then it is about the downward two-fingered swipe. With the reverse pinch the amount of fidelity required to do it is just too high. While the ending point is unimportant there is something about how to start the gesture that might require more precision and higher resolution than the system can handle consistently. For the two-finger swipe down to reveal the conversation, the difficult part is that you need to remember to keep your fingers on the glass or it will disappear. This leads to the constant repetition of the task lowering its utility. It is just easier to tap once on it and have it reveal itself that way.

How to Design for Motion

So what does all this mean for me?

First it means there is a huge opportunity. Loren made a huge name for himself as an accomplished iPhone designer/developer by innovating a new gestural paradigm. It catapulted his app into the limelight and eventually got him “acquired,” in this case by Twitter itself.

I don’t know what methods Loren used to come up with his spring-refresh design, but I can look at the work of Kicker Studio and their case study they published for the gestural TV remote control they designed. What is clear is that sketching & prototyping now requires a new methodology. We all need to learn to become solid actors if we are going to design interfaces that require the user to move in new ways outside of buttons, pointing devices & keyboards. When it comes to mobile devices and touch screens especially, we need to all become actors.

I’m reminded of the case study that Bill Buxton wrote about in his amazing book Sketching User Experiences on how the Palm Pilot was designed. They used a block of wood & a cut off pencil and played with various forms & felt how various gestures would play out. Binging our prototypes into the physical is going to be key as we design for mobile gestural platforms. We are going to have to act out scenarios of use, dance out gestures to complete new choreographies. We need to see gestures both as dancer and as audience.

One of the reasons these gestures work is also related to the visual cues for all the states of availability, direction, activity and completion. Rehearsing the gestures in front of others will cause people to ask questions like how do you know it will do something? And how do you know it when it is complete?

Gestural interface design is still very new. We can deeply appreciate the work of Apple , Microsoft, and Google in their leadership efforts but there are still lots of opportunities in this area to innovate even more. Having an understanding of all four of the foundational elements of interaction design will help you design more solid interfaces & interactions for better overall experiences.

Concluding Thoughts

I am cautious about adding this as a foundation of interaction design because it feels like it might fit within the context of “interactive design” or “interface design”. For now though I believe that there is a behavioral property that moves beyond the point of interaction itself towards embedding behaviors within human beings that become embedded culturally. The motions themselves then become akin to affordances of there own even though they do not connect to any visually perceived markers. They just become expected on one hand and they imbue an emotional aesthetic all their own.

References:
Foundations of Interaction Design article on Boxes & Arrows, and related podcast,
Revised article on Johnny Holland
Interaction09 Motion and Movement video and slides

The conclusion of the Nielsen Norman Group’s April 2010 study of iPad usability is that it has problems and more standards are the solution. Yes, the iPad is imperfect, but resorting to standards as the solution is an antiquated reaction that fails to consider how interactive systems have evolved. We’re not Usability Engineers anymore (not most of us, anyway); we’re User Experience Designers. Experience is more than just usability.

I’ve covered this ground on Johnny Holland before. Just after I got my iPhone I came to many of the same conclusions Nielsen did about the how the iPhone is difficult to learn. But here’s the thing; I didn’t stop there. I talked about how some of the factors that made the iPhone difficult to use also made it fun to use, which is why it has flown off shelves since it was introduced. As I got used to it I began to think more about how playfulness was more delightful than pure usability in some contexts and vice versa. Something I use occasionally for very specific tasks delights me if it is simple and usable. But something I use often or for more amorphous tasks that is simply usable will either provoke no emotional response or, at worst, will become tedious. In that context, a more playful interaction style will keep me engaged and might even lift my mood a little.

This is the perspective from which I’ll look at what Nielsen found, identify where it’s valuable, and point out where it’s a little myopic.

“Wacky Interfaces”

Wacky. Yes, “wacky.” As in, “Isn’t it cute how kids these days are trying to create beautiful experiences.” Beauty does not require an unusable interface, but a beautiful experience might ask you to engage with it a little more deeply through a lack of obvious affordances.

For more than a decade, when we ask users for their first impression of (desktop) websites, the most frequently-used word has been “busy.” In contrast, the first impression of many iPad apps is “beautiful.” The change to a more soothing user experience is certainly welcome, especially for a device that may turn out to be more of a leisure computer than a business computer. Still, beauty shouldn’t come at the cost of being able to actually use the apps to derive real benefits from their features and content.

He almost gets it. No, the iPad is no business computer, and that’s exactly why beauty is an asset. People will, much of the time, interact with this device in order to have an experience rather than complete a task. Nielsen’s wholesale discounting of beauty fails to take into account that some apps will be experiential and content based while some will be functional and task based. Engaging with a system is not what people want to do when they have a task to complete. That’s when basic usability is more delightful.

Long-standing GUI design guidelines for desktop user designs dictate that buttons look raised (and thus pressable) and that scrollbars and other interactive elements are visually distinct from the content.

The iPad does not have a Graphical User Interface but a gestural one. GUI design guidelines do not necessarily apply when users can interact directly with the content.

For the last 15 years of Web usability research, the main problems have been that users don’t know where to go or which option to choose — not that they don’t even know which options exist. With iPad UIs, we’re back to this square one.

The iPad is also not the Web. Interacting with apps is completely different from interacting with websites. Most apps have far fewer options than the average website, lessening the potential for confusion. On top of that, people use apps in a much more focused way than they use a website. Users can access the entire Web when they open their browser, but when they open an app they choose to focus on that app’s content and functionality only. In that context, a more deeply engaging, exploratory design can enhance the user’s experience.

“Inconsistent Interaction Design”

I take issue with this finding because Nielsen evaluated multiple applications. That’s like saying it’s bad that Microsoft Word and Adobe Photoshop are inconsistent. They allow completely different audiences to accomplish completely different tasks. He considers it confusing that the same gesture affects the same type of content differently in different apps. When there’s a limited gestural vocabulary (and there has to be) and a diversity of contexts, it’s easy and usually risk-free to experiment with figuring out the correct gesture if you get it wrong the first time. And because it’s gestural, it’s inherently playful and fun. It’s not a chore like trying to parse Word’s menus or toolbars.

Nielsen says that iPad UIs suffer from the “triple threat” of low discoverability (non-obvious controls), low memorability (difficult to remember inconsistently applied gestures), and accidental activation. I agree with the first and the last, mostly. Non-obvious controls can encourage exploration and playfulness in some contexts, but they can be frustrating in others. Accidental activation is certainly annoying, but it’s usually easy to deactivate whatever was activated. That problem in particular I think is due to the absolute newness of the apps and the platform. At least two iPad developers I’ve heard from indicated that they changed the design of their apps once the iPad was released.

The second problem he identifies, low memorability, I completely disagree with. My pre-literate two-year-old daughter knows how to unlock my iPhone & iPad, navigate to her favorite drawing app, launch it, draw with it, and change the various options. It took very few demonstrations before she learned this. If you look on YouTube there are videos of small children expertly navigating iPhones and iPads. You show them how to do it once, they do it, and they remember it.

The link between physical motion and cognitive development (especially in children) is well established in cognitive research, making gestural UIs much more easy to remember than your typical desktop GUI. On top of that, the number of gestures that are possible is pretty limited. Even if you don’t perform the correct gesture first, it won’t take long to figure out what the right one is.

“Crushing Print Metaphor”

Nielsen again complains that iPad apps are not like the Web.

The current design strategy of iPad apps definitely aims to create more immersive experiences, in the hope of inspiring deeper attachments to individual information sources. This cuts against the lesson of the Web, where diversity is strength and no site can hope to capture users’ sole attention.

My friend Pete Barry likes to talk about the value of experience. The reason people choose to consume content through these “limited” apps is because the experience they provide is valuable to them in some way. That experience is a benefit rather than a drawback. Besides, the open Web is just two taps away.

“Card Sharks vs. Holy Scrollers”

Nielsen references Jef Raskin’s differentiation between “two fundamentally different hypertext models,” Cards and Scrolls, indicating that iPad apps mostly fall into the Card model. On a Card, all the interaction occurs on a fixed size canvas that is swapped out to provide access to more content or functionality. And I’m sure I don’t have to tell you what a Scroll is.

Nielsen said:

There’s no real reason we can’t have both design models: cards on the iPad and scrolls on the desktop (and phones somewhere in the middle). But it’s also possible that we’ll see more convergence and that the Web’s interaction style will prove so powerful that users will demand it on the iPad as well.

If I read that right, I actually agree with him. The iPad doesn’t have to force all apps to subscribe to one model; each app can use whichever model is most appropriate for its context of use. I’ve even seen some apps that mix the models, like Early Edition. This newsreader arranges RSS feed articles like a newspaper, with a home page and different pages for each individual feed. Wherever an article appears on any of these pages, you can actually scroll in place to get a sense of what it’s about! Granted, this is something users are likely to discover accidentally, but it’s a pleasing, delightful interaction nonetheless.

Nielsen’s Recommendations

This is what really gets me going. And not in a good way. He has four, but they really roll up into three:

• Make iPad UIs look more like GUIs
• Make iPad interaction design more like the Web
• “Abandon the hope of value-add through weirdness.”

And yes, that third is a direct quote. In 2010. Beauty isn’t weird. Compelling interactions aren’t weird. Both of these are critical components of modern interaction design, where designers seek to go beyond simple usability and create positive emotional experiences that build loyalty and emotional attachment. What is perhaps most confusing about these recommendations, though, are the first two. Jakob Nielsen is a smart guy, and clearly the iPad exists within entirely different contexts of use than a desktop GUI or a website.

What I really want to know is this: why does Nielsen feel that iPad apps should be designed for contexts they won’t be used in?

—–

Header image by nDevil /CC 2.0

Portable computing, including tablet PCs,  has been the hot topic since Apple’s unveiling of the iPad in January. While the iPad has gained a lot of press and generated conversation about its intended audience and potential use, other tablet devices have been overlooked. Microsoft’s Courier is one such device. Courier—the soon-to-be production version of the Codex, a rumored tablet device widely discussed in 2009—provides a tablet platform for the student, techie, and slide-show-loving grandparent alike.

Apple's iPad

So why should this matter when the iPad has made it clear they intend to be the everyman’s casual Internet device? In reading the commentary on the device there is much being said about who the device is not for. It is not for the techie. It is barely for the student. The device’s lacks the processing power or input devices for programming or graphics editing. No cameras or stylus limits the device’s use as a social and note-taking tool. The iPad is intended for the non-techie to surf the internet, read eBooks, and share basic media with friends and family.

This is where Courier steps in. Rumored to have built in camera(s) and stylus input, the device features two seven inch touchscreen monitors, making it the techie and student’s dream. Connected through a WiFi or possibly 3G network, the device can be used to coordinate chats multiple devices. The dual screen offers opportunities to have a personal and shared workspace during a chat, during which documents can be transferred from one screen to another for collaborative work. With its stylus, it’s possible to take notes with simple handwriting recognition and to track which user is performing what action on a screen.

The Courier is not the cure all, though. In their demo video, Microsoft leans too heavily on physical metaphors. Needing to flip over a picture to view the notes is a cumbersome and unnecessarily literal translation from the physical world. Tack on excessive animations and the device becomes more of a showpiece than a utility. This is, of course, a demo  to show the device’s potential—but, like Apple with the iPad, I think Microsoft is selling itself short on the device’s potential.

Imagine having a Courier or iPad next to the door in every room in your home. Connect it through the WiFi to the HVAC, lighting, and entertainment systems and you have an integrated smart home. Want to change the lighting or turn the heat on? Dont get up. Forgot to preheat the oven? Set the temperature from your living room without having to pause the YouTube video you’re watching.

As mobile devices become more powerful, a shift is occurring as manufacturers look at how they can apply this mobile technology in our everyday home-life. The iPad and Courier are just two devices that show the possibilities of a computer not limited to the desk or lap—a mobile device not constrained by our mental models of phones, PDAs, or portable devices.

Header image via gizmodo