Click here to download:

Research_review_NUI_cfriedlander_v1_2.pdf (19.95 MB)

The Natural User Interface | Gestures, Forgivability, Learnability

A review of current research and journal articles looking at the Natural User Interface and its learnability / usability across multiple demographics.

Cameron Friedlander

June 3rd. 2010

Preface

I called over to my 19-month old son the other day and asked him how big do elephants grow, he stretched his arms out as wide as they could go. I followed this up by having him size up an ant, he brought his hands together and made a tiny gesture as to the size. His first interaction with his environment, was with his hands. He would point to any item he saw fit and gesture until I told him the name. This was well before he started developing any sort of vocabulary. These gestures seemed to be rooted in his learning. The more things I pointed out the more he began to interact with them. It came down to his hands, his main communication platform before his vocabulary started to kick in. The gesture.

Introduction

When we look at learnability, in HCI, across multiple user interfaces we see a progression. The computer desktop interface (GUI) is far better than the command line (CLI) for the average demographic, but yet it is still built upon these old ideas, principles, and values for when the computer first arrived on the market. This desktop metaphor, as it is frequently called, was there to help usher in generations of new users who were not familiar with this new computing device, the personal computer; a single input entry, the mouse, backed up by design patterns and metaphors to help us feel safe. However as the average user demographic and familiarity change, in a world surrounded by information and computers, there has been a want, a need, to be able to interact with the information differently, naturally. This has ushered in the trend of the Natural User Interface, or the NUI. This conversation is being forced to the table through the sales of the Apple iPad, which is by no means the only NUI device on the market. But why are users adopting this device? Is there anything natural about the NUI? What are the critics saying about gestures? How are gestures integral to human behavior? Is this device naturally learnable and potentially easier to use?

Market Trends

The NUI / gestural interface is in its infancy stage, clearly, but it is apparent that the NUI / tablet is in our near future. If there is any doubt as to how the tablet and the NUI is being embraced by our society look no further than the sales of netbooks. Fortune magazine recently looked at the sales of netbooks from today to one year ago. A year ago netbook sales were on the rise by 641%, whereas now they are at 5%, a staggering decline over a one year period. The article looks at the influence of tablet and its effect over the market across desktops, laptops, and iPods. At CES, in 2008, Stephen Prentice Gartner came out with the following article; Gestural Computing: The End of the Mouse. He discusses the recent trends, at the time, and suggests to all businesses, companies to the do the following: start interacting with gestural devices immediately, whether it’s a Nintendo Wii or an iPhone; immediately suspend all skepticism of gestural computing; and lastly to “relax” this is going to be a “long-term trend.” He summarizes the movement as a paradigm shift, while it won’t outright replace the keyboard, it will eventually. He even looks at enterprise systems and states that the NUI could be integrated within 3-5 years, from 2008. In essence the NUI isn’t going anywhere. But why? Why NUI, gestures?

The Basics of Gestures

Dan Saffer, of Designing Gestural Interfaces, set out to create a textbook around gestures since there didn’t appear to be a single source of information on the topic. He used, as a comparison, the art of learning music; done through gestures of the hands. How difficult it would be to learn the cello if one could not use their hands to express the music and notes. To him “gestures create meaning” (Designing Gestural Interfaces, pg xv). However the gestures that are to be integrated into a user interface need to support the motion, mechanics, and limitations of the physical body. He suggests the study of the body for would-be-designers of a NUI to get a better understanding of these constraints and limitations.

The Microsoft Surface Table [2] was an early pioneer into the NUI front, and by no means flawless, but they did take into account careful consideration of the metaphors used to help create a learnable environment that lends itself to discoverability instead of instruction and rote learning. In Using Metaphors to Create a Natural User Interface for Microsoft Surface (2010) Hoffmeester and Wixon explore these metaphors and the usability testing they did to bring these to life. They looked at the idea of a magnet board; this board is known as ‘the hub’; a communication space / bulletin board. The next metaphor was ‘the circle’, this circle is used to literally define the physical space in which a user is able to move within the environment. They enhanced ‘the circle’ by bringing in a metaphor called ‘my personal moon’ this is defined by a users solar system of planets, circles, that users can manipulate contextually; similar items equals orbiting planets. Lastly they looked at ‘fireside chats’, how users can gather around a campfire and participate, tell stories, and explore thoughts. Together they created the magnet board, the circle, the personal moon, and fireside chat to bring their metaphors to life. They set out to design a new interface around these concepts and test them to see which ones would be brought to the surface. They settled on a combination of ‘the circle’, and ‘my personal moon’ to define the metaphor for the concept Sphere; with circles they could show a clear sense of hierarchy that users could recognize. They combined this Spherical functionality with magnets; so properties, applications, and contextual menus would stick to their related Spheres. So what are the basic gestural interactions to use these metaphors?

Saffer opens up his section on design patterns with the following quote (Chapter 3):

"It is the tactile sense that demands the greatest interplay of all the senses."

—Marshall McLuhan, interviewed in Playboy, March 1960 

Never underestimate the power of touch. While exploring my own user observation with the iPad I had a young adult who had never used a paint program, like Adobe Photoshop [3]. The end result was that she was able to learn the basic tools of this paint program easily and have fun doing it.. Because she was able to paint with her hands, there was something far more natural about this than using a mouse. Even as she explored the menu looking for an item she would find new functions as she did so. This brought multiple learning’s to this observation; not only was she teaching herself how to use it and having fun, she was learning new metaphors and design concepts for later recall. Saffer’s gestural design metaphors/patterns are basic but they highlight the crux, of the gestural interface. The tap; this pattern can be used in multiple ways and users tend to have little difficulty with its functionality switch, it’s about context. This basic gesture can be parlayed over to my own observations of my 19-month old son, one of his first interactive physical gestures, was to point. When I purchased an Apple iPad for him it was only natural that he wanted to tap it and was easily able to understand this concept. The tap is not only used to open / activate but is also used to select an item. The drag components appears to be the most basic but also the most natural, the tactile idea that one can touch something with their finger and move it. In the YouTube video, A 2.5 Year-Old Has A First Encounter with An iPad (Telstarlogistics, 2010), the young girl easily picks up on this within minutes of her introduction to the iPad (1:40 minute mark). The flicking gesture surprised me only in the sense that I never expected my son to learn this, perhaps accidently (forgivability), without being taught but by combining a quick version of select, drag, and release he noticed that items would fly across the screen; furthering the relationship between the physical world and the interactive one. Simply tapping the item again would halt it, naturally. More complex items that I have not introduced to him but are basic to the gestural devices is the pinch; to shrink, and the spread; to enlarge. While adults appear to not have an issue with this I have yet to test, or show, my own son this concept.

Human Gestures and Learning

How integral is the gesture to learning and the idea of learnability? The idea, I can only assume, is that gesturing helps people learn. What would happen if we weren’t allowed to use gestures? Would this hinder us? Help us? Cook, Mitchell, Goldin-Meadow, set out to answer that question (Gesturing Makes Learning Last, 2008). They knew that gesturing was a basic part of human behavior through adult observation, noting that even persons with no sight since birth, use their hands to tell stories. In a study focused on children they saw that kids who were allowed to gesture during the learning phase did significantly better than kids who were not allowed to. In later recalls the children who were allowed and encouraged to use gestures were able maintain their learning, 4 weeks later, 85% of the time, while kids who could not use gestures faired much worse, at 33%. The data suggested that gestures supported knowledge change in casual learning however they could not say how it did so, but this was not the point. The point simply was to see the impact, short-term and long-term, on learning with and without gestures.

Learnability within Extreme Demographics

However are gestures easily learnable, in the sense that there must be a common set, and or environment in-which a child grows up in order to know what to do? Does extending a person’s arms out mimic flying, is this culturally relevant on a global scale? In the paper “Children’s Intuitive Gestures in Vision Based Action Games” they set out to create a fully immersed 3D gestural environment in which a child could interact. They wanted to test how easily children could pick up on gestures to complete the game. They found that be using subtle animation to mimic the gestures they were looking for within the gaming environment that the children were able to inherently pick up on this without having to run through a tutorial. So if they needed to have a bird to fly, the act of flapping its wings directed the children in the right direction if they didn’t know how. The other fascinating result within this game testing environment was that the avatar the children were controlling didn’t always have to mimic the exact gestural movement, if the bird (QuiQui as it was called) needed to swim children opted to use different styles of swimming to do so, it made no different that the bird was actual swimming sideways on the screen as they ran through doggy-paddle, breast stroke, and crawl. The children sought forgivablity within the interface itself, as mentioned earlier. An item for further exploration would be the connection between learnability and a forgivable interface.

But what of the single-point and click of the traditional desktop, how do children fair using common gestures, such as drag-and-drop, with these traditional means? A spelling game, on the Apple iPad [1], encourages kids to take their finger and place it on a particular letter for a word and drag it over to the correct placement. In my own child observation with this game he was able to quickly figure out how to move the letters around, and toss letters across the screen be flicking, instead of dragging. In a study of kindergartners and first graders, by Afke Donker and Pieter Reitsma (Drag-and-drop errors in young children’s use of the mouse, 2005), they set out to determine why in traditional desktop children’s educational software the motion of “drag-and-drop” had been all but abandoned. The initial hypothesis was that it was too hard for a child to maintain pressure on the button of a mouse and move it over a particular distance, so perhaps they could design a device that would make this easier. However the research showed that the act of selection and dropping were the areas that had the least amount of forgiveness; much like a fork when a young child/toddler is eating, it is harder for them to stab a particular item than it is to pick it up outright. So while the initial hypotheses was disproved the problems persisted. The mouse, as a device, that can easily be digested and understood by children appears to be an issue, but what is a child could actually touch the item, as in the iPad game?

Looking at the opposite end of the spectrum how do the older generations fare with a NUI device, are they able to comprehend the design metaphors behind the gestures easily? In the observational YouTube video, Virginia's new iPad (Sackr, 2010),  a 100 year old begins to read books and write again using the iPad, the NUI doesn’t seem to be an issue at all. Does this learnable forgivable interface actually help her out? In 2006 a group of researchers / software engineers evaluated a group of elderly and teens using a new surface tool for photograph manipulation (Apted, Kay, Quigly, Tabletop Sharing of Digital Photographs for the Elderly).  They did the study in multiple parts, sometimes explaining the new interface or by giving them a set of one-sheet of instructions. The overall results highlighted that the teens completed the tasks quicker but that all of the elderly evaluated were able to complete the tasks with limited frustration. This new device was a custom solution, not a Microsoft Surface table, nor an Apple product, so there appeared to be certain flaws in some the new logic that they were creating. They did note that two-handed gestures appeared to be harder for the elderly to handle, but it was not disclosed whether this was a flaw of the design of the observation or an issue that persisted universally across all gestures. The main takeaway was that even with this somewhat flawed gestural devices the participants were all able to complete the task and recall it.

Critics and Flaws of the NUI

Does the NUI device have a natural learnability? A common complaint from the Nielsen Norman Group (NNG) is the lack of visual cues and menus (Norman and Neilson, Gestural Interfaces a Step Backwards in Usability). However a recent posting sensation on YouTube, approaching 1 million views, showcases a 2 ½ year old using a tablet for the first time. It can be said that she appears to master the basic gestures of the interface as well as various applications within the first 5 minutes (A 2.5 Year-Old Has A First Encounter with An iPad, Telstarlogistics). As observed with my 19 month old who, with no instruction, was able to master drag, swipe, and flick. With a tad bit of further instruction he has gotten the hang of basic drag and drop functionality. There are issues and inherent flaws within the Apple iPad NUI design and it’s no wonder, because it’s so young. Unlike the Microsoft Surface table that allows multiple touch points the Apple products only allow one action / touch at a time. Having observed with my own son, as well as the YouTube video (A 2.5 Year-Old Has A First Encounter with An iPad, Telstarlogistics ) that kids like to ideally rest one hand on the screen as they proceed to do an action / gesture with the other hand. This flaw can sometimes lead to frustration on the child’s part as they struggle to comprehend why an action that used to work no longer does, note that within a week he was resting the tablet on the ground to limit this unwanted feedback.

So why is NNG so against the gestural interface, with its learnability, ageless adoption, and trending market? Nielsen’s take appears to be more focused on the flaws of the iPad vs. a gesturing system. In iPad Usability: First Findings From User Testing (May 2010) he looks at the inconsistent design features on various applications; pressing a photo on one app might do different things vs. another. Can we not say the same is true for the webspace metaphors as well? Nielsen went into more details about how the tab bar at the bottom of the screen will, of course, be blocked by your arm when using the iPad. Saffer even states within Designing Gestural Interfaces that this is an obvious no-no for contextually related menu items; this is not the functionality of the iPad tab bar however. Nielsen also lists out other flaws such as its forgivability, various gestures can be done in multiple ways to complete the same action, or lead to unintended feedback. As mentioned in Children’s Intuitive Gestures in Vision Based Action Games a child could swim but there are many ways to actually swim, can this be a benefit; to make it easily adoptable to younger and older demographics, by not having to do things exactly one way (with no forgiveness or strict constraints) enhance its learnability? When observing my son I noticed that his swiping gesture would take on different manifestations as he processed it; long hard swipe or a tap, flick in a particular direction, but it always had the same result; success. As my son became better at learning the gestures he would hit it more dead on but by experimenting and learning, through success, he never got to a moment of complete frustration, so much so that he would stop using it. This is the crux of learnability within a gestural environment. Teaching an elderly adult to use a desktop interface is quite chore, to get them to come back to it is almost an impossible feat. Yet here is Virginia, 100 years old, writing limericks and reading books on her iPad. What’s even more interesting is that Nielsen doesn’t even explore the positive aspects of the device, he merely dismisses it outright due to its flawed architecture. Donald Norman goes so far as to identify a crisis within the HCI community and generalizes towards all gestural interfaces. Why is this?

Norman’s take in Natural User Interfaces are not Natural is more high-level. He makes various comparisons to instances where the gestures do not follow nature directly. In one example he highlights the Nintendo Wii bowling game, with the often, natural, side effect of releasing the controller, as if it were a ball, when playing. He states that although he sees a place for gestures it is neither better nor worse than the keyboard and mouse. His argument is that they are both unnatural, therefore the gestures should follow the convention of the keyboard and mouse. Really? Is the gesture not part of how we learn, as discussed in Gesturing Makes Learning Last by Cook, Mitchell, Goldin-Meadow? Where exactly does the mouse fit in to all of this, what is the comparable metaphor? Norman does have respect for the gestures but in a later article, to be published in Interactions Magazine he gets bolder: “Gestural Interfaces: A Step Backwards in Usability.” He starts this article off with the following words “One step forward, two steps back.” The implication is that we are now worse off with the trending of gesturing devices, on May 28^th, 2010, at around 12pm central @NNgroup started tweeting this. His concerns are as follows, again attacking gestures through the usability of iPad, as if this is the sole gesturing device (see Nielsen’s iPad Usability: First Findings From User Testing): No set of established guidelines, he then calls out Apple and Google for ignoring these established conventions, and he attacks the developer community of these gestural applications for ignoring the said rules of conventional HCI. Is there not an inherent conflict of interest here? Can we, should we listen to the calls of Neilson and Norman seeing as to how invested they are in the GUI? Is the NUI not innovative and emerging? Where is the root of innovation? Is it in the universal design principle?

In a 2009 BBC interview (Cellan-Jones , Rory. “Listening to Mr. iPhone.”) with the iPhone designer, Jonathan Ive, he stated that they are solely about the product and innovation. If you spend too much time defining everything you end up with nothing. “If you have to spend time institutionalizing that, talking about it, you end up chasing your tail.” How do they maintain to be so customer centric with their devices? “’We don't do focus groups,’ he said firmly, explaining that they resulted in bland products designed not to offend anyone.” The article itself quotes Henry Ford who said that if he asked people what they would want they would have said “a faster horse.” In essence the iPad is in its infancy, it’s a child we will look back on and learn from. NNG should not be bashing the early pioneer into this foray of new interface design. It’s not as if Apple is without rules when it comes to the NUI. In fact they are very protective of it, so much so that they are refusing to put Adobe Flash into their devices because it lead to usability chaos. A gesture that means one thing can be overwritten to mean another, Steve Jobs talks to this in his open letter to Adobe (Thoughts on Flash, 2010), where he discusses how the gestures and meanings behind them can be over-written within the Flash framework, creating a bad user experience. It’s a start.

The Future of NUI

So what does the future of gestural computing look like? We can understand the learnability of it across multiple demographics and in various instances from education, to gaming, to browsing, reading and other everyday tasks. Sales of these devices are increasing tremendously, putting a halt to other devices such as the netbook; a miniature desktop PC utilizing the existing point-click desktop metaphor that seems even more flawed on a tiny device. There are clearly some forefathers of usability that take issue with this new interface, but at the same time there does seem to be a respect for it. Recent technology developments in 3D, particularly Augmented Reality may in fact bring on the next revolution or paradigm shift in the interface design. Augmented reality (AR) allows users to, literally augment, their physical reality in various ways using a device such as a phone or webcam. Either by manipulating elements that do not exist within their computer webcam or by helping users navigate their reality through a phone by receiving contextual information through GPS and their video player to display it. Petersen, Nils, and Stricker, Didier, in Continuous Natural User Interface: Reducing the Gap Between Real and Digital World, look to bring gestures and augmented reality together to create the next generation of interfaces, the continuous natural user interface (CNUI). The issue with augmented reality is that the user needs to break the paradigm in order to interact with reality vs. AR; using real-world gestures within an augmented reality environment one would need gloves and other devices. Their proposed experimental system takes real-world artifacts, including gestures, and is able to bring them into the AR experience. Essentially if you can take items from the physical space and bring them into the AR space without stepping out of context, and switching paradigms, from NUI to desktop to scan a document; we would be one step closer to a literal augmented reality. So by employing all the same natural gestures of the physical-world and allowing items to be scanned on the fly through what is known as “The Hub” they are able to transfer elements from one space to the next, as a user you are able to manipulate them the same way as you would, say a piece of paper in the real-world. Henderson and Feiner look to do a similar things within the AR space (Tabletop Opportunistic controls: leveraging natural affordances as tangible user interfaces for augmented reality, 2010) by focusing on similarities with how we manipulate real-world objects that will be augmented with contextual information. In this particular study they looked at motors, that have various needs and affordances that need to pay their way into the AR space. Item examples would be dials, knobs, screws and such that the user would be able to manipulate within the virtual AR space and receive contextual aid information as they do so. They call these opportunistic controls (OC) and look to use them across multiple verticals and categories. So as the NUI begins its journey to becoming the dominant interface people are already thinking about the next future step of such an interface, augmented reality. It is not hard to imagine a future where we travel through the physical space with augmented support, using both literal gestures to manipulate the physical space, backed up by virtual gestures (NUI) to manipulate objects, information, or affordances in the virtual space (AR). The future is coming Nielsen and Norman; are you with us?

Bibliography

[1] FirstWords Animals <http://itunes.apple.com/us/app/firstwords-animals/id294536447?mt=8 >

[2] Microsoft Surface <http://www.microsoft.com/surface>

[3] Learnability and the NUI / iPad <http://posterous.nullintovoid.com/user-observation-learnability-and-the-nui-ipa>

Petersen, Nils, and Stricker, Didier. “Continuous Natural User Interface: Reducing the Gap between Real and Digital World.” Proceedings of the 2009 8th IEEE International Symposium on Mixed and Augmented Reality. Oct 2009

Cook, Susan Wagner, and Mitchell, Zachary, and Goldin-Meadow, Susan, “Gesturing Makes Learning Last.” University of Rochester, Department of Brain and Cognitive Sciences. Cognition, Vol 106(2), Feb, 2008. pp. 1047-1058.

Norman, Donald A. “The Way I See It: Natural User Interfaces Are Not Natural.” Interactions. May 2010

Apted, Trent, and Kay, Judy, and Quigley, Aaron. “Tabletop Sharing of Digital Photographs for the Elderly.” Proceedings of the SIGCHI conference on Human Factors in computing systems. Apr 2006

Henderson, Steve J., and Feiner, Steven. “Tabletop Opportunistic controls: leveraging natural affordances as tangible user interfaces for augmented reality.” Proceedings of the 2008 ACM symposium on Virtual reality software and technology. Oct 2008

Höysniemi, Johanna, and Hämäläinen, Perttu, and Turkki, Laura, and Rouvi, Teppo. “Children’s intuitive gestures in vision-based action games." Communications of the ACM Volume 48, Issue 1 (January 2005) Pages: 44 - 50.

Hofmeester, Kay, and Wixon, Dennis. “Using metaphors to create a natural user interface for Microsoft Surface.” Proceedings of the 28th of the international conference extended abstracts on Human factors in computing systems. April 2010

Saffer, Dan.  Designing Gestural Interfaces. Sebastopol, CA. O’Reilly Media, Inc., 2009.

Prentice, Stephen. “Gestural Computing: The End of the Mouse." Gartner Feb. 2008

Donker, Afke, and Reitsma, Pieter. “Drag-and-drop errors in young children’s use of the mouse.” PI Research. 26 July 2005

Cellan-Jones , Rory. “Listening to Mr. iPhone.” BBC News. 1 July 2009. 31 May 2010 <http://www.bbc.co.uk/blogs/technology/2009/07/listening_to_mr_iphone.html>

Nielsen, Jakob. “iPad Usability: First Findings From User Testing.” Useit.com Alertbox, 10 May 2010. 20 May 2010 <http://www.useit.com/alertbox/ipad.html>

Norman, Donald, and Nielsen, Jakob. “GESTURAL INTERFACES: A STEP BACKWARDS IN USABILITY” Nielsen Norman Group. 30 May 2010 <http://jnd.org/dn.mss/gestural_interfaces_a_step_backwards_in_usability_1.html>

Jobs, Steve. “Thoughts on Flash” Apple. April 2010. 31 May 2010 < http://www.apple.com/hotnews/thoughts-on-flash/>

Beecher, Fred. “Usabilility Ain’t Everything – A Response to Jakob Nielsen’s iPad Usability Study” It’s All About Interaction. 26 May 2010, 30 May 2010 <http://johnnyholland.org/2010/05/26/usability-ain%E2%80%99t-everything-a-resp...>

Buckleitner, Warren “Is the iPad a Child’s Best Friend” The New York Times: Gadgetwise. 30 Jan 2010. 30 May 2010 <http://gadgetwise.blogs.nytimes.com/2010/01/30/is-the-ipad-a-kids-best-friend/>

Elmer-DeWitt, Philip “How the iPad gobbles up netbook sales” CNN Money: Fortune Magazine. 6 May 2010. 31 May 2010 <http://tech.fortune.cnn.com/2010/05/06/how-the-ipad-gobbles-up-netbook-sales/>

"Virginia's new iPad." Video. YouTube. Sackr. 7 April 2010. 30 May 2010 <http://www.youtube.com/watch?v=ndkIP7ec3O8>.

"A 2.5 Year-Old Has a First Encounter with An iPad." Video. YouTube. telstarlogistics. 5 April 2010. 30 May 2010 <http://www.youtube.com/watch?v=pT4EbM7dCMs>.

Awhile back, perhaps a long time ago in the digital age I used to run a BBS. I used the Telegard BBS software and ran it with a 9600 baud modem, rocking a 386sx (the sx was for slow xtreme) cpu.

Doing some research for command line game mechanics I was trying to recall some of the earlier games that one would play on the boards; Yankee Trader and such. These game mechanics tend to be the basis for the popular Mafia Wars and Farmville that we see today trending within the social communities; some things never change perhaps.

Interestingly enough a quick search of "Telegard BBS" within Google landed me in a frozen moment in time. Most of these sites were last updated in 1999. Some of them claiming, soon to be released, y2k updates (oh yes remember this? Cobol guys are still lamenting the end of the gold rush I suppose). There was / is something almost eerily depressing about it all. The Telegard main homepage, last updated on Christmas of 1999, talks about the newly redesigned website and how the creator, Tim Strike, will be releasing new software shortly. Yet there it is, hasn't been touched since and a historical Wikipedia article probably usurped its top ranking on Google a few years back.

Again I trolled Google trying to remember the name of a maintenance RPG that I had installed on my BBS (Which BTW was called Prophets of Rage; a friend of mine tried to convince me to rename it to Profits of Wage at the time. I didn't see the humor/cleverness in this until many years later). So I entered in the search terms "Telegard BBS games cyborg"; I was thinking the game had "cy" or "sy" in it, just can't remember...still don't. The results came back, so many phreakers and sysops named cyborg brought up usenet entries for random BBS's; their names, phone numbers, endless chatter. Large elite BBS phone lists came back promoting Lunatic Fringe and Yankee Clipper, ones that I personally used to belong to (zero day warez people). All frozen moments, seemingly petrifying right before the new millennium; we can only glance in and see the artifacts left over. Perhaps a search or some field work within Google will bring to the surface more of these ruins.

So I ask, somewhat lamely I admit, will we soon have a major for digital archeologists? In theory I guess we have already experienced a bit of this with data. The Rosetta Stone is still able to be translated yet a floppy and or punch card from the 1900's is perhaps almost lost, as the media erodes over time. But what about all this information, content now being liquid will soak into the tubes (cheers to Alaska) and get stuck in the corners like the fat within the arteries of a middle aged American. Who will explore this? Scrape it out and paste it on a wall as artwork? 

"Look at this late 1900's artifact from the early Arpanet period. So beautiful." She said.

Fire Escape's 314 Area Code BBS Directory - July 1995

Well I had to get your attention somehow, didn't I?

Has the mobile device killed traditional branding? Arguably, for the time being, it has. Currently brands and agencies are trying so hard to get iPhone applications out, mobile sites, blackberry apps, that what has fallen to the wayside of late is the initial overall object; branding. This of course isn't done on purpose but it is almost as if the technologies / platforms that are used to deliver the content are speaking more to the brand than the brand itself. If a company wants to suggest they are "cutting edge" (yeah well...I used it, f off) or "innovative" (christ...can I get a shout out to "leverage" while I’m at it) they have more of a shot by using a specific technology, then trying to ‘design’ around it. If a major brand has an app in the iStore (??? iStore - can you blame me?) that is brand enough for them.

Look around at all these mobile applications, all these sites; so simple, straight to the point, clutter free... breath it in cause it feels pretty good if you ask me. I jumped to the American Express mobile site the other day and it actually looked good, it was honest and naked. Go to their current debacle of a website and watch as they try to squeeze AJAX into a 20th century framework and it oozes pain...you want to get in and out so bad it reminds you of a ....hmmm not going to go there...but it was bad and probably had something to do with porn...

So what is it that I am rambling on about...yes...currently mobile is so fresh and new that the actual platform speaks more to the brand itself than the brand. Next time you visit a mobile site enjoy the simplicity of it all...and perhaps ask yourself if you can pay this forward into the traditional webspace (which is dying anyway but we'll chat later on that)... content truly is king at the moment; you either have something I want or need or don't...let's hope that, for now, it stays that way.

The ADA within the commercial space for non brick and mortar commerce is an opaque area. Most litigation related to the ADA and commerce tends to be settled out of court; so currently there are no laws or rulings established. For those of you familiar with Target and perhaps Southwest you may argue otherwise but they too were also settled out of court. The closest I have seen so far is the suggestion that for whatever product you sell within an actual brick and mortar store it must be just as accessible online as well; of course this completely removes Amazon from any responsibility towards ADA compliance.

As consortiums, councils, and unions gather together to discuss the endless details of compliance within aging technologies, yes I'm talking about WCAG, there might, perhaps, be another way, at least in the short term, to satisfy many ADA requirements.

The screen reader technology should focus more on the mobile version of a website vs. the .com experience. I have found through testing of various screen readers (JAWS, SuperNova, ZoomText) that there is an endless array of useless information that needs to be given through voice (yes I'm specifically talking about users with a visual impairment here - not all people with disabilities). It doesn't allow the end user any particular way to scan this information (now I have an answer for this too but that's not the point of this piece). However, for whatever reason, we recognize this with mobile. For mobile we instantly say, we can't present all this BS, or let’s limit the results to 5 instead of 100, let’s build some quick filtering and allow the user to determine what they need, or let's just put up search/product/pay as 3 easy steps...

If we could get businesses to design mobile sites, and get screen readers to point to mobile or be detected as mobile we might be able to solve a few issues quickly and efficiently and most importantly using modern and relevant technology (WCAG that's you again).

This is obviously an incomplete thought but I wanted to get it out there for potential discussion. It's interesting because I now make almost 50% of my purchases through mobile for various items that range from an MP3 to furniture. Yet currently, and I have no statistics to back this up, I doubt many .COM commerce sites allow for a user with screen reader to purchase anything with ease, if at all.

Look for more on this topic; I'll present some research into this field within the next few months.

The foundations of GUI interfaces (and some NUI) have been built upon the metaphor of the desktop. This existed because they needed to help users understand the "new" interface they were working within. Throughout the years we have supported this metaphor as if it was some sort of axiom that all HCI / UX / IA / UXA (geez really - so many more too) followed. You can see it in almost every website (file cabinets), every application (spreadsheets), and most Operating Systems (recycle bin). For the new generation of apps, sites, and systems this metaphor should disappear altogether. It is no longer relevant.

For Babyboomers they will cling to the desktop because they grew up around it; it was in their home, in their office, it was the way they organized their work life (arguable not many played in their office). However for Millennials, Gen Yers and even Gen Xers their world isn't limited to this metaphor. As pervasive (ubiquitous) computing becomes commonplace the desktop metaphor holds us back. We no longer work in a singular office, we take everything to go. The "office" is an extension of our body (so to speak - not trying to be dramatic here), your mobile device could be considered your main office, the laptop an extension of it. The desktop, with its singular categorization of files and structures and limited interface potential, holds us back.

The visualization of information and content, the 140 char scanning of content, context awareness and geolocation, all yearn for smarter applications and environments to work within. To this day I see sites go up that still support a "main navigation" ; yet everyday it's search, scan, skim, save, and traceback. Why are we making the users work so hard? Implicit and explicit information choices should drive the structure. Use the models of attract and repel to display information (this also works well within a gesturing environment).

I was chatting with someone, just the other day, that perhaps a site should be a stream (like Twitter) of 140 char content, since it supports the model of scan and skim as a user behavior. This content could be backed up by a 'blog' when users want to search deeper within their selected content. Show the relationships to other content, as a user navigates forward (always forward) the experience attracts and repels content based on their implicit / explicit choices.

Embrace the search, it's a place to start. As a brand try a stream of information (backed by search) and once a user selects a 140 char item of interest build the display content around it. With a mobile device you can obviously add in location awareness, and perhaps take a guess as to context but always allowing the user to "flick" content away or drag content to a stream.

Of course what is this site I'm talking about here? An office or central location hub of information that people go to? That is dead as well. We are currently working with a Fortune 500 on what I like to call "freeing the content" - essentially turning all content into custom feeds of choice that vendors, dealers, and users can do with as they please; content syndication on distributed system. The next brand sites should be feeds, supported by scaffolding that allows for flexibility and choice in the presentation. I shouldn't have to go to the "office" to see what you are up to...i just need to check my "stream" and start the journey from there.

Welcome to the current.

The New Yorker had an article that ran awhile back titled "How David Beats Goliath: When underdogs break the rules." by Malcolm Gladwell. I'm glad to see that the New Yorker has made this piece available for free.

How David Beats Goliath

I'm not planning to get into all the details here about the article itself. He talks of many instances where people come at a problem with no predefined ground rules or expectations. They look at it with a completely fresh and open mind and take a different approach. For example:

"It isn’t surprising that the tournament directors found Eurisko’s strategies beyond the pale. It’s wrong to sink your own ships, they believed. And they were right. But let’s remember who made that rule: Goliath. And let’s remember why Goliath made that rule: when the world has to play on Goliath’s terms, Goliath wins."

Lenat was in a tournament for military navel strategy and he ran some scenarios into a neural network to see how AI might approach them... He won this tournament twice before they created rules around the strategies used, to essentially kick him out of the tournament. On a side note these winning strategies resembled similar tactics used by terrorists.

Malcolm speaks to real-time processing of information to then determine the strategy. Typically this is done in batches, or batch processing; you analyze the situation by the current set of rules stated and then deploy your "new" strategy. Real-time processing allows you to take in the information with little or no context and roll out a strategy that suits the needs as you see fit. Often times this results in innovative winning approach that otherwise would never have been uncovered. He uses David as an example because if he played by Goliath's rules; strength and brawn, things might have turned out differently. Instead he did the unexpected, by running towards Goliath and using a slingshot. This both surprised and ultimately defeated Goliath.

Malcolm also uses his own experience with his daughters basketball team. None of the girls had ever truly played basketball. He ran them to get their endurance up and then played full court press against all the teams in the league. This resulted in an almost undefeated season until a ref on an opposing team started calling heavy fouls and the team had to go to half court press.

So what is it I'm talking about here? Perhaps I ramble a bit but the underlying theme here is to always do the unexpected; don't over think, over analyze or over research. Agency's always tend to come heavy with the process; a process built on old fundamentals, old ideas, traditional thoughts. Take a step back and try to unlearn the expected. Analyze everything in real-time; keep iterating in small steps and evolve.