iPad's Natural User Interface at Work

John Cartan, User Experience Architect, Applications User Experience

John Cartan

Many people have been surprised by the immediate success of Apple's iPad. It is already beginning to transform the way we read and relax at home - and also the way we work. Why did it succeed where so many earlier tablets failed? And will it (and its inevitable imitators) also transform the enterprise?

The answer to both of these questions, I believe, lies in something called the NUI, the Natural User Interface. We are in the early stages of a paradigm shift that will indeed transform the enterprise. I think I know how it will play out because I've seen it happen once before.

NUI image is an iPad screenshot of the DICE HD application, Fullpower Technologies, Inc.

In 1984, Apple introduced a new computer, the Macintosh, and a new interaction paradigm developed at Xerox PARC called the Graphical User Interface (GUI). Its windows, icons, menus, and pointing device - the "mouse" - were a radical change from the then reigning Command Line Interface (CLI). At first it was dismissed as a fad or a game system with no relevance for the workplace.

But its influence increased steadily because, for most things, GUIs worked better than CLIs. GUIs required less training and made possible whole new kinds of applications. Within a few years, mice began to appear on every desktop.

For all of its power and ease of use, though, a GUI has limitations. Instead of interacting directly the way we do with people, you still have to formulate commands and interpret responses. The windows desktop is a metaphor, rather than a direct representation. And the mouse requires you to move your hand horizontally in order to move a pointer vertically - while also chaining you to your desk.

So even as the GUI age was dawning, researchers were already working on its successor, a new approach which came to be known as a NUI. So what is a NUI?

A NUI is an interface that lets people use their natural behaviors to interact directly with information. I find that NUIs have four defining characteristics:

  1. Direct, natural input
  2. Realistic, real-time output
  3. Content, not chrome
  4. Immediate consequences

"Direct, natural input" can include 3D gestures, speech recognition, facial expressions, and anything else that comes naturally, but for now it mostly means multi-touch. Multi-touch goes far beyond mouse clicks. It allows natural, expressive gestures like pinching, stretching, twisting, and flicking. And it's especially well-suited for devices like phones and tablets that you are already holding in your hands.

In any interface, richer input demands richer output. In order to harness natural responses, a NUI output has to be as fast and convincing as nature itself. When the user makes a natural gesture like a pinch, the display has to respond in an animated, often photorealistic way in real time, or else the illusion will be broken.

The GUIs of today, with their windows and icons and menus, are laden with visual signals and controls, or "chrome". This is one of the most unnatural features of a computer interface and tends to distract users from the actual content they are trying to work with. A NUI strips most of this away and lets users focus on one thing at a time.

Finally, a NUI is not just spatially realistic, but temporally realistic as well. In the real world, actions have immediate consequences. If you want to go swimming, you don't have to wait for a river to "boot up". Splashes happen as you swim and will not be lost if you forget to save them. Similarly, NUI devices and applications start instantly and stop on a dime. Changes are saved as you go.

Apple did not invent the NUI, but its iPhone was the first device to take these concepts mainstream. Competing smart phones, with their GUI interactions and rows of tiny buttons, were no match for the iPhone's chrome-free, fully-responsive, multi-touch UI. The iPhone was NUI's proof of concept.

But the iPhone did not trigger a full paradigm shift, because there is a relatively small overlap between smartphone use cases and desktop use cases. The iPad, however, is a different beast altogether.

Most smartphone tasks follow what our Oracle mobile team calls the two-minute rule: you take the phone out of your pocket, do your task, and put it right back. Tasks on the iPad, in contrast, often last just as long as desktop tasks. In fact, they are often the very same tasks.

The iPad's screen size makes all the difference. It allows a much fuller expression of NUI interactions with innovations like popovers and orientation-sensitive split screens not possible on a pocket-sized device. The result is that many common tasks currently performed on a desktop or laptop can be done more efficiently and more pleasantly on an iPad.

CLIs did not go away, and neither will GUIs. Both are still superior for certain types of tasks. For awhile at least, GUIs will be preferred for complex, desk-bound tasks that really require multiple windows and lots of chrome.

But for simpler tasks, like reading, surfing the web, dealing with email, sketching diagrams, writing blogs, and for unfettered tasks now done with paper or clipboards in warehouses or hospitals or hallways or airports, more and more people will prefer NUIs.

Just as GUI did many things better than CLI, so NUI now does many things better than GUI. The iPad is a tipping point, just as the Mac was. And because this change will be so far-reaching, the impact on the enterprise is not "if," but "when."

Related Links


Excellent analysis John, agee with your assessments here. I wonder where the NUI concept will take us in terms of user assistance, or indeed if the very concept as we know it, even applies! http://blogs.oracle.com/userassistance/2010/07/natural_user_assistance.html

Posted by Ultan on July 15, 2010 at 10:30 PM PDT #

Nice explanation. May I have a question, how long it will take for having NUI on our every day PC or laptop...? Why is the industry waiting on regards this technology? Greeting from Clearwater, FL.

Posted by Pablo on July 25, 2010 at 09:13 PM PDT #

To use an old paradigm, it seems that we are finally coming to the true fruition of object-oriented design, where each chunk of content is an object with the actions intrinsic to it. Actions are not relegated to chrome so chrome is unnecessary. Every item in a database can be exposed as content: no more tables, no more grids, watch the users flip their lids! Even RSS readers are getting away from list format (Flipbook, Google Play)...the list is dead, long live the object.

Posted by Melody on October 07, 2010 at 07:03 AM PDT #

Thanks, John, for your article. I responded over here_ http://blogs.oracle.com/mprove/entry/reipadsnui

Posted by mprove on May 26, 2011 at 02:07 AM PDT #

Post a Comment:
  • HTML Syntax: NOT allowed

The Oracle Applications User Experience (OAUX) Usable Apps in the Cloud blog.

Oracle Applictions User Experience logo

Twitter Usable Apps on Twitter

Main OAUX Website:

Usable Apps