John Cartan, User Experience Architect, Applications User Experience

After thirty years of pointing and clicking, human computer interaction is entering a new phase. Input methods are becoming more sophisticated and flexible (multi-touch, touchless, tactile, audio). At the same time, output methods are becoming both bigger (surface displays, electronic whiteboards) and smaller (smart phones, head-mounted displays, flexible plastic displays).

Though these innovations usually first catch on in the consumer marketplace, it’s just a matter of time before they are adopted at the enterprise level. Each new input/output combination (iPhone, Wii) creates new implications for user interface design. At Oracle, we are trying to understand these implications and prepare for them.

A finger tends to be less precise than a mouse, so we may need more focused screen layouts with fewer, bigger, and more widely-spaced buttons. Mouse-optimized controls like drop-down fields may need to be replaced with finger-optimized controls like animated spin wheels. Or better yet: allow dynamic target resizing – temporarily magnifying the part of a screen the user is looking at or reaching for.

Display size creates new challenges, whether big or small. Small screens may require built-in panning and zooming along with task flows that focus on one thing at a time.
Big screens require new ways to keep controls always within reach and readable whether up close or far away, and must display more information without overwhelming users.

Display size also creates fundamentally new kinds of interaction. Small devices are highly mobile and can respond to their changing location and orientation to become context aware. They can automatically switch from touch input to speech recognition when held near the user’s ear or reconfigure choices based on the device’s location or the detection of nearby objects.

Big screens create the possibility of truly collaborative multi-user interactions. This will require the UI to resolve who is doing what and provide new paradigms to establish and release control. Updates, undos and submits will become trickier. Task flows that work at a desk may not work when standing in a crowd. Public interactions can create more time pressure and less tolerance for errors and side-trips. Users can stand up and walk away, and need ways to effortlessly pass control and access privileges.

The richer set of gestures made possible by multi-touch inputs has already spurred new interactions like stretching, rotating, and flicking. Users can convey more information with each gesture and expect the UI to respond appropriately. In particular, users expect that natural inputs will produce natural reactions: natural motions when dragging or tossing objects, and scrolling that accelerates and bounces. We are learning how to embed richer, more realistic animations into our applications.

Multi-touch inputs may soon be superseded by touchless 3-D gesturing in both large and small devices. As we move toward ever more natural and immersive interactions, meaning will start to be encoded in gestures instead of controls. Instead of pointing to a button on the screen which represents some predefined action, users could make an “OK” sign to approve, cross their fingers to delete, or clap their hands to submit. 3-D gestures will spur 3-D UIs: new forms of navigation that employ the Z axis, or the ability to fold or bunch-up tables.

Once users are given new ways of expressing their intent, computer UIs must inevitably provide new ways of responding. The mouse was initially dismissed as a toy but soon gave rise to windows, buttons, and icons on every business desktop. As users cut their mouse cords and step away from their desks, UIs must again respond. Within a few years, even enterprise UIs may be as different from today’s GUIs as GUIs were different from green-screen command-line interfaces.

Image 1. Gesture-based interactions create new possibilities.