Most common gestures currently used by digital systems

In this section I would like to show the most common gestures currently used by systems. There are a few gestures which are very common and already so established that we already use them intuitively.

Tap or point to open, select or activate
Most touch based applications use the tap to send click events. It is used to open a function, select an item or activate it. Most touch screen mobile phone use this as one of the most used gesture. But if you look at controller free applications like Microsoft Kinect this function is hardly used yet. „Pointing is the most natural gesture for selection.“ [Saffer, Dan (2008): Designing Gestural Interfaces. O‘Reilly Media, Inc. ISBN 978-0-596-51839-4 Page: 76]

Drag and Drop
From GUIs on personal computers we know drag and drop functions. In a natural user interface such a function can be a very clear gestures as it can be directly transformed in how we move objects in real life.

Pinch to shrink and spread to enlarge
Since Apples iPhone and iPod Touch got really popular nearly everybody knows that the pinch or spread of fingers or hands gestures can change the size of an object. It became one of the most popular gesture without a direct real life counterpart.

Wave to activate
Waving is a simple gesture and has already a wide usage area. Not only is it needed to activate the user tracking on Microsoft‘s Kinect , it‘s also very common in public restrooms for the water tap, paper spender or the toilet flush.

Possible In-Window Purchase

When we design the next generation of shop windows its important to think about the additional values. Not only can such a system create an entertaining buzz, but it can also drive sales. If we add the ability for the customers to purchase products directly from the window, even when the shop itself is closed. To establish an easy connection between the shopping window and an online store we have a few possibilities.

Order form
Having an order form directly integrated in the system can make it easy for people to order a product. There will be no additional device needed to order a product.

Video Order
Shoppers will record their adress on video using a camera and a microphone built into the shopping window. This way no additional device is needed for the shoppers.

Order using SMS
Nearly everybody owns a mobile phone. Being able to use text message from a mobile phone is a convenient way to order a product.

QR Code
The QR (Quick Response) Tag, was developed by a Japanese company in 1994. Its a 2 dimensional code to store information. Most smart phones have the possibility to read QR codes. Using a QR code in the description of the product makes it easy for the user to order it online. One advantage of such a system is that the order address or payment option not directly will be handled in the shopping windows which makes the order process more secure and private.

QR-Code which contains the URL:

Google Wallet (NFC)
With Google launching a mobile payment system based on NFC Chip payment we will soon get the possibility to pay all over the world with our mobile phones. This system will allow to pay products through the window directly from the street

Technologies of Interactive Shopping Windows

Interactive shop windows use different technologies. Currently the most common are touchscreens. Many examples us this technology because it is already well established. Newer touch-screens use multi-touch which improves the user experience. Using a depth sensing camera or multi camera tracking setup is the latest trend in interactive shop windows. Shoppers will no longer need to touch anything and can use their hands or body to control the information displayed on the shop window. New technologies allow multiple users and 3D gesture tracking. With such a system in place window shopper can look at products from any angle using simple gestures. Whenever there is a user and the system knows what the user is looking at, it can track this information. Using this information in an analytics system can give valuable information of which product is looked at most and for how long. This technology would allow to take a picture of each person who looks at a product and store this information together with the products the person looked at (even though under the current law in Switzerland this would be prohibited).

Further reading:

Mindmap: First Concept Directions

During my research process I decided to focus on a public space. I was thinking about what media I could use and what gestures might be interesting to design a dialog between a system and a user.

Interactions in a Digital System

When we think about interactions in digital systems we can divide it in three areas:

Digital Manipulation
Whenever we press a button, drag a scrollbar or move an object in our graphical user interface we manipulate a digital system. Our language involves different patterns like single click, double-click, press and move, release and roll over. All these patterns are performed by a mouse, keyboard or a similar device.

Gestures in 2D
Most touchscreens and modern trackpads allow gestures to manipulate the software. Since Apple released the iPod Touch and iPhone we got familiar with simple gestures with our fingers, such as flicking album covers with one finger or zooming and scrolling with two fingers.

Gestures in 3D
After Nintendo launched the Wii controller many people got used to interact with gestures in a 3D space. New gestures like shaking, turning or spinning were introduced. With the launch of Microsoft’s Kinect gestures with the hole body became popular and accessible to a general audience. Currently such systems are mainly developed for gaming, exhibitions or experiments.

Further reading: Bachelor Thesis TWYE Fabian Kuhn Page 16-18

Accessibility in Gestural Controlled Applications

While designing a gestural interface we always need to consider people with physical disabilities and handicapped people. When we design an application mainly based on hand gestures, we need to think about how people with limited hand movement possibilities can also use the system. It is important that such systems use a small number of gestures, all of which should be easy to perform.
But not only will a gestural interface be harder for certain people to use, it may also be easier if a person has problems performing the small movements to control a mouse or a keyboard.

Further reading: Dan Saffer, Designing Gestural Interface 978-0-596-51839-4 Page 44

2D vs. 3D – a few thoughts

During my research I came across the question if I should build a 2D or 3D interface for my application. Here a few thoughts I had.

Advantages of a 2D interface
The 2 dimensional applications are very common. Most of the graphical user interfaces have been designed in 2D. Users can deal with such systems using common input devices like a mouse or a keyboard.

If a spatial input is used but the interface remains flat, it can be confusing for users. A transmitted thinking is required.

Advantages of a 3D interface
If a 3 dimensional representation of a user interface is used, users will more likely use 3 dimensional gestures. To support the intuitive action of the user, spatial input should be used for a 3D interface.

Is an interface designed in 3D and controlled by 3D gestures it will be relatively quickly tiring. 3D interfaces are often slower to reach a goal.

Further reading

Background Research

Background Research
During my research with gesture based applications I found that many applications are difficult to use. But all analyzed projects had a few interesting parts in them. One major lack in gesture based applications is that we do not have a gesture standard yet. There are only a few common gesture which have quite established in currently used systems . If we focus on application which will be available in public space and easy to use we have to use this gestures or find a way to display our new gesture in a very easy way.

The use of touch-free interaction with an application in public space seems a good solution for my bachelor project. The research showed that in this field already many different systems got established. Some of them had a complicated multi camera setup wile others used a depth sensing camera setup. Using a depth sensing system will make it easier to calibrate and use.
In my bachelor project I will build an easy to use, Kinect controlled application, that uses an easy to learn gesture set.

Questions I would like to answer during this project:

  • When does it make sense to use a gesture and what gestures are easy to learn?
  • What gesture will normal users use for a common tasks?
  • What gestures can be used in public space?

download BA Background Research Paper (PDF)