Gesture control: The next wave of consumer electronics user interface evolution

January 13th, 2015, Published in Articles: EngineerIT

 

Why have some smartphones and tablets been more successful than others? The answer is quite simple: they’re successful because of their intuitive user interface. It’s the part of the system that drives the user experience providing access and control of the services, applications and features of the device in a very easy to understand way. At the heart has been the wide adoption of touchscreens and touch-based gestures. This is all good for tactile devices but what about other devices?

tempe-113-01-jan-2015-fig0

Imagine you’re sitting in the living room watching TV and suddenly the phone rings. You struggle to find the TV remote control, grab it and then struggle to find the “mute” button.  It takes so long to find that you miss the call.  Wouldn’t it be great if you could use a simple hand gesture to the remote to mute the sound?  It’s not too far-fetched because we have experienced playing on the latest generation games console that use gestures to control the game play.  However, even the most sophisticated use technology built on infrared or using cameras-based technology – neither of which aresuitable for a wide-range of consumer applications.

What’s needed is a technology that will allow users to interact with a device using a combination of gestures and on display information i.e. a touch-free gesturing.  Taking the previous example of turning the volume down on the TV, what if you made a gesture close to left side of either the TV or remote and on the TV screen a rotating volume control appeared and you then rotated your hand in the air to turn the volume down.

The same technology would make it easier and safer to use many other everyday electrical devices, especially in the kitchen with wet or dirty hands.  It also holds great potential for enabling the infirm or visually impaired to wake up devices by using hand gestures and be able to control them without having to use small hard to locate buttons or switches.

There is now a technology that can make this happen. A new solution from Microchip Technology, based on the GestIC technology chip – a tracking and gesture controller). It is the world’s first 3D gesture controller to utilise electric fields (e-fields) for hand position tracking with free-space gesture recognition.

Because the controller only detects changes in nearby e-fields caused by conductive objects, such as the human body, it is resistant to environmental influences, such as light and sound.  Additionally, because its range is 15 cm, it can ensure that only gestures from the intended user are detected, such as preventing false detects from other people close-by.  Another big advantage of GestIC is the fact that all the sensing electrodes can be invisibly integrated behind the upper coating layer of furniture, displays etc.

Compared to other 3D gesture technologies, such as infrared, ultrasound or camera-based solutions, this technology provides several additional advantages for use in consumer devices.  For example, camera-based solutions need a certain amount of light in order to operate properly while simultaneously requiring dynamic light compensation. Furthermore, a camera has a fixed angle of view, which creates blind spots, particularly in environments where users can be very close.

The physics behind it

Fig. 1: Undistributed electrical field. Different colours show the equipotential lines while arrows represent the field lines.

Fig. 1: Undistributed electrical field. Different colours show the equipotential lines while arrows represent the field lines.

The basic sensor setup is described in Fig. 1: An isolation layer separates a single full-plane transmitter electrode located over a ground layer from the receiver electrodes located on the top layer. Controlled by the MGC3130, the transmitter electrode generates an electrical field with a frequency of about 100 kHz. Without any external disturbances, this electrical field looks like the evenly distributed field in Fig. 1. Whenever an object enters this electrical field, it disturbs/changes the field lines. A typical example of such a disturbance is shown in Fig. 2.

Fig. 2: Disturbed electrical field – in this case by a hand: The formerly even distribution of equipotential lines as well as of field lines is significantly changed.

Fig. 2: Disturbed electrical field – in this case by a hand: The formerly even distribution of equipotential lines as well as of field lines is significantly changed.

Due to the disturbance, the formerly even distribution of the equipotential lines as well as of the field lines is significantly changed. By entering the electrical field, the conductive object (in the applicable cases: a hand) absorbs the electrical field as it conducts electrical charges to ground, resulting in a change of both the electrical field and the equipotential lines. Using the controller, it is possible to measure the minuscule signal deviations at the receiver electrodes generated by the hand and to process these results. The closer the hand gets to the receive electrode, the higher the local influence of the hand. Utilising four receiver electrodes (north, south, east and west) the MGC130 allows tracking of the hand’s position in X, Y and Z direction within the sensing area. In a further computing step, gestures are identified by applying a hidden Markov model (HMM)-based gesture recognition engine which allows an exceptionally high, user-independent gesture recognition. When, for example, the hand flicks from the right to the left, there is a high signal deviation on the right side at the beginning. This signal deviation on the right electrode decreases during the movement, while the signal deviation on the left electrode simultaneously increases. Movement patterns like this are recognised and calculated on-chip, and provided in a predetermined manner at the outputs.

The X/Y/Z tracking resolution is up to 150 dpi, depending on the electrodes’ design and the hand’s position. The closer the hands position relative to the sensing area, the higher the signal to noise ratio and the higher the resolution. GestIC technology enables this mouse-like accuracy without almost any jitter. For demonstration purposes, the design team controlled a notebook PC’s cursor jitter-free, just by moving the hand and without the use of a mouse.

The electrodes

The receive electrodes are always located above the transmitter electrode (Fig. 3). For example, in the demonstration setup that Microchip showed at the Electronica trade show, the receive electrodes consisted of copper layers on the upper side of a PCB. Electrodes can be realised by using any type of solid conductive material such as solid PCBs, flexible printed circuit boards (FPCs), LDS electrodes (laser direct structured), conductive foils, and the aforementioned ITO layer already found in displays.

Fig. 3: Standard Electrode Design. While the North/South/East/West provide the x/y coordinates of the hand, the centre electrode delivers the z coordinate.

Fig. 3: Standard electrode design. While the north/south/east/west provide the x/y coordinates of the hand, the centre electrode delivers the z coordinate.

Microchip’s GestIC technology is able to work with thin sensing which allows for an invisible integration behind the target device’s housing,without increasing the overall thickness of the product’s industrial design. Therefore, the electrodes are not only low-cost but also low-impact in terms of the overall design. This is of high importance in consumer electronics, where these electrodes can be hidden behind a fascia ensuring an attractive and stylish finish.

As mentioned earlier, the reuse of existing conductive structures, such as the ITO coating of a displays touch panel, makes GestIC technology a very cost-effective system solution. Currently, Microchip is working with major display manufacturers, on the pre-integration of GestIC technology into a complete display module. By connecting the MGC3130 to the ITO coating, the touch area of the display is transformed into an electrical-field electrode sensor field without disturbing the multi-touch functionalities of the underlying touch display. Due to its seamless integration, GestIC technology initiates the third dimension of sensing as soon as the fingers are removed from the display surface. In consumer applications, this can enable the system to display different items based on the direction from which the hand approaches. While a vertical approach might enable the menu, an approach from the lower left side could enable the system’s setup menu. Another possibility is switching between basic menus by using flicking gestures in the air.

The technology is very flexible, as it not only recognises linear gestures but also symbol gestures, circular gestures and others. In the consumer environment, this capability could be used in order to increase/decrease the volume of an entertainment system, by drawing a virtual volume knob in the air. Regardless of the function assigned to a particular gesture, the end result is that users would be able to control a device with their hands by gesturing in the air whilst being able to keep their eyes glued to reading their book or screen.

Chip technology

The MGC3130 is a configurable mixed-signal controller consisting of an analogue front end with one transmit and five-receive channels and a digital signal processing unit  (SPU). Four of these five channels are used for recognising gestures or the hand’s position, while the fifth channel enables touch detection and improves low distance accuracy. Each channel undergoes signal conditioning. The pre-conditioned analogue signals are then digitised and processed by the integrated SPU.

At the output, the SPU provides the calculated results via I2Ctm or SPI interfaces. Microchip provides an application programming interface (API) running on the application or host controller. This API allows the design engineer to easily map the relevant signals to the target destination. This means that designers do not have to concern themselves with the signal conditioning, as Microchip pre-processes the X/Y/Z hand position data as well as comprehensive gesture-recognition software on chip, known as the Colibri suite. In order to enable designers to create highly individual special features through application-specific post processing, Microchip also passes through the filtered electrode signals to the outputs.

The Colibri suite software

The Colibri suite uses a HMM- based gesture recognition engine, in conjunction with X/Y/Z hand-position vectors. HMM provides the highest user-independent recognition rates for 3D hand and finger gestures. This means that GestIC technology consistently provides exceptionally high gesture-recognition rates, regardless of who is controlling the device.

At the controller’s digital output, the Colibri suite delivers high-resolution X/Y/Z hand position tracking data as well as flick, circle and symbol gesture words. For flick gestures, the Colibri suite not only detects basic movements, such as left, right, up and down, but also more sophisticated flick gestures, such as from the inside to the outside and vice versa, over the entire or partial sensor areas. The user is then able to perform input commands, such as “open application,” point, click, zoom, scroll, mouse-over and many others, without the need to touch the device.

If the design requires a specific gesture that is not included in the suite, a gesture recording and training module is provided, that allows design engineers to add their own gestures to the library. Due to the MGC3130’s flash architecture; designers can download the recognition parameters for these new gestures into the IC.

Furthermore, approach detection is a programmable function that scans for user activity while the chip is in self-wake-up mode. If real user interaction is detected, the system will automatically switch into full sensing mode while alternating back to a very low power self-wake-up mode once the user’s handwill has left the sensing area.

Flexible adaption to the environment

GestIC technology operates with a carrier frequency of around 100 kHz. Whenever noise is detected from devices such as motors, inverters, chargers and fluorescent-lamp drivers, the MGC3130 automatically adapts its field emission frequency to a noise-free channel in the range between 70 and 130 kHz, thus avoiding RF interference and providing a very robust solution. On the other hand, its emitted energy is very low. This means that GestIC technology doesn’t cause interference with other systems, elements as evidenced by the fact that it passed EMI tests, such as the IEC 61000-4-3. Furthermore, while the maximum current consumption of the MGC3130 IC is 70 mW, the majority of its power consumption is needed for the interpretation, evaluation and classification of the disturbed electrical field signals—not for the emission of the electrical field itself.

The story behind it

For more than ten years, the Munich/Germany-based company Ident Technology – which was acquired by Microchip Technology in 2012 and joined its Human Machine Interface Division (HMID) – has gained experience and know-how around the topic of using electrical fields, and it has filed numerous patents. This know-how includes how to apply the theories of e-field’s in a real environment, the design of the sensing electrodes, the chip design, and the algorithms used to process the raw sensing data.

The wavelength of the 100 kHz signal used by GestIC technology is roughly 3 km. This means that the dimensions of the sensing electrode area, which is typically less than 15 cm x 15 cm, is several orders of magnitude smaller than the emitted signal’s wavelength. This combination results in a very stable quasi-static e-field during operation that can be used for sensing conductive objects such as the human body, while the magnetic component is practically zero and no wave propagation takes place.

Enabling future applications

Fig. 4: Basic software setup principle for gesture recognition.

Fig. 4: Basic software setup principle for gesture recognition.

In order to facilitate application designs, the HMID team is currently preparing a document that describes all of the relevant factors one must consider during the design and positioning of sensing electrodes. This will enable automotive customers to design their own electrodes or implement them using materials already present in their designs.

Microchip also plans to offer a development kit named Hillstar, which is intended to support customers during the design-in phase. It features GestIC technology, including the Colibri suite, and connects to a PC through a USB interface, enabling engineers to conveniently connect their electrode designs and parameterise the MGC3130 chip on a PC.

Microchip’s feature-rich graphical user interface, dubbed Aurea, runs on the Windows 7 Operating System and provides control of the MGC3130’s parameters and settings, making it easy to update and save parameters (Fig. 4).

Contact Willem Hijbeek, Tempe Technologies, Tel 011 455-5587, willem.hijbeek@tempetech.co.za