Smartwatch Detects Finger Taps, Helps Tuning Guitar

Researchers from Carnegie Mellon University discovered that a smartwatch is capable of detecting and distinguishing a wide range of flicks, taps and scratches by the fingers and hands, and all that is needed is a software upgrade that repurposes the existing accelerometer of the device.

This functionality enables new applications that make use of regular gestures for controlling the smartwatch, and eventually various other objects linked through the internet of things. The smartwatch monitors vibrations that take place when people use tools or hold objects, making it capable of identifying activities and objects.

It can also be used for tuning a guitar and also for displaying the note transmitted as each string is plucked and adjusted by the guitarist.

It's as if you're using your hand as a detection device. The hand is what people use to interact with the world.

Gierad Laput, Ph.D. Student, Human-Computer Interaction Institute (HCII)

Dubbed ViBand, the new technology was developed by Laput and Ph.D. student Robert Xiao, both members of the HCII's Future Interfaces Group, with their adviser Chris Harrison, assistant professor of human-computer interaction. A paper that will be presented by them will explain the work at the Association for Computing Machinery's User Interface Software and Technology (ACM UIST) Symposium this week in Tokyo.

Generally, a smartwatch accelerometer identifies when a person lifts an arm in order to activate the screen, or sometimes to count the number of footsteps. This is achieved by the accelerometer by taking measurements about 100 times a second.

However, when the sampling frequency was increased by the researchers to 4,000 a second (4 kHz), they discovered that the accelerometer behaved like a vibrational microphone. The accelerometer coupled with the body for detecting bio-acoustic signals instead of detecting sounds transmitted through the air.

ViBand isn't just a way to control your smartwatch. It enables you to augment your arm. It's a powerful interface that's always available to you.

Chris Harrison, Assistant Professor, Carnegie Mellon University

A ViBand-enabled watch can notify if someone is tapping on the back of the hand, palm or the forearm. It can detect scratches, finger flicks and other such motions. This watch is also capable of sensing if an individual holds a variety of electrical and mechanical tools, such as, a handsaw, power drill or an electric toothbrush. Each device, body tap or activity has unique bio-acoustic signals.

The team increased the frequency of the sampling rate of the accelerometer by developing a custom kernel, which is considered to be the heart of the smartwatch's operating system. That is the only modification needed and can be performed as a software update, Laput stated.

A number of demonstration apps for ViBand were developed by the team, also including the usage of hand gestures close to the watch in order to control its apps. Similar gestures can also help controlling remote devices, such as a TV, lights or various other appliances linked via the internet of things.

The team further demonstrated that it could be used for object-aware apps, such as providing visual feedback when an acoustic guitar is tuned or monitoring meal preparations.

The researchers finally recommended an application using a vibro-tag, which refers to a small object emitting structured, inaudible vibrations containing data. For instance, a vibro-tag on an individual’s office door might transmit information related to office hours or alternative contact information.

This work was supported by the David and Lucile Packard Foundation, a Google Faculty Research Award and Qualcomm.