New Wrist-Mounted Device Continuously Tracks Entire Human Hand in 3D

Scientists from Cornell University and the University of Wisconsin, Madison have achieved what could be a possible landmark in the field of wearable sensing technology.

They have developed a wrist-mounted device that constantly monitors the complete human hand in 3D.

Dubbed FingerTrak, the bracelet is capable of sensing the multiple positions of the human hand and translating into 3D, by making use of three or four tiny, low-resolution thermal cameras that read outlines on the wrist. The positions include 20 finger joint positions, as well.

According to the team, the device could be utilized for human-robot interaction, mobile health, virtual reality, translation of sign language, and other similar fields.

This was a major discovery by our team – that by looking at your wrist contours, the technology could reconstruct in 3D, with keen accuracy, where your fingers are, it’s the first system to reconstruct your full hand posture based on the contours of the wrist.

Cheng Zhang, Assistant Professor of Information Science and Director, SciFi Lab, Cornell University

Yin Li, an assistant professor of biostatistics and medical informatics at the University of Wisconsin, Madison School of Medicine and Public Health, supported by developing the software behind FingerTrak.

The paper titled “FingerTrak: Continuous 3D Hand Pose Tracking by Deep Learning Hand Silhouettes Captured by Miniature Thermal Cameras on Wrist” was published in the Proceedings of the Association for Computing Machinery on Interactive, Mobile, Wearable and Ubiquitous Technologies in June 2020.

Moreover, the study will be presented at the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing, scheduled to be held virtually from September 12^th to 16^th, 2020.

Previous wrist-mounted cameras have been regarded as very heavy and more prominent for daily use, and a majority of them were able to rebuild only a few discrete gestures of the hand.

FingerTrak’s novel feature is a lightweight bracelet that enables free movement. Rather than making use of cameras to instantly capture the finger position, which was the focus of a majority of the previous studies, FingerTrak employs a combination of machine learning and thermal imaging to reconstruct the hand virtually.

The four tiny, thermal cameras provided in the bracelet—each with a size of a pea—capture several “silhouette” images to develop a sketch of the hand.

Then, a deep neural network combines such silhouette images together and rebuilds the virtual hand in 3D. This technique enabled Zhang and his colleagues to capture the complete hand pose, even if an object is held in the hand.

The most novel technical finding in this work is discovering that the contours of the wrist are enough to accurately predict the entire hand posture. This finding allows the reposition of the sensing system to the wrist, which is more practical for usability.

Cheng Zhang, Assistant Professor of Information Science and Director, SciFi Lab, Cornell University

The team believes that the technology could find an extensive range of potential applications. However, according to Zhang, the most potential one is its use in the translation of sign language.

“Current sign language translation technology requires the user to either wear a glove or have a camera in the environment, both of which are cumbersome,” Zhang added. “This could really push the current technology into new areas.”

Li notes that FingerTrak could even influence health care applications, particularly in tracking disorders that impact fine-motor skills.

How we move our hands and fingers often tells about our health condition. A device like this might be used to better understand how the elderly use their hands in daily life, helping to detect early signs of diseases like Parkinson’s and Alzheimer’s.

Yin Li, Assistant Professor of Biostatistics and Medical Informatics, School of Medicine and Public Health, University of Wisconsin, Madison

Besides Zhang and Li, the FingerTrak team contains three collaborators who were visiting undergraduate students at Cornell’s SciFi Lab last fall—study first author Fang Hu of Shanghai Jiao Tong University; Peng He of Hangzhou Dianzi University; and Songlin Xu of the University of Science and Technology of China.

Journal Reference:

Hu, F., et al. (2020) FingerTrak: Continuous 3D Hand Pose Tracking by Deep Learning Hand Silhouettes Captured by Miniature Thermal Cameras on Wrist. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Tech. doi.org/10.1145/3397306.