New application allows controlling robots with headphones?

Scientists have built an application that allows robots to be controlled while wearing an Apple Vision Pro VR virtual reality headset .

Scientists have developed an app for the Apple Vision Pro mixed reality headset that allows users to control the robot using just head and hand gestures. It can be used for remote control in various situations, especially in disaster areas.

An app called "Tracking Streamer" tracks human movements - specifically how the head, wrists and fingers move - and transmits this data over a Wi-Fi connection to the robot to control it. (Photo: Younghyo Park).

An app called "Tracking Streamer" tracks human movements - specifically how the head, wrists and fingers move - and transmits this data over a Wi-Fi connection to the robot on the same network. The connected robot will then translate the data into corresponding movements.

The researchers published their work in a paper recently posted on Github, an open source repository. The system tracks 26 points in the hand and wrist, as well as separate points in the head. It also records spatial data - such as how high above the ground you are.

App developer Younghyo Park, a doctoral student at the Massachusetts Institute of Technology (MIT), published a video to social network X showing the system in action.

In a short video, study co-author Gabe Margolis, a graduate student at MIT, controls a four-legged robot with his hands and body movements.

Margolis performs gestures to command the robot to approach a closed door, turn the handle with the gripper, and enter. In another scene, he signals the robot to open the trash can lid and throw items into the trash. At times, he also bent down - this caused the robot to imitate him.

Apple launched the Vision Pro virtual reality (VR) headset in February 2024, to assist wearers in their daily lives with footage. Just before launch, scientists published a study to investigate how well VR headsets work if you use them in everyday life.

They found that the devices had a time lag, limited peripheral vision, and caused distortions similar to a smiling circus mirror. They warn against wearing a VR headset while driving or walking down stairs.

Park and Margolis speculated that some users could wear Apple Vision Pro all the time, like glasses, and this would provide more data to teach the robot how humans move and it would follow suit.