3D Measurement Technology Provides Real-Time Gesture-Based Human-Machine Interaction

Humans and machines will increasingly support each other in the workplace. For processes to be efficient, the machine must respond to the human worker without any time delay. Due to sophisticated high-speed 3D measurement and sensor technology, researchers at the Fraunhofer Institute for Applied Optics and Precision Engineering (IOF) are making this real-time interaction possible.

As gesture control represents a seamless interface between humans and machines, more and more machines, robots and devices are able to respond to gestural cues. Researchers at Fraunhofer IOF in Jena, Germany are raising human-machine interaction to a new level: the high-speed 3D measurement and sensor technology developed in the 3D-LivingLab research project enables them to capture and interpret even complex movements – and does so in real time. At the upcoming Hannover Messe 2019, the research team is demonstrating its gesture-based human-machine interaction technology using the example of a wall made up of 150 spheres, which copies in 3D every head, arm and hand movement of a person standing in front of it. The wall of spheres effectively imitates the body movements with contact-free 3D reactions in real time, free from time lags. The wall of spheres was created as part of the “3D-LivingLab“ project.

Workflows Greatly Simplified

The spheres respond instantaneously to the movements of the person standing in front of the wall.

The system is made up of several modules: 3D sensor, 3D data processing and image fusion as well as the actuator system itself comprising 150 individual actuators. “The wall of spheres is not only a great toy, it also represents cutting-edge technology. Real-time 3D capture and interpretation of multiple gestures without tracking sensors can radically simplify workflows – from production scenarios to health and safety,” says Dr. Peter Kühmstedt, scientist and group leader at Fraunhofer IOF.

The demonstrator system responds to the behavior of people, captures complex movements such as gestures and physical actions, and gives real-time feedback through a technical actuator system that converts electrical signals into movement on the wall of spheres. It is the person’s posture that controls the actuators. Specially developed algorithms enable human 3D movements to trigger control of the actuators, thus causing the spheres to move. “We are demonstrating very rapid measurement technology – the data is captured by a new generation of 3D sensors –, very rapid low-latency processing – the data is interpreted and converted immediately – and very rapid reactions in real time. According to the calculation results, the wall of spheres immediately mirrors the movement of the person in front of it,” says the researcher. In production environments, for example, the technology could be used to monitor a worker who is interacting with a robot and handing it parts. It could also be transferred to other application fields, such as health and safety, where it can make processes safer and more efficient. Other conceivable applications for the 3D sensor technology and the interaction components are in assembly assistance and quality control systems. Moreover, they also qualify for the monitoring of biometric access points.


The goal of the ”3D-LivingLab” research project is to build up a transportable demonstrator system that can be expanded on a modular basis. The demonstrator is designed to facilitate intuitive interaction between humans and technology in a representative fashion and to be available to the 3Dsensation consortium (see below) for further research purposes. The wall of spheres being presented at the Hannover Messe in Hall 2, Booth C22 illustrates the results of the project. The following partners are involved in the ”3D-LivingLab”: Fraunhofer IOF, Fraunhofer Institute for Machine Tools and Forming Technology IWU, Fraunhofer Institute for Telecommunications, Heinrich-Hertz-Institute, HHI, Chemnitz University of Technology, Friedrich Schiller University Jena, SICK AG and FusionSystems GmbH.

For more information: www.iof.fraunhofer.de