Sciencetech insightTechnology

Machine Learning and Finite Element Simulation for Real Time Virtual Haptic Signal Synthesis

Go Touch VR Machine Learning VR Touch Touch VR Virtual Reality Wearable

The synthesis of haptic stimuli is a challenging problem. This has an impact for Virtual Reality, where the implementation of viable virtual objects manipulation methods is critical. Today, the scientific understanding of the touch sense is not comparable with vision or hearing. For the touch sense, it has yet to be identified a standard set of signals to linearly combine to reproduce the perception of a material. This set of signal could be the touch parallel of the RGB standard for the visual signal reproduction. The identification of such set is particularly complicated considering the extension of the haptic perception along all the body, the not clear identification of haptic “colors”, and the mechanical complexity of haptic devices aiming to reproduce the full perceptual bandwidth in human.

These difficulties prevent the implementation of an easy way to encode, synthesize and store haptic signals.

To overcome this long standing issue, we developed a computational method based on these points:

  1. Finite element simulation of the forces acting on the finger while interacting with a parametric 3D surface
  2. Training of a neural network storing the behavior of the forces for all the parameters
  3. Real time synthesis of the haptic signal based on the trained network.

Here how it works:

An anatomically and mechanical accurate model of the finger is interfaced with a 3D parametrized surface during the interaction, and the forces applied to the finger are recorded and stored in a database. In the following picture, the model of the finger is pictured sliding over a bump.

Go Touch VR Machine Learning VR Touch Touch VR Virtual Reality Wearable

Once recorded these stimuli for multiple strokes of the finger over different surfaces, we were able to train a neural network to render in real time any kind of surface chosen before the parametrization.

Go Touch VR Machine Learning VR Touch Touch VR Virtual Reality Wearable

We applied this method to reproduce electrovibration haptic signals as a proof of concept. We built a tool to freely calculate the electrostatic voltage necessary to reproduce the physical signal for friction modulation displays. The tool is freely available on

Go Touch VR Machine Learning VR Touch Touch VR Virtual Reality Wearable

We will work in the future to implement this automatic method into 3D virtual haptic signal reproduction for realistic grasping sensation based on the Go Touch VR technology.

This is part of the background work that led to the creation of Go Touch VR.

This was the first, and up to now only paper, combining machine learning and haptics.

More information on the work can be found in this article we wrote last year:

Vidrih Z., Vezzoli E. (2016) Electrovibration Signal Design. In: Bello F., Kajimoto H., Visell Y. (eds) Haptics: Perception, Devices, Control, and Applications. EuroHaptics 2016. Lecture Notes in Computer Science, vol 9775. Springer, Cham