The power of audio in 6 Degrees-of-Freedom (6dof)

Recordings made with 6DOF Development Kit by Zylia

​What would happen if on a rainy and cloudy day, during a walk along a forest path, you could move into a completely different place thousands of kilometers away from you? Putting the goggles on would get you into a virtual reality world, you would find yourself on a sunny island in the Pacific Ocean, you would be on the beach, admiring the scenery and walking among the palm trees listening to the sound of waves and colorful parrots screeching over your head.
​
It sounds unrealistic, but such goals are determined by the latest trends in the development of Augmented / Virtual Reality technology (AR / VR). Technology and content for full VR or 6DoF (6 Degrees-of-Freedom) rendered in real time will give the user the opportunity to interact and navigate through virtual worlds. To experience the feeling of "full immersion" in the virtual world, realistic sound must also follow a high-level image. Therefore, only each individual sound source present in virtual audio landscape provided to the user as a single object signal can reliably reflect both the environment and the way the user interacts with it.

What are Six Degrees of Freedom (6DOF)

"Six degrees of freedom" is a specific parameter count for the number of degrees of freedom an object has in three-dimensional space, such as the real world. It means that there are six parameters or ways that the object can move.

Six degrees of freedom consists of the following movement parameters:

• Translation – Moving along the different axes X, Y and Z
  • Moving up and down along the Y axis is called heaving.
  • Moving forwards and backwards along the X axis is called surging.
  • Moving left and right along the Z axis is called swaying.
• Rotation – Turning in order to face a different axis
  • Moving between X and Y is called pitch.
  • Moving between X and Z is called yaw.
  • Moving between Z and Y is called roll.

​Fig. Six degrees of freedom (Wikipedia Commons).
* Source – techopedia.

There are many possibilities of using a 6DoF VR technology. You can imagine exploring a movie plan in your own pace. You could stroll between the actors, look at the action from different sides, listen to any conversations and paying attention to what is interesting only for you. Such technology would provide really unique experiences.

​A wide spectrum of virtual reality applications drives the development of technology in the audio-visual industry. Until now, image-related technologies have been developing much faster, leaving the sound far behind. We have made the first attempts to show that 6DoF for sound is also achievable.

---

Read more about 6DOF Development Kit

---

How to record audio in 6DoF?

​It's extremely challenging to record high-quality sound from many sources present in the sound scene at the same time. We managed to do this using nine ZYLIA ZM-1 multi-track microphone arrays evenly spaced in the room.

In our experiment the sound field was captured using two different spatial arrangements of ZYLIA ZM-1 microphones placed within and around the recorded sound scenes. In the first arrangement, nine ZYLIA ZM-1 microphones were placed on a rectangular grid. Second configuration consisted of seven microphones placed on a grid composed of equilateral triangles.

Fig. ​Setup of 9 and 7 ZYLIA ZM-1 microphone arrays

​Microphone signals were captured using a personal computer running GNU/Linux operating system. Signals originating from individual ZM-1 arrays were recorded with the specially designed software. 

​We recorded a few takes of musical performance with instruments such as an Irish bouzouki (stringed instrument similar to the mandolin), a tabla (Indian drums), acoustic guitars and a cajon.

Fig. Three ZYLIA ZM-1 mics in a line. ZYLIA ZM-1 with synchronization mechanism attached

Fig. Recorded musicians – from left – Przemyslaw Sledziuha Sledz, Michal Obrebski and Kasia Mizerny

Unity and 3D audio

To present interesting possibilities of using audio recorded with multiple microphone arrays we have created a Unity project with 7 Ambisonics sources. In this simulated environment, you will find three sound sources (our musicians) represented by bonfires among whom you can move around. Experiencing fluent immersive audio becomes so natural that you can actually feel being inside of this scene.

MPEG Standardization Committee
​

​During exploration experiments, MPEG Standardization Committee has realized that there is a lack of test material for audio and video experiments into MPEG-I Immersive Video. Our recordings have been accepted as a reference material in MPEG Standardization Committee. Each of the experts involved in standardization works on MPEG-I can now use the Zylia recordings to develop and test audio technologies for VR/AR applications. 

​Our informal listening tests prove that it is possible to achieve satisfactory 6DoF effect (with restricted position of listening) by probing space using 3rd order Ambisonics microphones like ZYLIA ZM-1. We are getting closer to make the real VR more real😊

6DOF Development Kit by Zylia