3DAudio – on mobile devices

Hear that chopper closing in on your troops? Turn your head and see… 3D Sound in your cellphone? Read this article to find out how it is done and how you make it happen on your phone.

The ability to simulate spatial sound allows listeners to experience multimedia applications in “a captivating and natural way” [1]. Spatial sound or 3DAudio is possible through a mathematical model that simulates how sound actually enters the human ear and head. The Advanced Multimedia Supplement API, i.e. JSR234 [2], provides these capabilities for the Java Micro Edition developer and is now becoming available on a number of Java enabled devices (like the Nokia S60 3rd Edition and Sony Ericsson JP-8 Java platform).

Human hearing [3]

To understand 3DAudio we need to understand some basic principles behind hear-
ing and how humans perceive and interpret sound. We remember from school how
sound waves are captured by the outer-ear and collected and transported through
the auditory canal to the eardrum. The eardrum starts to vibrate and these vibra-
tions are transferred to the brain through a sophisticated system of evolutionary
designed bones, membranes and fluids. The brain interprets these signals and we are
able to hear. Now, sound is not only collected through the pinnae but also through
the head and torso. This is important for the perception of 3D audio since sounds
take different paths through the body depending on the sounds origin.
Basic 3D audio perception is based on two principles: inter-aural intensity IID and inter-aural time differences ITD.

Figure 1
Figure 1. Inter-Aural Intensity Difference (IID)

IID is the fact that the sound intensity is higher at the ear that is closer to the sound source. The signal that hits the other ear is also shadowed by the listener’s head (see figure 1). ITD is the time difference between a sound arriving at one ear before the other, that is, unless the sound source is located at the exact same distance from both ears (see figure 2).

Figure 2
Figure 2. Inter-Aural Time Difference (ITD)

Synthetic HRTFs

Before the sound reaches the eardrum it bounces off a listeners head, torso and outer ear. These reflections create a unique pattern for the sound location in space relative to the listener. The patterns are summarized in a set of acoustic filters called Head Related Transfer Functions (HRTFs).
The details of the actual implementations are often proprietary and are therefore
hidden from the public domain, but most 3D audio systems are based on the con-
cepts of HRTFs. HRTFs are audio filters for each ear that contain all the listening
cues as the sound travels from the source, through the environment, and finally ar-
rives at the listener’s ear drum.

3DAudio for Java ME

The 3D world is setup in a similar fashion which is used when designing 3D graph-
ics. We have a right-handed coordinate system and use Cartesian coordinates to
specify the location of sound source objects in our 3D audio scene. We specify a
listener or spectator that does the actual listening, and are able to move the sound
source and the spectator around in the virtual acoustic space. Furthermore, distance
attenuation, Doppler and orientation effects are applied to our sound objects, and
to even further enhance the 3D audio user experience we can use reverb and equal-
izer effects.

Figure 3
Figure 3. Right-handed coordinate system

The first we need to do when we design our virtual acoustic space or audio scene is to create some audio players. In Java ME we use the class Manager from the multimedia API, i.e. JSR135 [4].

The class GlobalManager lets us create SoundSource3D object which represents a sound
source in virtual acoustic space.

The listener we place at the origin, x = 0, y = 0, z = 0. The package “javax.microedition.
amms.control.audio3d” is shortened to “javax…”.

We now have a listener and two sound sources from which we can create controls to ma-
nipulate these objects. We place the helicopter sound 250 meters up and 250 meters to the left of the listener. The coordinates are specified in millimetres.

The car is placed 100 meters to the right of the listener.

We also want to specify the distance attenuation for the helicopter sound source. With
this control we can specify the maximum distance and the minimum distance for the at-
tenuation of the sound. We should mute after the sound source has passed the maximum
distance, and where we think the distance does not affect the attenuation. The roll off factor is specified in thousands and higher values means faster attenuation.

To make the listener to be rotated to the left, looking to the sky where the helicopter is, there is an OrientationControl which can be created from the listener. We use heading 30 degrees around the y-axis and roll 15 degrees around the z-axis so we can see where the helicopter sound is originating.

Figure 4. Orientation
Figure 4

We now have an audio scene in virtual acoustic space. All we need to do is to start playing the sound, put our headphones on and listen.

Audio effects such as reverb and equalizer are also part of the JSR234. The ReverbControl
is retrieved from GlobalManager, and there are a number of presets available that we can
use to improve the user experience of a 3D audio world.

Using 3DAudio

Psycho-acoustic research shows that humans tend to hear what they see. This means that
we need to connect 3D graphics and sound to make the experience of 3D sound com-
plete. Java ME lets us do just that. The use of 3D audio in real applications has in the past
been limited to games and to some extent it has been used in commercial video clips. The next natural step is to use 3D audio on the 3D desktop. The combination of 3D graphics techniques and 3D audio will improve user interaction on hand-held devices and desktop computers. Cell phones are often used together with headphones today. This means that 3D audio techniques could and, according to me, should be used, to improve usability on modern cellular phones.

References

[1],[3] 3D Audio Rendering and Evaluation Guidelines, version 1.0, MIDI Manufacturers Association, Los Angeles CA. 1998
[2] JSR234: Advanced Multimedia Supplements, http://jcp.org/en/jsr/detail?id=234
[4] JSR135: Mobile Media API, http://jcp.org/en/jsr/detail?id=135

Originally published in JayView.

Leave a Reply

Close Menu