The Sound of Standing is an experiment in sensory augmentation and movement sonification that translates body sway movement during standing, of which we are usually unaware, into sound. It is largely inspired by the early sensory substitution experiments by Paul Bach-y-Rita.

In this research, the sway movement is captured with an accelerometer and mapping into various parameters controlling a vowel-formant synthesiser as a sonification method. A series of experiment are conducted to evaluate the effectiveness of different mapping strategies.

Sonic impression:

The audio feedback system showed positive effects on postural control. Most subjects preferred pitch decrease as they swayed forward, because it simulates an natural scream as they fall forward. The continuous scaling function was reported as more responsive despite that there was no reported perceptual difference between the two scaling functions tested in experiment 2. The panning technique was adopted after experiment 3 because more subjects found it easier to control comparing with binaural technique. Also, the non-individualised Head Related Transform Function (HRTF) may resulted in perceptual differences among individuals. Lastly, more subjects thought that the choice of associating pitch change with front/back sway and timbre change with left/right sway was more informative than timbre change with front/back sway and pitch change with left/right sway.

The Sound of Standing is part of my master graduation project for Media Technology program at Leiden University. Read the full thesis here.