CHI 97 Electronic Publications: Late-Breaking/Short Demonstrations

CHI 97 Electronic Publications: Late-Breaking/Short Demonstrations

The Magic Carpet: Physical Sensing for Immersive Environments

Joseph Paradiso, Craig Abler, Kai-yuh Hsiao, Matthew Reynolds
MIT Media Laboratory
20 Ames St. E15-410
Cambridge, MA. 02139 USA
+1 617 253 8988
joep@media.mit.edu, cba@mit.edu, khsiao@mit.edu, matt@media.mit.edu

ABSTRACT

An interactive environment has been developed that uses a pair of Doppler radars to measure upper-body kinematics (velocity, direction of motion, amount of motion) and a grid of piezoelectric wires hidden under a 6 x 10 foot carpet to monitor dynamic foot position and pressure. This system has been used in an audio installation, where users launch and modify complex musical sounds and sequences as they wander about the carpet. This paper describes the floor and radar systems, quantifies their performance, and outlines the musical application.

Keywords

Doppler radar, PVDF, piezoelectrics, immersive environment, musical interfaces, foot sensing

ABSTRACT
Keywords
INTRODUCTION
THE SENSOR FLOOR
THE DOPPLER RADARS
MUSICAL MAPPINGS
ACKNOWLEDGMENTS
REFERENCES

INTRODUCTION

Many systems have been developed to monitor the position and motion of people for various applications; e.g. motion capture, security, interactive/virtual-reality environments, etc. The majority of tetherless people-sensing systems in the virtual reality community have used computer vision techniques, e.g. [1], where data from a set of video cameras is processed to obtain real-time tracking information. Although this technology will continually improve as hardware and algorithms increase in capability, most current real-time vision systems still have significant drawbacks, such as slow response and limited robustness to changing lighting conditions or clutter. For many applications, such as intruder detectors [2] or the system described here, the fine-grained information available from a video camera is unnecessary or potentially inadequate, hence other sensor channels that more directly measure the parameters of interest are better applied.

The system described in this paper was initially developed to support the Brain Opera [3], a large, touring musical installation, where a general audience can interact with musical sound and structure at a variety of interactive stations that exploit many sensing technologies to map different kinds of physical gesture into musical expression and graphics. One of the interactive environments conceived for this project involved creating a space where the position and pressure of a performer's feet would be measured together with upper-body and hand motion. This data would be used to create a truly "immersive", tetherless musical environment, where any kind of body motion would be directly and immediately converted into expressive sound.

This system has indeed been constructed, and although it does not currently tour with the Brain Opera, it has been used in several musical installations. The sensor systems used to measure the feet and upper body are described below, as is the interactive musical application created to demonstrate this environment.

Figure 1: Sensor Arrangement for the Carpet System

THE SENSOR FLOOR

Figure 1 shows the physical sensor layout for the carpet and radar systems, as employed in this installation. A 16 x 32 grid of piezoelectric wires, running across the carpet at a roughly 4" inter-wire pitch, is used to sense foot pressure and position. Although this wire appears as standard RG-174, 0.1" diameter, shielded, coaxial cable, the insulation between center and outer conductors is made from PVDF (polyvinylidene fluoride) polymer [4], which is a commonly available piezoelectric material, often used as a transducer. This cable will thus produce a voltage when pressed or flexed anywhere along its length. Previous sensor-floor interfaces [5] have measured foot pressure with force-sensitive resistor sheets; the PVDF wire, however, is much more rugged, readily available, and exhibits a much higher dynamic range in pressure response.

The signal from each PVDF wire under the carpet is buffered by a high-impedance operational amplifier, and a pressure profile is produced by a simple diode/capacitor envelope detector. A 68HC11 microprocessor scans the resulting signals via a multiplexer (the unit that was constructed scans up to 64 wires 60 times per second), digitizing each into 8 bits. Whenever a new peak in pressure is detected, the processor sends out a MIDI (Musical Instrument Digital Interface) Note-On event, with the note number corresponding to the particular wire generating the data, accompanied by a 7-bit pressure value sent as the note velocity. Corresponding Note-Off events are sent when the pressure value from a formerly active wire decays back to the baseline. The sensor floor is very responsive; the transmitted velocity data easily distinguishes between soft foot motion and hard impacts, while careful shielding of the pickup wires and electronics virtually eliminates stray pickup and crosstalk.

THE DOPPLER RADARS

Although the PVDF wires do a good job of measuring foot dynamics and position, another system was necessary to complete the immersive sensing by tracking movement of the arms and upper body. A pair of microwave motion sensors, as indicated in Figure 1, were used for this task.

The sensor heads are composed of a simple, inexpensive circuit board containing a single-transistor, 2.4 gigaHertz (GHz) CW oscillator, coupled to a 4-element micropatch antenna, which forms a broadside beam roughly 20deg. in width (although with significant sidelobes). As the radiated output is below 10 milliwatts, this system is entirely safe and well within regulation. Since nonconductive material does not significantly absorb this signal, these antennas can be easily hidden behind walls, projection displays, etc.

Doppler-shifted reflections from a performer moving within the beam return to the antenna, where they are mixed with the transmitted signal in a hot carrier diode. This produces beat frequencies in the range of 0-5 kilohertz (kHz) that directly represent the performer's dynamic state (the frequency is a function of velocity, and the beat amplitude is a combined function of the size and distance of the reflecting object). Two such diodes, placed roughly an eighth-wavelength apart, produce a quadrature pair of signals, thus their correlation determines the direction of motion along the antenna boresight. These radars respond to motion within a range of at least 15 feet.

Rather than process the Doppler signals in the Fourier domain, a simple analog signal conditioner was designed to minimize real-time computing requirements. This circuit produces three analog signals for each radar head. One of these is just the low-pass filtered amplitude envelope of the Doppler beats; this corresponds to the amount of general motion that the radar detects. Another is derived by first high-pass filtering the Doppler beats before detecting the envelope; the amplitude of this signal corresponds to the detected velocity. A third signal is derived from an analog correlation between the signals produced by the diode pair; the polarity of this voltage indicates the direction of detected motion. These 3 signals were 8-bit digitized at roughly 50 Hz, and directly used by the music-generating algorithm.

MUSICAL MAPPINGS

The sensor hardware is connected to a PC running music software written in Visual C++. MIDI output is sent to a Roland Sound Expansion module. The music generated by the system consists of a low voice, a middle voice, and a high voice. The low voice basically acts as a pedal point; a single steady low note is triggered by stepping on the carpet and is sustained for a length of time. The middle voice, also triggered by the carpet, plays harmonizing fifths. The pitch and panning of the middle voice is controlled by the user's position on the carpet. The high voice is a twinkling melody line whose speed, pitch, and panning is controlled by the motion detected by the radars, and whose timbre and structure is controlled by the carpet. Finally, the direction of movement detected by the radar units controls the chord on which everything is played.

The overall, combined effect of the entire installation is a relaxing soundscape that responds to subtle movements on the part of the performer. The sound mappings allow for a good deal of expression, yet they are intuitive and simple enough for players to immediately appreciate the connection between their movements and the sound produced. The complete "Magic Carpet" system is shown in Figure 2, as installed in an elevator lobby at the MIT Media Laboratory.

Figure 2: The complete Magic Carpet, as installed at MIT

ACKNOWLEDGMENTS

We thank Tod Machover and the Brain Opera team, Neil Gershenfeld and the Physics and Media Group, and our other MIT Media Lab colleagues for many helpful discussions. Vic Chatigny and Dave Port at AMP Sensors are thanked for supplying the PVDF wire. We acknowledge the support of the Things That Think Consortium and our other sponsors at the MIT Media Laboratory.

REFERENCES

1. Maes, P., et al. The ALIVE System: Full-body Interaction with Autonomous Agents. Proceedings of Computer Animation `95, IEEE Press, 1995.

2. Petruzzelis, T. The Alarm, Sensor, & Security Circuit Cookbook. McGraw-Hill/TAB, New York, 1994.

3. Paradiso, J. New Instruments and Gestural Sensors for Interactive Music Performance. Submitted to the 1997 International Computer Music Conference (ICMC).

4. Paradiso, J. The Interactive Balloon: Sensing, Actuation, and Behavior in a Common Object. IBM Systems Journal, Vol. 35, Nos. 3&4, 1996, pp. 473-487.

5. Pinkston, R., et. al. A Touch Sensitive Dance Floor MIDI Controller. Proc. of the ICMC, 1995, pp. 224-225.

CHI 97 Electronic Publications: Late-Breaking/Short Demonstrations