Motion Control — Gesture-to-Sound Mapping

Captures free-hand motion from a webcam (energy, vertical/horizontal position, speed, stillness, radius, acceleration) and maps four user-defined slots to amplitude, pitch, spectral brightness, and stereo pan. Optional live audio preview during capture. Offline render uses the same mapping graph for reproducible results.

Author: Shai Cohen Affiliation: Department of Music, Bar-Ilan University, Israel Version: 1.4 (2026) License: MIT License Repo: https://github.com/ShaiCohen-ops/Praat-plugin_AudioTools

Contents:

What this does Quick start 7 Presets 7 Gesture Sources 4 Mapping Slots Applications

What this does

This script implements gesture-to-sound mapping using a webcam. The Python worker opens the camera, calibrates a background model (2 seconds of stillness), captures motion, and derives seven normalized gesture sources: energy, vertical/horizontal position, speed, stillness, radius from centre, and acceleration. Four user-editable mapping slots route those sources to amplitude, pitch, spectral brightness, and equal-power stereo pan. Optional live audio preview during capture provides responsive rehearsal monitoring.

How it works: The Python worker uses frame differencing and motion-weighted centroid tracking. Background calibration (2s stillness) builds a per-pixel noise model. During capture, each video frame is analysed for motion energy and the centre of motion (X,Y). These raw signals are smoothed, normalised, and mapped through user-defined slots to control amplitude (attenuation only), pitch (semitone shifts), brightness (HPF modulation), and stereo pan (equal-power). The final offline render applies these controls to the selected Praat Sound. Live audio preview uses the same mapping graph with causal normalisation.

Key Features:

7 Gesture Sources — Motion energy, Vertical position, Horizontal position, Speed, Stillness, Radius from centre, Acceleration
4 Mapping Slots — Each slot maps one source to one destination (Amplitude, Pitch, Brightness, Stereo pan) with amount and invert
7 Performance Presets — Subtle, Expressive, Spatial, Spectral, Kinetic, Wild, Meditative, plus Custom
Optional Live Audio Preview — Hear your gesture control in real time during capture (requires sounddevice)
Offline Render — Applies the same mapping to the selected Sound after capture (authoritative result)
Fallback Handling — If the webcam fails, the script bypasses transformation and copies the source unchanged
Visualisation — Input/output waveforms, control curves (amplitude, pitch, brightness, pan), and mapping summary

Live vs. Offline: Live preview is a responsive rehearsal monitor — it uses causal normalisation and a granular pitch approximation. The offline Praat render is the authoritative final result, using take-relative normalisation and the full Praat pitch/amplitude/brightness processing chain. A failed camera capture bypasses transformation entirely (source copied unchanged).

Quick start

In Praat, select exactly one Sound object.
Run script… → MotionControl.praat.
Choose a Performance_character (Subtle, Expressive, Spatial, Spectral, Kinetic, Wild, Meditative, or Custom).
Enable Live_audio_during_capture to hear real-time preview (requires sounddevice).
Optionally enable Edit_mappings to configure the four mapping slots (source → destination, amount, invert).
Click OK — a webcam preview opens. Hold still for 2 seconds (calibration), then move freely for 3–60 seconds (capture duration matches your Sound).
The script applies the mapping to your Sound, imports the result as originalname_motion.

Quick tip: Start with Expressive for balanced mapping (energy → amplitude, vertical → pitch, horizontal → brightness, horizontal → pan). Spatial maps horizontal → pan, radius → amplitude, vertical → brightness, speed → pitch — good for immersive movement. Wild maximises all mappings for dramatic effects. Enable Edit_mappings to create custom routes — e.g., speed → pitch for acceleration-based vibrato, stillness → amplitude for "freeze" effects.

Important: Python dependencies required: pip install numpy opencv-python. For live audio preview: pip install sounddevice. The capture duration matches your Sound's duration (3–60 seconds). If your Sound is shorter than 3 seconds, capture is 3 seconds and the gesture is time-scaled to the Sound. If longer than 60 seconds, capture is 60 seconds and the gesture is time-scaled. If the webcam fails, the script bypasses transformation — the source is copied unchanged and the Info window shows "FALLBACK".

7 Performance Presets

Preset	Slot 1	Slot 2	Slot 3	Slot 4	Smooth	Character
Subtle	Energy→Amp 0.5	Vertical→Pitch 0.25	Horizontal→Bright 0.4	Horizontal→Pan 0.45	9	Gentle, refined movements
Expressive	Energy→Amp 0.8	Vertical→Pitch 0.5	Horizontal→Bright 0.8	Horizontal→Pan 1.0	5	Balanced, musical default
Spatial	Horizontal→Pan 1.0	Radius→Amp 0.45	Vertical→Bright 0.35	Speed→Pitch 0.25	5	Immersive spatial mapping
Spectral	Horizontal→Bright 0.8	Vertical→Pitch 0.35	Energy→Amp 0.55	Radius→Bright 0.35	6	Spectral focus
Kinetic	Speed→Pitch 0.55	Acceleration→Bright 0.9	Energy→Amp 0.85	Horizontal→Pan 0.8	4	Dynamic, gesture-driven
Wild	Energy→Amp 0.92	Vertical→Pitch 1.0	Horizontal→Bright 1.2	Horizontal→Pan 1.0	3	Dramatic, wide-range
Meditative	Stillness→Amp 0.3	Vertical→Pitch 0.15	Horizontal→Pan 0.3	Radius→Bright 0.2	18	Slow, narrow, inertia-heavy

7 Gesture Sources

Motion Energy

Frame-to-frame pixel change after background subtraction. High energy = fast, broad movement. Low = stillness.

Unipolar — 0 (still) to 1 (maximum motion).

Vertical Position

Motion-weighted centroid in Y (top = 1, bottom = 0). Tracks where motion happens vertically.

Bipolar — 0.5 centre, 1 top, 0 bottom.

Horizontal Position

Motion-weighted centroid in X (left = 0, right = 1). Tracks lateral motion.

Bipolar — 0.5 centre, 1 right, 0 left.

Speed

Rate of change of position (derivative of X/Y). Fast gestures = high speed.

Unipolar — 0 (still) to 1 (fast).

Stillness

Complement of energy and speed: 1 when still, 0 when moving.

Unipolar — 1 (still) to 0 (moving).

Radius from Centre

Distance from the centre of the frame (0 at centre, 1 at edge).

Unipolar — 0 (centre) to 1 (edge).

Acceleration

Rate of change of speed (second derivative). Jerky movements = high acceleration.

Unipolar — 0 (constant speed) to 1 (rapid changes).

Source character: Bipolar sources (Vertical/Horizontal) are centred at 0.5 — mapping amount 1.0 gives ± full range around centre. Unipolar sources (Energy/Speed/Stillness/Radius/Acceleration) start at 0 — amount 1.0 gives 0 to +full range. Invert flips the source (1 → 0, 0 → 1).

4 Mapping Slots — Source → Destination

Amplitude

Unipolar only: source 0–1 → gain 1–(1-amount). Attenuation only (no boost). Multiple slots multiply.

Example: Energy → Amplitude amount 0.8: still → gain 1.0, full motion → gain 0.2 (80% attenuation).

Pitch

Bipolar: source maps to ±pitch_span_st. Unipolar: source maps 0→+pitch_span_st.

Example: Vertical → Pitch amount 1.0: top → +12 st, bottom → −12 st. Energy → Pitch: still → 0 st, full motion → +12 st.

Brightness

Bipolar: source maps to ±brightness_span (HPF gain). Unipolar: source maps 0→+brightness_span. Negative side limited to -1.

Example: Horizontal → Brightness: right → +0.8, left → −0.8. Radius → Brightness: centre → 0, edge → +0.8.

Stereo Pan

Bipolar: source maps to ±1 (equal-power). Unipolar: source maps 0→+1.

Example: Horizontal → Pan: left → −1, centre → 0, right → +1.

Mapping evaluation (vectorised):

Each slot: value = clamp(source, 0, 1)
If invert: value = 1 - value
Amplitude: gain *= 1 - amount × (1 - value)
Pitch/Brightness/Pan: mod = value - 0.5 (bipolar) or mod = value (unipolar)
control += mod × amount × range

Multiple slots to the same destination are summed (except Amplitude, which multiplies).

Applications

Expressive performance (Expressive preset)

Use case: Perform a sound using natural gestures — energy controls volume, vertical controls pitch, horizontal controls brightness and pan.

Settings: Expressive preset. Move your hand up for higher pitch, right for brighter timbre and panning, faster for louder volume.

Spatial immersion (Spatial preset)

Use case: Place a sound in stereo space using hand position, with radius controlling amplitude and speed controlling pitch.

Settings: Spatial preset. Horizontal position controls pan, radius from centre controls volume (centre = quiet, edge = loud).

Kinetic / rhythmic control (Kinetic preset)

Use case: Speed and acceleration drive the sound — fast gestures create pitch and brightness changes.

Settings: Kinetic preset. Speed → Pitch (faster = higher), Acceleration → Brightness (jerky = brighter).

Meditative / slow control (Meditative preset)

Use case: Stillness controls amplitude (freeze when still), with slow, smooth responses.

Settings: Meditative preset. Stillness → Amplitude (still = loud, moving = quiet), heavy smoothing (18 frames).

Workflow: Voice → Expressive gesture performance

Source: Vocal recording.
Settings: Expressive preset, live audio on.
Result: Sing into the microphone while gesturing in front of the webcam — the voice follows your hand: up = higher pitch, right = brighter/pan, energy = volume.

Workflow: Synth pad → Spatial immersion

Source: Sustained synth pad.
Settings: Spatial preset, live audio on.
Result: Move your hand left-right to pan the pad, in/out (radius) to control volume, up/down to control brightness.

Workflow: Custom mapping → Speed vibrato

Source: Any monophonic sound.
Settings: Custom, Slot 1: Speed → Pitch amount 0.6, Slot 2: Energy → Amp amount 0.5.
Result: Fast hand movements create pitch vibrato; energy controls amplitude. A gestural tremolo.

Troubleshooting:
• Webcam not opening: Check that your camera is connected and not in use by another app. On macOS, grant camera permission to your terminal/Praat. On Linux, ensure video devices are accessible.
• Live audio preview not working: Install sounddevice: pip install sounddevice. Check your audio output device. The preview uses the same mapping graph as offline render but with causal normalisation.
• Tracking confidence low: Improve lighting, use a plain background, wear contrasting clothing. The script reports confidence; <30% triggers a warning.
• Output is silent / amplitude too low: Amplitude mapping attenuates only (gain ≤ 1). If energy → amplitude amount is high, loud gestures attenuate the signal. Increase amplitude_max or reduce mapping amount.
• Pitch shifts are too extreme: Reduce pitch_span_st in the Details editor (default 12 st). The Details editor also controls brightness span and HPF cutoff.
• Fallback mode (source copied unchanged): If the camera fails, the script bypasses transformation entirely. Check the Info window for the "FALLBACK" message and the camera error.

Visualisation

When Draw_visualization is enabled, the script generates:

Original waveform — grey, time-aligned
Output waveform — blue, mapped result
Amplitude gain curve — green, 0–1 attenuation envelope
Pitch shift curve — blue, semitones over time
Brightness control curve — purple, HPF modulation
Stereo pan curve — violet, -1 L / 0 C / +1 R
Mapping summary — slot listing, tracking confidence, camera FPS, capture/sound duration

The control curves show exactly how your gesture mapped to each destination.