Audiovisual Sonification — User Guide
A two-mode real-time audiovisual engine: seven tightly specified audio/visual mapping presets and a multi-voice generative composition mode in which visual horizontal position also controls stereo pan.
What this does
The script combines two related audiovisual workflows in one persistent Praat interface. AV Presets generates a fixed stereo test signal for one of seven mappings and animates a Demo window while the full sound plays continuously. Some presets derive the visual state from measurements of the generated Sound; others visualize the exact control function used by the synthesis. Poly Composition generates one to eight voices, animates geometric objects, and maps each voice's visual X trajectory directly to equal-power stereo pan.
No input Sound is required. Audio is synthesized internally at 44.1 kHz. The animation advances in nominal 80 ms frames (about 12.5 frames/s), while audio is launched once with asynchronous playback so the sound device is not restarted every frame.
Quick start
- Run
Audiovisual_Sonification.praat. No Sound selection is needed. - Choose AV Presets for one of the seven fixed mapping demonstrations, or Poly Composition for the multi-voice engine.
- Set Total duration, Master volume, and the phase/seed options. In Poly mode also choose shape, colour palette, movement, synthesis type and voice count.
- Click Run. The script synthesizes the full Sound first, starts one continuous asynchronous playback, and then animates the Demo window.
- After the run, inspect the reported number of late visual frames. The dialog then reopens for another run.
Audiovisual mapping and “sonification”
In its strict research sense, sonification means mapping data into non-speech audio so that information can be perceived through sound. This script uses the term more broadly for audiovisual parameter coupling; it does not import or sonify an external dataset.
The direction of mapping depends on the mode. In AV Presets, generated audio or its synthesis controls drive the image: presets 1, 2 and 7 use measurements of the generated Sound, while presets 3–6 expose the same control law that drives synthesis. In Poly Composition, the relationship is partly bidirectional: the movement system determines the visual position, and the same X coordinate is used as the stereo-pan control for that voice.
Parameters
| Parameter | Default | Meaning |
|---|---|---|
| Mode | AV Presets | Selects the fixed audiovisual demonstrations or the multi-voice generative engine. |
| Preset | 1. RMS → Size | Used only in AV Presets mode. |
| Shape | Circle | Poly visual geometry: Circle, Triangle, Square, Pentagon, Hexagon, Star, Plane, Points, or Lines. |
| Color palette | Rainbow | Poly visual colour mapping: Rainbow, Fire, Ice, Cyan, Magenta, Gold, or White. |
| Movement | Orbit | Poly trajectory: Orbit, Spiral, Bounce, Pendulum, or Expand / Contract. X also controls stereo pan. |
| Synthesis type | Pure Sine | Poly oscillator model: Pure Sine, FM, AM, Additive, or six-partial bright harmonic stack (“Pulse”). |
| Number of voices | 3 | Poly voice count. Values are clamped to 1–8. |
| Total duration | 4 s | Must be greater than zero; maximum 300 s. |
| Master volume | 0.85 | Clamped to 0–0.98. For a nonzero signal it is used as the target peak via Scale peak; 0 creates silence. |
| Show trails | Yes | Poly only. Stores and draws a 14-position trail for every voice. |
| Randomize initial phase | Yes | Draws one shared time offset in the range 0–100 s before synthesis/animation. The offset changes the initial state of the control functions. |
| Random seed | 0 | Positive values reproduce the randomized time offset. 0 uses the current unpredictable RNG state. Ignored when phase randomization is off. |
AV Presets
All seven presets create a two-channel master Sound. Presets 1–6 place the same synthesized signal in both channels; preset 7 deliberately generates different left and right oscillators.
| Preset | Audio | Visual mapping | Mapping type |
|---|---|---|---|
| 1. RMS → Size | 150 Hz sine with 1.5 Hz amplitude modulation. | Each frame measures channel-1 RMS over the corresponding audio window. Circle radius is normalized to the maximum RMS measured during that run; high normalized RMS turns the circle red. | Measured audio → visual. |
| 2. Pitch → Y-Position | Eight click-safe notes per second. The repeating pitch set is 200, 500, 800, 350 and 650 Hz. | Praat Pitch analysis uses a 5 ms time step, 75 Hz floor and 1000 Hz ceiling. Valid tracked F0 values define the run-specific vertical range; during dropouts the last valid pitch is held. | Measured audio → visual. |
| 3. FM Index → Star Sharpness | 300 Hz carrier, 600 Hz modulator. FM index follows a raised-cosine cycle from 0 to 5 at 0.5 Hz. | The same FM-index control changes the star's inner radius: low index gives a broad shape, high index a sharper star. | Synthesis control → visual. |
| 4. Voice Frequency → Swarm | Five voices centred at 200, 400, 600, 800 and 1000 Hz, each with a mathematically integrated ±100 Hz sinusoidal frequency deviation. Modulation rate is 0.1 × voice number Hz. | Each particle's Y coordinate is computed from that voice's instantaneous frequency. X follows a slower identity orbit. | Synthesis state → visual. |
| 5. Phase Modulation → Rotation | 100 Hz tone plus a 100.5 Hz tone whose phase is modulated by ±4π radians at 0.2 Hz. | The exact phase-modulation term contributes directly to the spiral rotation, together with a steady base rotation. | Synthesis control → visual. |
| 6. FM Index → Geometry | 250 Hz carrier, 500 Hz modulator, signed FM index 6·sin(2π·0.4t). | The signed index changes the trochoid parameter; its absolute magnitude also controls the displayed colour component. | Synthesis control → visual. |
| 7. Stereo Waveform → Path | Left: 220 Hz. Right: 330 Hz with π/4 phase offset. | A 20 ms window is sampled from the actual master Sound for every frame. Thirty-two measured L/R sample pairs form the Lissajous path. | Measured stereo audio → visual. |
Poly Composition
Poly mode assigns each voice a fixed pitch from an A2-based ratio set: 1, 9/8, 5/4, 3/2, 5/3, 2, 9/4, 5/2 multiplied by 110 Hz. Voice phases are evenly distributed around the cycle. A shared randomized time offset, when enabled, changes the initial audio and movement state without changing these pitch ratios.
Synthesis types
| Type | Implementation |
|---|---|
| Pure Sine | One sinusoid at the voice frequency. |
| FM | Carrier at the voice frequency with a modulator at 2× carrier and phase-modulation index 3. |
| AM | Voice sine multiplied by a 4 Hz unipolar 0–1 amplitude modulator. |
| Additive | Partials 1–4 with weights 0.55, 0.25, 0.13 and 0.07. |
| Pulse | Six harmonic partials with approximate 1/n weights: 1, 0.5, 0.33, 0.25, 0.20 and 0.17. This is the script's bright six-partial harmonic implementation. |
Before spatialization, each voice is scaled by 0.8 / √N. The complete stereo mix is then edge-faded and peak-scaled to the selected Master volume, so the final level setting is a target peak rather than a simple post-mix gain multiplier.
Movement and stereo coupling
Every movement model generates a visual position (X,Y). The audio connection uses X only: pan = X / 100. Left and right gains are complementary equal-power gains √(1-pan) and √pan. Consequently, the visible horizontal trajectory and stereo motion are mathematically the same control path.
| Movement | Visual behavior | Audio consequence |
|---|---|---|
| Orbit | Circular orbit with 35-unit radius. | Cosine horizontal motion becomes stereo pan. |
| Spiral | Radius expands and contracts while the point rotates. | The changing spiral X position becomes pan. |
| Bounce | Voice-dependent Lissajous-like X/Y rates. | Only the X sine controls pan. |
| Pendulum | Coupled X/Y sinusoidal motion. | The X pendulum trajectory controls pan. |
| Expand / Contract | Each voice remains on its radial angle while radius breathes in and out. | The resulting X projection controls pan. |
What remains visual-only
Shape, colour palette, visual brightness, trail drawing and the Y coordinate do not alter synthesis. Shapes breathe and rotate visually; colour brightness follows a 0.5 Hz voice-phased modulation. Trails contain 14 stored positions and are drawn from older/dimmer to newer/brighter points.
Timing, level and reproducibility
- Sample rate: fixed at 44,100 Hz.
- Animation step: 0.08 s, nominally 12.5 frames/s. Frame count uses
ceiling(duration / 0.08), so a partial final frame is included. - Continuous playback: the full master Sound is started once with asynchronous playback; animation is paced separately with
sleep(). - Late frames: if rendering consumes more than a frame budget, that visual frame is counted as late. Audio continues independently, so visual timing can fall behind on a slow run without forcing audio-device restart gaps.
- Edge fade: a global linear fade of up to 10 ms is applied at both ends before peak scaling.
- Master volume: values above 0 are target peak values. The script clamps the requested value to 0.98 maximum.
- Seed: a positive seed reproduces the randomized 0–100 s initial time offset. After drawing the offset, Praat's global RNG is restored to unpredictable initialization.
Playback and outputs
The script is designed for live audiovisual demonstration rather than persistent object creation. In both modes the Sound named internally as the master is synthesized, played asynchronously, used during animation, and then removed. No audio file is written and no final Sound remains selected in the Objects window.
The visible result is the Praat Demo window. At the end of a run it reports AV run complete or Composition complete together with the number of late visual frames. The parameter dialog then returns automatically because the script runs inside a persistent repeat … until 0 session.