Ikeda-inspired Audiovisual Studies — User Guide
Four compact Praat audiovisual studies inspired by selected principles in Ryoji Ikeda's work. Each study derives sound and image from a shared data or control representation and uses periodic hard re-synchronization to keep audiovisual timing error bounded.
Scope
This script does not reconstruct, emulate, or model Ryoji Ikeda's works at sample level. It creates four small audiovisual studies that investigate selected ideas associated with test pattern, datamatics, spectra, and supercodex.
shared data/control object → audio mapping
shared data/control object → visual mapping
The important relationship is therefore not merely “a sound happens when an image happens.” In three of the four studies, the same generated data directly determines both domains. In the spectra-inspired study, one analytic energy control drives both the sound field and the visual beam.
Quick start
- Run
Ikeda_audiovisual.praat. No input Sound is required. - Choose one of the four Studies.
- Set Duration, Audio sample rate, Visual frame rate, and Sync block duration.
- Optionally choose a fixed Random seed, adjust Master volume, enable color accents, or enable High flicker mode.
- Run the script. The full stereo master Sound is generated first, then divided into frame-aligned synchronization blocks for audiovisual playback.
- After playback, the generated master Sound remains in Praat's Objects window for analysis, reuse, or saving.
Controls
| Control | Range / default | Behavior |
|---|---|---|
| Study | 4 choices | Selects the shared-data mapping and visual grammar. |
| Duration | 0–120 s; default 16 s | Exact master-Sound duration. The visual frame count uses ceiling(Duration × fps), so the final partial frame is retained. |
| Audio sample rate | 16–192 kHz; default 44.1 kHz | Direct synthesis rate of the retained stereo master Sound. |
| Visual frame rate | 4–20 Hz; default 8 Hz | Target Demo-window update rate. The frame duration is 1 / frame rate. |
| Sync block | 0.20–2.0 s; default 0.5 s | Requested duration of each periodic hard-sync interval. It is aligned to an integer number of visual frames. |
| Random seed | 0 or positive integer | 0 = unpredictable. Positive values reproduce generated binary words and numerical-grid data. The spectra-inspired audio/beam trajectory itself is deterministic. |
| Master volume | >0 to 2; default 0.75 | Multiplies the completed master Sound before down-only peak protection. |
| Enable color accents | on/off | Affects only selected visual accents; it does not change the audio. |
| High flicker mode | on/off | Removes the default multi-frame word hold in the two binary studies. |
Frame-aligned sync duration
The requested Sync block is converted to an integer number of frames:
For the defaults, 0.5 s × 8 fps = 4 frames, so the aligned block remains exactly 0.5 s.
Audiovisual synchronization
Version 2.1.3 uses the hybrid synchronization architecture introduced in v2.1.2. The complete master audio is generated first and then pre-sliced into frame-aligned sync blocks. Playback is asynchronous inside each block, but a new block creates a periodic hard re-lock.
This is a compromise between two earlier approaches:
- One asynchronous playback for the whole piece: smoother audio, but timing drift could accumulate throughout the entire work.
- One blocking audio chunk per visual frame: exact frame/chunk lockstep, but repeated playback starts can make the audio less continuous.
With the hybrid method, timing error can develop only inside the current sync block. It is reset when the next block begins.
Late-frame policy
If a frame is more than 80% of one frame period late, the stale visual frame is skipped so the image can catch up to the audio clock. A frame more than 25% late but still rendered is counted as “late.” Both counts are reported at the end.
After the final sync block, the final data frame is held for a short latency guard of approximately half a frame, bounded to 40–150 ms, to absorb a small output-device tail.
The four studies
1. test pattern-inspired Binary Transduction
A generated 16-bit integer is the shared data object. The same word determines the barcode and several audio properties.
| Data feature | Audio mapping | Visual mapping |
|---|---|---|
| 16 individual bits | Indirectly contribute to Hamming weight and transition count. | Sixteen vertical barcode cells: white for 1, dark for 0. |
| Number of 1-bits | Pitch: 1800 + 570 × HammingWeight, limited by 0.45 × Fs. | Reported in the HUD. |
| Bit-to-bit transitions | Controls event amplitude and tone/noise mixture. | Reported in the HUD. |
| Normalized 16-bit value | Equal-power stereo pan from 0.05 to 0.95 and starting phase. | The complete word is displayed as the barcode state. |
Each frame produces a short event of up to 42 ms with a Hann envelope. The source combines a sine tone and Gaussian noise using equal-power tone/noise gains.
2. datamatics-inspired Data Scan
A seeded 12×12 matrix of random values is generated once. The scan advances one cell per visual frame and the active cell value drives both audio and image.
| Grid property | Mapping |
|---|---|
| Cell value, 0–1 | Logarithmic pitch map from approximately 180 Hz toward 8 kHz before sampling-headroom limiting; also controls amplitude, event duration, and starting phase. |
| Grid column | Equal-power stereo pan from 0.05 to 0.95. |
| Full 12×12 field | Displayed as grayscale cells, with the scanned cell outlined and a vertical scan marker. |
Each scanned cell generates one short Hann-windowed sine event. The scan wraps after all 144 cells.
3. spectra-inspired White-Light Field
This mode is deliberately an abstraction of spectra, not a screen reconstruction of a large-scale white-light installation. A single slow energy trajectory is shared by sound and image:
The stereo audio is intentionally minimal. The left channel contains 50 and 50.8 Hz components; the right contains 50.3 and 51.1 Hz components. Each side also carries a weak high-frequency sine component.
v2.1.3 high-frequency headroom fix
The nominal high-frequency pair is 9800 / 9813 Hz. At low output sample rates those frequencies could exceed safe sampling headroom, so v2.1.3 applies one common scale:
This preserves the pair's relative frequency geometry while avoiding aliasing from the previously fixed 9.8-kHz components. At the default 44.1-kHz sample rate, no scaling is needed.
4. supercodex-inspired Data Pulse Field
A 16-bit word is decomposed into sixteen ordered micro-events within each visual frame. The visual left-to-right barcode order is exactly the temporal order of those audio micro-events.
| Bit value | Frequency | Amplitude |
|---|---|---|
| 0 | 800 + 310 × bitIndex, limited by safeTop | 0.022 |
| 1 | 1600 + 620 × bitIndex, limited by safeTop | 0.070 |
Both zeros and ones are sonified, so the entire word is represented rather than only its set bits. Bit index also determines equal-power pan from 0.03 to 0.97. Each micro-event has a Hann envelope and a maximum duration of 1.8 ms or 60% of its sub-slot, whichever is smaller.
Color accents and flicker
Enable color accents
This control does not globally recolor the visualization. Its current effects are specific:
- datamatics-inspired Data Scan: the active-cell outline and vertical scan line become cyan instead of white.
- supercodex-inspired Data Pulse Field: markers under 1-bits become cyan.
- test pattern-inspired Binary Transduction: no color change.
- spectra-inspired White-Light Field: remains grayscale/white-light based.
High flicker mode
Only the two binary studies use the shared-word hold. With High flicker off:
At the default 8 fps, one binary word is therefore held for 2 frames, giving about four binary state changes per second. High flicker mode sets holdFrames = 1, allowing a new word every frame.
Audio output and level
| Property | Behavior |
|---|---|
| Input | No input Sound is used. |
| Output channels | Always stereo. |
| Duration | Exactly the requested Duration. |
| Sample rate | Selected 16–192 kHz rate; synthesis occurs directly at this rate. |
| Master volume | Applied once to the complete generated stereo master. |
| Peak protection | Automatic down-only protection: if the post-volume peak exceeds 0.98, the complete master is scaled down to 0.98. |
| Normalization | No upward normalization. If peak ≤ 0.98, the user-selected level is preserved. |
| Output object | IkedaStudy_<study name>, retained in Praat after audiovisual playback. |
Synchronization and output QC
The Info window reports the conceptual scope, duration, sample rate, target frame rate, requested and aligned sync-block durations, number of frames and blocks, binary hold length, seed, Master volume, and pre/post-protection peak/RMS.
After playback it also reports:
- Completed sync blocks.
- Skipped stale visual frames.
- Late but rendered frames.
- Block overruns.
- Maximum local frame lateness.
- Final visual latency guard.
Official project references
The conceptual descriptions above are intentionally limited to claims supported by Ryoji Ikeda's official site:
- test pattern — official project page. Ikeda describes conversion of data/audio signals into binary or barcode patterns and the project's concern with device-performance and perceptual thresholds.
- datamatics — official project page. The project uses pure data as source material for sound and visuals and includes real-time computation and data scanning.
- spectra — official project page. A series of site-specific large-scale installations using intense white light, often together with sine-wave sound fields.
- supercodex — documented on Ikeda's official recordings archive as the 2013 album and on the concerts archive as a live set. The present 16-bit barcode/pulse mapping is specific to this AudioTools study.