Spectral Image Sonification — User Guide

RGB-column sonification by additive synthesis. Horizontal image position becomes time; sparsely sampled rows are averaged into red, green, and blue control curves; those curves drive three interleaved harmonic families, while red–blue balance controls constant-power stereo position.

Author: Shai Cohen Affiliation: Department of Music, Bar-Ilan University, Israel Version: 0.5 (2026) License: MIT License Repo: Praat AudioTools

Contents:

What this does Quick start Image reduction RGB → harmonics Time interpolation Stereo mapping Controls Sampling & safety Output & level Visualization Historical context Further reading

What this does

The script converts one selected Praat Photo into a continuous stereo additive-synthesis texture.

Process:
Photo → sparse RGB sampling → per-column RGB means → three harmonic families with 1/h roll-off → linear control interpolation → constant-power stereo mapping → short global edge fade → target peak normalization

The mapping is intentionally one-dimensional in image space: horizontal position maps to time. The vertical dimension is not mapped to pitch or harmonic number. Instead, selected rows are averaged within each sampled image column.

The output therefore represents the horizontal evolution of the image's sampled RGB balance, not a full two-dimensional raster scan.

Quick start

Select exactly one Photo object in Praat.
Run Spectral_Image_Sonification.praat.
Set Duration, Fundamental frequency, and Max harmonics.
Enable Edit details if you need to change Sample rate, analysis resolution, Stereo width, or Output peak.
Run the script. A stereo Sound named spectral_<PhotoName> is created.

Image reduction

Praat's red, green, and blue Photo channels are extracted as Matrix objects. Their values are used directly as synthesis controls and are clamped only to the interval 0–1 for robustness.

There is no image-relative contrast stretching in v0.5. A flat grey image therefore remains grey in the control domain, and a white image remains high-valued instead of collapsing because its local min and max are equal.

Analysis columns

The image width is reduced to the requested number of analysis columns, capped at the actual image width. Each analysis position samples a column near the center of its horizontal bin.

Sampled rows

Within each analysis column, the script samples a requested number of approximately evenly distributed image rows, capped at the image height.

The sampled values are averaged independently:

R[c] = mean sampled red values in column c
G[c] = mean sampled green values in column c
B[c] = mean sampled blue values in column c

This is a sparse vertical estimate, not a full-pixel column average unless the number of sampled rows happens to equal the image height.

RGB → harmonic families

The synthesizer uses one fixed harmonic series above the selected fundamental frequency:

f[h] = h × F0

Harmonics are assigned cyclically to RGB:

Color control	Harmonics
Red	1, 4, 7, 10, 13, …
Green	2, 5, 8, 11, 14, …
Blue	3, 6, 9, 12, 15, …

For harmonic h, amplitude is:

A[h,t] = Cgroup[h,t] / h

where Cgroup is the interpolated red, green, or blue control assigned to that harmonic family.

The 1/h factor supplies a fixed spectral roll-off. Color therefore changes the relative strength of interleaved harmonic families while harmonic number still contributes a systematic decrease in amplitude.

No brightness gate

Version 0.5 does not skip low-brightness columns and does not multiply the synthesis by brightness a second time. A dark control region simply produces proportionally small harmonic amplitudes before final output normalization.

Continuous time interpolation

The RGB column means become equally spaced control points across the requested audio duration.

For more than one analysis column:

control interval = Duration / (AnalysisColumns - 1)

Within each interval, every RGB control is linearly interpolated between adjacent column means. Harmonic amplitudes therefore change continuously instead of being reset or windowed at analysis boundaries.

Constant images no longer pulse at the analysis-column rate. A constant RGB image produces constant harmonic controls across time, apart from the short global fade at the beginning and end.

The oscillators use absolute Sound time:

sin(2π f x)

so harmonic phase is continuous across control-column boundaries; there is no phase reset at each image column.

If Analysis columns = 1, one constant RGB spectrum is synthesized for the complete duration.

Stereo mapping

Stereo position is derived from the red–blue difference:

pLeft(t) = 0.5 + 0.5 × StereoWidth × [R(t) - B(t)]

pLeft is treated as a left-channel power fraction. Channel gains are:

gainL = sqrt(pLeft)
gainR = sqrt(1 - pLeft)

Therefore:

Stereo width = 0 keeps the complete spectrum centered.
Red dominance moves the complete spectrum toward the left.
Blue dominance moves the complete spectrum toward the right.
Green does not influence the pan coordinate.

Green harmonics are not independently fixed at the center. The whole additive spectrum receives the same instantaneous stereo gains; green is simply neutral in the calculation of the red–blue pan coordinate.

Controls

Main page

Control	Default	Behavior
Duration	5.0 s	Final stereo Sound duration.
Fundamental frequency	110 Hz	F0 of the harmonic series.
Max harmonics	12	Highest integer harmonic included; accepted range 1–64 subject to Nyquist safety.
Edit details	off	Opens technical analysis, stereo, and output controls.
Draw visualization	on	Draws the source→mapping→synthesis→output process display.
Play result	on	Plays the generated stereo Sound.

Details page

Control	Default	Behavior
Sample rate	44100 Hz	Direct synthesis and final output rate; minimum 8000 Hz.
Analysis columns	80	Horizontal control resolution; valid 1–512 and capped at image width.
Sample rows per column	8	Number of approximately evenly distributed vertical samples; valid 1–512 and capped at image height.
Stereo width	.60	Scales the red–blue power-pan displacement; valid 0–1.
Output peak	.90	Final target peak normalization; valid >0…1.

Sampling and safety

Nyquist guard

The highest requested harmonic is:

highest frequency = FundamentalFrequency × MaxHarmonics

It must remain at or below 95% of Nyquist. Unsafe settings stop rather than rescaling the harmonic series.

Control-rate resolution

When more than one analysis column is used, adjacent control points must be separated by at least two audio samples. If the requested Duration, sample rate, and analysis-column count would violate that requirement, the script stops.

Image-size clamping

If the requested number of columns or sampled rows is greater than the actual Photo dimensions, the analysis resolution is automatically reduced to the available dimensions and this is reported in the Info window.

Output and level

Property	Behavior
Input	Exactly one selected Photo.
Channels	Stereo.
Duration	Exactly the requested Duration.
Sample rate	Exactly the Details-page sample rate.
Global fade	Cosine fade-in and fade-out of up to 20 ms each, capped at one quarter of the total duration.
Normalization	Every non-silent result is peak-scaled once to Output peak. This is target normalization, not a down-only ceiling.
Object name	`spectral_<PhotoName>`.

Because final target normalization is global, absolute overall image brightness is not preserved as final peak level across separate renders. RGB brightness still controls the relative spectral/time structure within a render, but a uniformly dim and a proportionally brighter version of the same image can be brought to the same final peak.

Visualization and QC

Panel	What it shows
A — Image reduction	Left: the actual sparse RGB sample grid. Right: the resulting red, green, and blue column-mean control curves over sonification time.
B — Spectral control score	Every harmonic is shown on its own row. Color identifies the RGB family; line weight follows the actual `C(t)/h` coefficient.
C — Stereo mapping	Left/right constant-power gain curves derived from interpolated red–blue balance. The centered reference is `sqrt(.5)`.
D — Measured output	Final measured left and right waveforms after additive synthesis, interpolation, stereo gain, edge fades, and peak normalization.

The QC strip reports image dimensions, actual sampled resolution, mean RGB values, control interval, F0 and highest harmonic, Stereo width, mean left-power fraction, final peak/RMS, and Nyquist frequency.

Historical context

Sonification developed as a field concerned with systematic mappings from data or other non-auditory information into sound. By the 1990s the International Community for Auditory Display (ICAD) had established a research community around auditory display, and the 1999 Sonification Report described the state of the field and its research agenda.

This script belongs to the family often called parameter-mapping sonification: measured properties of a source are converted into continuously varying synthesis parameters. Here, image position controls time, RGB values control harmonic-family amplitudes, and red–blue balance controls stereo position.

The synthesis side also draws on the much older additive-synthesis tradition of computer music. Early digital music systems associated with Max Mathews and Bell Laboratories made explicit control of sinusoidal components a central technique; Mathews' 1969 The Technology of Computer Music, written with contributions from colleagues including J. C. Risset, documents that formative period.

Relation to this script: Spectral Image Sonification is not a historical reconstruction of a specific earlier sonification or Bell Labs instrument. Its RGB→interleaved-harmonic mapping, sparse vertical reduction, linear control interpolation, and color-derived stereo law are the specific design of this AudioTools implementation.