Photo Brightness-Controlled Pitch Sonification — User Guide
A deterministic monophonic column-projection sonification. Image X position is preserved as time; the complete image height is intentionally averaged inside each horizontal analysis bin. Weighted brightness controls instantaneous pitch, while red/blue balance controls equal-power stereo position.
Scope
This engine intentionally performs a monophonic column projection. It preserves the image's horizontal axis as musical time, but collapses the vertical axis by averaging the full image height within every horizontal analysis bin.
image X position → time
full-height RGB average → weighted brightness + red/blue balance
weighted brightness → instantaneous pitch
red/blue balance → equal-power stereo pan
It therefore does not claim to preserve full 2-D image geometry, identify visual objects, or infer semantic image content.
Brightness controls pitch only. It does not also control amplitude. The output level is controlled independently by Master amplitude.
Quick start
- Select one Photo object in Praat.
- Run
Photo_Brightness-Controlled_Pitch_Sonification.praat. - Set Duration, Sample rate, Minimum and Maximum pitch, mapping type, Analysis columns, Master amplitude, and Colour pan strength.
- Run the script. The image is reduced to a full-height brightness/pan projection, interpolated across time, and rendered as one phase-continuous oscillator.
- The final result remains selected as
Brightness_Pitch_<Photo name>.
Full-height horizontal-bin analysis
The script extracts Praat's red, green, and blue matrices from the selected Photo and converts each channel Matrix into a two-dimensional integral image.
For every horizontal analysis bin, the complete image height and the complete X extent of that bin are averaged exactly. This is not a center-column sample and not a sparse-row approximation.
Input channel scale
The engine first checks the raw range of the extracted RGB matrices:
| Observed RGB range | Scaling used |
|---|---|
| Approximately 0…1 | Fixed black=0, white=1. |
| Approximately 0…255 | Fixed byte-like fallback: black=0, white=255. |
| Other values | Observed minimum/maximum fallback, explicitly reported in Info. |
This is important for uniform images. With standard normalized Photo data, a uniform white image remains brightness 1 rather than being contrast-normalized down to the minimum pitch.
Weighted brightness
After conversion to the 0–1 working range, each horizontal bin receives one weighted brightness value:
The coefficients provide a Rec.709-like perceptual weighting. In this script they are used as a transparent RGB brightness proxy; the engine does not claim a color-managed photometric luminance conversion.
Because the entire image height is averaged, vertical features contribute to the mean of their X bin but do not retain a separate vertical pitch location.
Brightness → pitch
Logarithmic mapping — default
The default mapping is logarithmic in frequency:
Equal increments of brightness therefore correspond to equal musical intervals. Brightness 0 maps to the effective minimum pitch and brightness 1 maps to the effective maximum pitch.
Linear-Hz mapping
If Logarithmic pitch mapping is disabled:
This makes frequency in hertz proportional to brightness rather than musical interval.
Image control interpolation and phase continuity
Version 0.4 no longer renders one independent sine segment per image column.
Each analysis-bin brightness value is stored as one sample in a low-rate control Sound. Output time is mapped exactly from the first to the last control-sample center:
Praat's time-based Sound access interpolates between adjacent control samples. The interpolated brightness is then converted into instantaneous frequency using the selected pitch mapping.
Audio-rate phase integration
The oscillator phase is accumulated sample by sample:
The mono source is:
This makes the oscillator phase-continuous while the target frequency follows the interpolated image projection. There are no phase resets at analysis-bin boundaries.
Red / blue balance → stereo position
Panning is calculated separately from brightness. The relative red/blue balance is:
balance = (B - R) / denominator
The stereo coordinate is:
and is limited to .02–.98. With the conventional coordinate used here:
- 0 = left
- .5 = center
- 1 = right
Red-dominant bins therefore move left, blue-dominant bins move right, and equal red/blue balance remains centered.
The pan control is interpolated through time using the same horizontal-bin trajectory as brightness. Stereo gains are equal-power:
rightGain = sqrt(pan)
Controls
| Control | Default | Range | Behavior |
|---|---|---|---|
| Duration | 3.0 s | >0–180 s | Exact final Sound duration and complete left-to-right scan time. |
| Sample rate | 44100 Hz | 8000–192000 Hz | Direct audio synthesis and final output rate. |
| Minimum pitch | 100 Hz | ≥20 Hz | Requested lower brightness endpoint before common Nyquist scaling. |
| Maximum pitch | 1000 Hz | > minimum | Requested upper brightness endpoint before common Nyquist scaling. |
| Logarithmic pitch mapping | on | boolean | On = equal brightness increments map to equal pitch intervals; off = linear Hz. |
| Analysis columns | 200 | 2–2000 | Requested number of horizontal bins, capped by image width. |
| Master amplitude | .55 | >0–2 | Mono oscillator amplitude before equal-power stereo panning and final protection. |
| Colour pan strength | .90 | 0–1 | Depth of red-left / blue-right displacement. |
| Peak protection | on | boolean | Enables final down-only ceiling at .92. |
Frequency and sampling safety
The source is a pure sinusoidal oscillator, so the engine reserves:
If Maximum pitch exceeds that boundary, one common scale is applied to both pitch endpoints:
This preserves the ratio between the requested minimum and maximum. If common scaling would push the effective minimum below 20 Hz, the script stops instead.
Output and level
| Property | Behavior |
|---|---|
| Input | One selected Praat Photo. |
| Duration | Exactly the requested Duration. |
| Sample rate | Exactly the selected 8–192 kHz rate. |
| Channels | Stereo. One common mono oscillator is distributed with time-varying equal-power pan. |
| Amplitude mapping | Brightness does not control amplitude; Master amplitude does. |
| Randomness | None in image analysis or audio synthesis. The random temporary ID used for object naming does not affect the Sound. |
| Edge fade | Common 15 ms linear fade-in/out, capped at 10% of Duration. |
| Peak protection | If enabled and the stereo peak exceeds .92, the complete Sound is scaled down once to .92. |
| Normalization | No upward normalization. If peak ≤ .92, the level set by Master amplitude is preserved. |
| Object name | Brightness_Pitch_<Photo name>. |
Visualization and QC
The current visualization documents the actual mapping and measured audio rather than presenting an abstract image-sonification diagram.
| Panel | What it shows |
|---|---|
| A — Actual Image Projection | Weighted full-height brightness and the stereo pan trajectory across time. |
| B — Brightness → Pitch Transfer | The selected logarithmic or linear-Hz mapping shown as musical interval above the effective minimum. Sampled actual image-bin values are overlaid. |
| C — Actual Target Pitch | The interpolated image-control trajectory converted to target pitch before audio-rate phase integration. |
| D — Model → Measurement | Measured spectrogram of the left output channel with the target instantaneous-frequency trajectory overlaid. |
The summary block reports the intentional column-projection scope, realized brightness and pitch ranges, sample rate, safe frequency boundary, common frequency scale, Master amplitude, pre-protection peak/RMS, and whether down-only peak protection was applied.
Further reading
These references are directly relevant to the general sonification framework used here:
- Hermann, T., Hunt, A., & Neuhoff, J. G. (Eds.). (2011). The Sonification Handbook. Logos Verlag Berlin. ISBN 978-3-8325-2819-5.
- Kramer, G., Walker, B. N., Bonebright, T., Cook, P., Flowers, J. H., Miner, N., Neuhoff, J. G., et al. (1999). Sonification Report: Status of the Field and Research Agenda. International Community for Auditory Display. ISBN 0-9670904-0-7.