Photo Brightness-Controlled Pitch Sonification — User Guide

A deterministic monophonic column-projection sonification. Image X position is preserved as time; the complete image height is intentionally averaged inside each horizontal analysis bin. Weighted brightness controls instantaneous pitch, while red/blue balance controls equal-power stereo position.

Author: Shai Cohen Affiliation: Department of Music, Bar-Ilan University, Israel Version: 0.4.1 (2026) License: MIT License Repo: Praat AudioTools
Contents:

Scope

This engine intentionally performs a monophonic column projection. It preserves the image's horizontal axis as musical time, but collapses the vertical axis by averaging the full image height within every horizontal analysis bin.

Mapping:
image X position → time
full-height RGB average → weighted brightness + red/blue balance
weighted brightness → instantaneous pitch
red/blue balance → equal-power stereo pan

It therefore does not claim to preserve full 2-D image geometry, identify visual objects, or infer semantic image content.

Brightness controls pitch only. It does not also control amplitude. The output level is controlled independently by Master amplitude.

Quick start

  1. Select one Photo object in Praat.
  2. Run Photo_Brightness-Controlled_Pitch_Sonification.praat.
  3. Set Duration, Sample rate, Minimum and Maximum pitch, mapping type, Analysis columns, Master amplitude, and Colour pan strength.
  4. Run the script. The image is reduced to a full-height brightness/pan projection, interpolated across time, and rendered as one phase-continuous oscillator.
  5. The final result remains selected as Brightness_Pitch_<Photo name>.
Analysis columns is automatically limited to the Photo's actual pixel width. The script requires at least two requested analysis columns.

Full-height horizontal-bin analysis

The script extracts Praat's red, green, and blue matrices from the selected Photo and converts each channel Matrix into a two-dimensional integral image.

For every horizontal analysis bin, the complete image height and the complete X extent of that bin are averaged exactly. This is not a center-column sample and not a sparse-row approximation.

analysisColumns = min(requestedAnalysisColumns, imageWidth)

Input channel scale

The engine first checks the raw range of the extracted RGB matrices:

Observed RGB rangeScaling used
Approximately 0…1Fixed black=0, white=1.
Approximately 0…255Fixed byte-like fallback: black=0, white=255.
Other valuesObserved minimum/maximum fallback, explicitly reported in Info.

This is important for uniform images. With standard normalized Photo data, a uniform white image remains brightness 1 rather than being contrast-normalized down to the minimum pitch.

Weighted brightness

After conversion to the 0–1 working range, each horizontal bin receives one weighted brightness value:

brightness = 0.2126 × R + 0.7152 × G + 0.0722 × B

The coefficients provide a Rec.709-like perceptual weighting. In this script they are used as a transparent RGB brightness proxy; the engine does not claim a color-managed photometric luminance conversion.

Because the entire image height is averaged, vertical features contribute to the mean of their X bin but do not retain a separate vertical pitch location.

Brightness → pitch

Logarithmic mapping — default

The default mapping is logarithmic in frequency:

f = effectiveMinPitch × (effectiveMaxPitch / effectiveMinPitch)^brightness

Equal increments of brightness therefore correspond to equal musical intervals. Brightness 0 maps to the effective minimum pitch and brightness 1 maps to the effective maximum pitch.

Linear-Hz mapping

If Logarithmic pitch mapping is disabled:

f = effectiveMinPitch + brightness × (effectiveMaxPitch - effectiveMinPitch)

This makes frequency in hertz proportional to brightness rather than musical interval.

The visualization expresses both mappings in semitones above the effective minimum so their perceptual/musical curvature can be compared directly.

Image control interpolation and phase continuity

Version 0.4 no longer renders one independent sine segment per image column.

Each analysis-bin brightness value is stored as one sample in a low-rate control Sound. Output time is mapped exactly from the first to the last control-sample center:

controlPosition(t) = 0.5 + t × (analysisColumns - 1) / Duration

Praat's time-based Sound access interpolates between adjacent control samples. The interpolated brightness is then converted into instantaneous frequency using the selected pitch mapping.

Audio-rate phase integration

The oscillator phase is accumulated sample by sample:

phase[n] = phase[n-1] + 2π × frequency[n] / Fs

The mono source is:

mono[n] = MasterAmplitude × sin(phase[n])

This makes the oscillator phase-continuous while the target frequency follows the interpolated image projection. There are no phase resets at analysis-bin boundaries.

Red / blue balance → stereo position

Panning is calculated separately from brightness. The relative red/blue balance is:

denominator = max(0.08, R + B)
balance = (B - R) / denominator

The stereo coordinate is:

pan = 0.5 + 0.48 × ColourPanStrength × balance

and is limited to .02–.98. With the conventional coordinate used here:

Red-dominant bins therefore move left, blue-dominant bins move right, and equal red/blue balance remains centered.

The pan control is interpolated through time using the same horizontal-bin trajectory as brightness. Stereo gains are equal-power:

leftGain = sqrt(1 - pan)
rightGain = sqrt(pan)

Controls

ControlDefaultRangeBehavior
Duration3.0 s>0–180 sExact final Sound duration and complete left-to-right scan time.
Sample rate44100 Hz8000–192000 HzDirect audio synthesis and final output rate.
Minimum pitch100 Hz≥20 HzRequested lower brightness endpoint before common Nyquist scaling.
Maximum pitch1000 Hz> minimumRequested upper brightness endpoint before common Nyquist scaling.
Logarithmic pitch mappingonbooleanOn = equal brightness increments map to equal pitch intervals; off = linear Hz.
Analysis columns2002–2000Requested number of horizontal bins, capped by image width.
Master amplitude.55>0–2Mono oscillator amplitude before equal-power stereo panning and final protection.
Colour pan strength.900–1Depth of red-left / blue-right displacement.
Peak protectiononbooleanEnables final down-only ceiling at .92.

Frequency and sampling safety

The source is a pure sinusoidal oscillator, so the engine reserves:

safeTop = 0.45 × Fs

If Maximum pitch exceeds that boundary, one common scale is applied to both pitch endpoints:

frequencyScale = min(1, safeTop / MaximumPitch) effectiveMinPitch = MinimumPitch × frequencyScale effectiveMaxPitch = MaximumPitch × frequencyScale

This preserves the ratio between the requested minimum and maximum. If common scaling would push the effective minimum below 20 Hz, the script stops instead.

Output and level

PropertyBehavior
InputOne selected Praat Photo.
DurationExactly the requested Duration.
Sample rateExactly the selected 8–192 kHz rate.
ChannelsStereo. One common mono oscillator is distributed with time-varying equal-power pan.
Amplitude mappingBrightness does not control amplitude; Master amplitude does.
RandomnessNone in image analysis or audio synthesis. The random temporary ID used for object naming does not affect the Sound.
Edge fadeCommon 15 ms linear fade-in/out, capped at 10% of Duration.
Peak protectionIf enabled and the stereo peak exceeds .92, the complete Sound is scaled down once to .92.
NormalizationNo upward normalization. If peak ≤ .92, the level set by Master amplitude is preserved.
Object nameBrightness_Pitch_<Photo name>.

Visualization and QC

The current visualization documents the actual mapping and measured audio rather than presenting an abstract image-sonification diagram.

PanelWhat it shows
A — Actual Image ProjectionWeighted full-height brightness and the stereo pan trajectory across time.
B — Brightness → Pitch TransferThe selected logarithmic or linear-Hz mapping shown as musical interval above the effective minimum. Sampled actual image-bin values are overlaid.
C — Actual Target PitchThe interpolated image-control trajectory converted to target pitch before audio-rate phase integration.
D — Model → MeasurementMeasured spectrogram of the left output channel with the target instantaneous-frequency trajectory overlaid.

The summary block reports the intentional column-projection scope, realized brightness and pitch ranges, sample rate, safe frequency boundary, common frequency scale, Master amplitude, pre-protection peak/RMS, and whether down-only peak protection was applied.

Further reading

These references are directly relevant to the general sonification framework used here: