Percussive Image Sonification — User Guide
Transforms image columns into rhythmic clicks with stereo panning, frequency content, and timing derived from pixel brightness and color.
What this does
This script performs percussive sonification — converting visual image data into audible rhythmic patterns. Each column of the image becomes a percussive click with timing, pitch, volume, and stereo position determined by the column's color characteristics. The result is a stereo soundscape where bright image columns produce louder, higher-pitched clicks with faster rhythms, while color differences create left/right panning.
Key Features:
- Column-wise analysis: Each image column analyzed independently
- Multi-parameter mapping: 5 audio parameters derived from image data
- Stereo panning: Red/blue balance controls left/right position
- Complex clicks: Each click contains 3 frequency components (harmonically related)
- Rhythmic patterns: Timing creates natural rhythmic variations
- Enveloped sounds: Raised-cosine amplitude envelope for smooth percussive attacks
What is sonification? Sonification is the use of non-speech audio to convey information or perceptualize data. It transforms data dimensions into sound parameters (pitch, volume, timing, timbre). In this script:
- Visual data → Audio representation
- Columns → Temporal sequence (left to right becomes time progression)
- Brightness → Volume & timing (brighter = louder, faster clicks)
- Color balance → Stereo panning (red = left, blue = right)
- Overall brightness → Pitch range
Technical Implementation: (1) Image decomposition: Extract RGB channels as separate matrices. (2) Column analysis: For each column, compute average RGB values. (3) Normalization: Scale values to 0-1 range using global min/max. (4) Parameter calculation: Brightness = (R+G+B)/3, Pan = 0.5 + 0.5×(R-B). (5) Sound generation: Create stereo sound with clicks at computed intervals. Each click = 3 sine waves (base, mid, high) with amplitude envelope. (6) Stereo placement: Left channel = (1-pan)×signal, Right channel = pan×signal. (7) Normalization: Scale peak amplitude to prevent clipping.
Quick start
- In Praat, select exactly one Photo object (image).
- Run script… →
percussive_image_sonification.praat. - Set duration(seconds): Total audio length (e.g., 4.0 seconds).
- Set fs(Hz): Sample rate (44100 = CD quality).
- Set minClickInterval: Minimum time between clicks (0.08 seconds).
- Set maxClickInterval: Maximum time between clicks (0.3 seconds).
- Set clickDuration: Length of each click sound (0.05 seconds).
- Click OK — script processes image, generates stereo audio.
Quick tip: Start with a medium-sized image (500-1500 pixels wide). Very wide images produce many clicks in short duration; very narrow images produce sparse rhythms. The duration parameter controls how long the sonification runs — image columns cycle if needed. Brightness range is automatically detected from the image — no manual adjustment needed. Listen on stereo speakers/headphones to hear the panning effect. Try different image types: photographs, artwork, charts, text, patterns. Each produces distinctive rhythmic character.
Important: Requires a Photo object in Praat (use "Read from file..." to import images). The script analyzes columns only (vertical strips) — horizontal image features are averaged vertically. Works with color images (RGB) — grayscale images will produce centered panning (equal red/blue). Very dark images may produce quiet/rare clicks. Very bright images may produce dense, loud clicks. The script does not resize images — large images may cause slower processing (but Praat handles up to ~4000 pixels). Output is stereo — ensure playback setup supports stereo.
Sonification Theory
Data-to-Sound Mapping Principles
🎨 Visual Features → 🎵 Audio Parameters
Core concept: Systematic transformation of visual properties into auditory dimensions:
| Visual Feature | Audio Parameter | Mapping Direction |
|---|---|---|
| Column brightness (R+G+B)/3 | Click volume & timing interval | Brighter → Louder, Faster |
| Red-blue difference (R-B) | Stereo panning | Redder → Left, Bluer → Right |
| Overall brightness | Pitch range | Brighter → Higher frequencies |
| Column position (x-coordinate) | Temporal position | Left→Early, Right→Late |
| Vertical average | Unified click (no vertical mapping) | Entire column → Single event |
Design rationale: Intuitive mappings that match perceptual expectations: brightness↔loudness (both intensity), color↔space (chromatic↔spatial), time↔horizontal position (reading direction).
Why Percussive Sonification?
Advantages of percussive approach:
- Rhythmic patterns: Creates musically interesting structures
- Temporal clarity: Discrete events easier to perceive than continuous tones
- Natural mapping: Brightness→loudness is perceptually direct
- Spatial awareness: Stereo panning creates auditory image width
- Multi-parametric: Each click encodes multiple image properties simultaneously
Compared to other sonification methods:
- vs. Continuous pitch mapping: More rhythmic, less monotonous
- vs. Granular synthesis: Clearer event boundaries, easier to follow
- vs. Parameter sweeps: Creates identifiable patterns, not just textures
- vs. Pure data streams: More musical, less abstract
Mathematical Foundations
Brightness Calculation
For each column c:
R_avg(c) = (1/nrows) × Σ_{r=1}^{nrows} R(r,c)
G_avg(c) = (1/nrows) × Σ_{r=1}^{nrows} G(r,c)
B_avg(c) = (1/nrows) × Σ_{r=1}^{nrows} B(r,c)
Normalize to 0-1 range:
R_norm(c) = (R_avg(c) - overallMin) / (overallMax - overallMin)
G_norm(c) = (G_avg(c) - overallMin) / (overallMax - overallMin)
B_norm(c) = (B_avg(c) - overallMin) / (overallMax - overallMin)
Brightness:
bright(c) = [R_norm(c) + G_norm(c) + B_norm(c)] / 3
Where:
overallMin = minimum(R,G,B) across entire image
overallMax = maximum(R,G,B) across entire image
Panning Calculation
Panning position (0=left, 1=right):
pan(c) = 0.5 + 0.5 × [R_norm(c) - B_norm(c)]
Properties:
- If R_norm = B_norm (grayscale): pan = 0.5 (center)
- If R_norm > B_norm (reddish): pan > 0.5 (rightward)
- If R_norm < B_norm (bluish): pan < 0.5 (leftward)
- Range: theoretically 0-1, practically clamped by normalization
Stereo distribution:
Left channel gain = 1 - pan(c)
Right channel gain = pan(c)
Parameter Mapping Details
Timing Mapping
⏱️ Brightness → Click Interval
Mapping: Brighter columns = shorter intervals between clicks
interval(c) = maxClickInterval - bright(c) × (maxClickInterval - minClickInterval)
Where:
bright(c) = normalized brightness (0-1)
minClickInterval = user parameter (e.g., 0.08s)
maxClickInterval = user parameter (e.g., 0.30s)
Examples (with min=0.08, max=0.30):
bright=0.0 → interval = 0.30s (slow, dark)
bright=0.5 → interval = 0.19s (medium)
bright=1.0 → interval = 0.08s (fast, bright)
Perceptual effect: Bright regions create rapid-fire clicks; dark regions create sparse, spaced clicks. Creates natural rhythmic variations.
Volume Mapping
clickVolume(c) = bright(c) × 1.2
Where:
bright(c) = normalized brightness (0-1)
1.2 = scaling factor (allows some overshoot, normalized later)
Rationale:
- Linear mapping: brightness ↔ loudness
- Factor 1.2: Allows brighter-than-average clicks to exceed unity
- Final normalization: Peak scaled to 0.9 prevents clipping
Pitch Mapping
🎵 Brightness → Frequency Spectrum
Three-component harmonic structure:
baseFreq(c) = 800 + bright(c) × 1200 Hz
midFreq(c) = baseFreq(c) × 2.5 Hz
highFreq(c) = baseFreq(c) × 4.0 Hz
Frequency ranges:
bright=0.0: 800Hz + 1600Hz + 3200Hz
bright=0.5: 1400Hz + 3500Hz + 5600Hz
bright=1.0: 2000Hz + 5000Hz + 8000Hz
Amplitude ratios:
Base: 60% (strong fundamental)
Mid: 30% (first harmonic)
High: 10% (second harmonic)
Sonic character: Bright clicks = higher, sharper, more piercing. Dark clicks = lower, mellower, more subdued. The harmonic relationship creates consonant, bell-like tones.
Stereo Panning Mapping
Panning examples:
Pure red (R=1, G=0, B=0):
R_norm=1.0, B_norm=0.0 → pan = 0.5 + 0.5×(1-0) = 1.0
Left gain=0.0, Right gain=1.0 → completely right channel
Pure blue (R=0, G=0, B=1):
R_norm=0.0, B_norm=1.0 → pan = 0.5 + 0.5×(0-1) = 0.0
Left gain=1.0, Right gain=0.0 → completely left channel
Gray (R=0.5, G=0.5, B=0.5):
R_norm=0.5, B_norm=0.5 → pan = 0.5 + 0.5×(0.5-0.5) = 0.5
Left gain=0.5, Right gain=0.5 → centered, both channels equal
Cyan (R=0, G=1, B=1):
R_norm=0.0, B_norm=1.0 → pan = 0.5 + 0.5×(0-1) = 0.0
Left gain=1.0, Right gain=0.0 → blue dominates, left channel
Pure red (R=1, G=0, B=0):
R_norm=1.0, B_norm=0.0 → pan = 0.5 + 0.5×(1-0) = 1.0
Left gain=0.0, Right gain=1.0 → completely right channel
Pure blue (R=0, G=0, B=1):
R_norm=0.0, B_norm=1.0 → pan = 0.5 + 0.5×(0-1) = 0.0
Left gain=1.0, Right gain=0.0 → completely left channel
Gray (R=0.5, G=0.5, B=0.5):
R_norm=0.5, B_norm=0.5 → pan = 0.5 + 0.5×(0.5-0.5) = 0.5
Left gain=0.5, Right gain=0.5 → centered, both channels equal
Cyan (R=0, G=1, B=1):
R_norm=0.0, B_norm=1.0 → pan = 0.5 + 0.5×(0-1) = 0.0
Left gain=1.0, Right gain=0.0 → blue dominates, left channel
Amplitude Envelope
Raised-cosine (Hanning-like) window:
env(t) = [1 - cos(2π × (t - t_start) / clickDuration)] × 0.5
Where:
t = current time (seconds)
t_start = click start time
clickDuration = user parameter (e.g., 0.05s)
Properties:
- Smooth attack: starts at 0, rises smoothly
- Smooth decay: returns to 0 at end
- No discontinuities: avoids clicks/pops
- Maximum at center: env(clickDuration/2) = 1.0
Complete Click Formula
For left channel at time t during click c:
left(t) = (1 - pan(c)) × env(t) × clickVolume(c) ×
[0.6×sin(2π×baseFreq(c)×(t-t_start)) +
0.3×sin(2π×midFreq(c)×(t-t_start)) +
0.1×sin(2π×highFreq(c)×(t-t_start))]
For right channel:
right(t) = pan(c) × env(t) × clickVolume(c) ×
[0.6×sin(2π×baseFreq(c)×(t-t_start)) +
0.3×sin(2π×midFreq(c)×(t-t_start)) +
0.1×sin(2π×highFreq(c)×(t-t_start))]
Where all parameters derived from column c brightness/color
Parameters
Sonification Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
| duration(seconds) | real | 4.0 | Total output audio length |
| fs(Hz) | integer | 44100 | Sample rate (samples/second) |
| minClickInterval | real | 0.08 | Minimum time between clicks (seconds) |
| maxClickInterval | real | 0.3 | Maximum time between clicks (seconds) |
| clickDuration | real | 0.05 | Length of each click sound (seconds) |
Derived Parameters (from image)
| Parameter | Source | Range | Description |
|---|---|---|---|
| brightness(c) | (R+G+B)/3 normalized | 0.0-1.0 | Column average brightness |
| pan(c) | 0.5 + 0.5×(R-B) | 0.0-1.0 | Stereo pan position |
| clickInterval(c) | max - bright×(max-min) | min-max | Time to next click |
| clickVolume(c) | bright × 1.2 | 0.0-1.2 | Click amplitude |
| baseFreq(c) | 800 + bright×1200 | 800-2000 Hz | Fundamental frequency |
Fixed Parameters (in code)
| Parameter | Value | Description |
|---|---|---|
| midFreq multiplier | 2.5 | Mid frequency = base × 2.5 |
| highFreq multiplier | 4.0 | High frequency = base × 4.0 |
| base amplitude | 0.6 | Base frequency component strength |
| mid amplitude | 0.3 | Mid frequency component strength |
| high amplitude | 0.1 | High frequency component strength |
| volume scale | 1.2 | Brightness to volume multiplier |
| final peak | 0.9 | Normalization target (prevents clipping) |
Processing Pipeline
Complete Algorithm
1. INPUT VALIDATION
Check: Photo object selected
Exit if no photo
2. IMAGE DECOMPOSITION
Extract red channel → Matrix redID
Extract green channel → Matrix greenID
Extract blue channel → Matrix blueID
3. GLOBAL STATISTICS
For each channel (R,G,B):
Find min and max across entire image
Compute overallMin = min(R_min, G_min, B_min)
Compute overallMax = max(R_max, G_max, B_max)
Compute range = overallMax - overallMin
If range=0: set range=1 (uniform image)
4. COLUMN ANALYSIS
For each column c (1 to ncols):
Compute R_avg = average of column c in red matrix
Compute G_avg = average of column c in green matrix
Compute B_avg = average of column c in blue matrix
Normalize:
R_norm = (R_avg - overallMin) / range
G_norm = (G_avg - overallMin) / range
B_norm = (B_avg - overallMin) / range
Compute parameters:
brightness##[c] = (R_norm + G_norm + B_norm) / 3
pan##[c] = 0.5 + 0.5 × (R_norm - B_norm)
5. SOUND GENERATION
Create stereo sound "percussiveSonification" (2 channels, duration)
Initialize: currentTime = 0, col = 1
While currentTime < duration:
Get brightness = brightness##[col]
Get pan = pan##[col]
Calculate click parameters:
interval = maxClickInterval - brightness×(maxClickInterval-minClickInterval)
volume = brightness × 1.2
baseFreq = 800 + brightness × 1200
midFreq = baseFreq × 2.5
highFreq = baseFreq × 4.0
t_start = currentTime
t_end = currentTime + clickDuration
Apply to sound buffers:
For left channel: (1-pan) × envelope × volume × [0.6sin+0.3sin+0.1sin]
For right channel: pan × envelope × volume × [0.6sin+0.3sin+0.1sin]
Advance:
currentTime = currentTime + interval
col = col + 1
If col > ncols: col = 1 (wrap around)
6. POST-PROCESSING
Scale peak amplitude to 0.9
Rename to "image_percussive_sonification"
Play sound
Clean up temporary matrices
Visualization of Process
🖼️ Image Processing Flow
Original Image: RGB photo with nrows × ncols pixels
Step 1: Channel Separation:
R Channel Matrix: nrows × ncols red values
G Channel Matrix: nrows × ncols green values
B Channel Matrix: nrows × ncols blue values
Step 2: Column Averaging:
For column 1: average all rows → R_avg(1), G_avg(1), B_avg(1)
For column 2: average all rows → R_avg(2), G_avg(2), B_avg(2)
...
For column n: average all rows → R_avg(n), G_avg(n), B_avg(n)
Step 3: Parameter Arrays:
brightness[ ] = [0.2, 0.8, 0.5, 0.9, 0.1, ...] (ncols values)
pan[ ] = [0.3, 0.7, 0.5, 0.9, 0.2, ...] (ncols values)
Step 4: Temporal Mapping:
Time: 0.0s → column 1 click
Time: 0.25s → column 2 click (interval based on brightness[1])
Time: 0.42s → column 3 click (interval based on brightness[2])
...
If run out of columns: wrap to column 1
Memory and Performance
Memory usage:
- Input: Original Photo object (nrows × ncols × 3 bytes)
- Intermediate: 3 Matrices (nrows × ncols each)
- Output: Stereo Sound (2 × duration × fs samples)
- Temporary: 2 arrays (brightness, pan) of length ncols
- Most expensive: Column averaging (O(nrows × ncols))
- Sound generation: Formula evaluation per sample during clicks only
- Scaling: Linear with image size and audio duration
- Typical times: 1-10 seconds for modest images (1000×1000)
Applications
Accessibility Tool
Use case: Making visual content accessible to visually impaired users
Technique: Sonify diagrams, charts, artwork
Benefits:
- Conveys spatial layout through stereo panning
- Communicates brightness variations through rhythm and volume
- Differentiates color regions through left/right positioning
- Creates memorable audio signatures for different image types
Data Exploration
Use case: Analyzing spatial data patterns through sound
Technique: Convert heatmaps, spectrograms, scientific images
Workflow:
- Generate sonification of data visualization
- Listen for patterns: regular rhythms, sudden changes, clusters
- Compare different datasets by their sonic signatures
- Use alongside visual analysis for multi-modal verification
Artistic Creation
Use case: Generating musical material from visual sources
Techniques:
- Use paintings/photographs as "scores"
- Create rhythmic patterns from architectural photos
- Generate ambient soundscapes from nature images
- Combine multiple sonifications into compositions
Scientific Visualization
Use case: Complementing visual data with auditory display
Applications:
- Astronomy: Sonify star fields, nebulae images
- Microscopy: Sonify cell structures, tissue samples
- Geophysics: Sonify terrain maps, seismic data
- Medical imaging: Sonify X-rays, MRI slices (non-diagnostic)
Practical Examples
🏙️ Urban Skyline Photo
Image: City skyline against sky
Sonification characteristics:
- Buildings (dark): Sparse, low-pitched clicks
- Windows (bright): Rapid, high-pitched clusters
- Sky (blue): Left-panned (blue→left) medium clicks
- Sunset (red/orange): Right-panned bright clicks
Result: Rhythmic pattern revealing building spacing and window distributions
📊 Bar Chart / Histogram
Image: Statistical chart with colored bars
Sonification characteristics:
- Tall bars (bright): Loud, frequent clicks
- Short bars (dark): Quiet, spaced clicks
- Color-coded data: Different pan positions for series
- Regular spacing: Even temporal rhythm (columns equally spaced)
Result: Auditory representation of distribution shape and outliers
🎨 Abstract Painting
Image: Color field painting (Rothko, etc.)
Sonification characteristics:
- Large color fields: Sustained similar clicks
- Color transitions: Gradual panning shifts
- Texture variations: Brightness changes → rhythmic variations
- Composition balance: Left/right distribution audible
Result: Sonic translation of color relationships and spatial composition
Parameter Tuning Strategies
For different image types:
- High-contrast images: Use wider interval range (0.05-0.4s) for dramatic rhythm changes
- Low-contrast images: Narrow interval range (0.1-0.2s) for more uniform rhythm
- Wide images: Longer duration to hear all columns without wrapping
- Colorful images: Default panning works well
- Grayscale images: Consider modifying pan formula to use luminance variations
- Detailed images: Shorter clickDuration (0.03s) for crisper sounds
- Simple images: Longer clickDuration (0.08s) for fuller tones
Advanced Modifications
Customizing the mapping:
- Change frequency range: Modify baseFreq formula (e.g., 200-1000 Hz for deeper sounds)
- Add more harmonics: Include additional sine components with different ratios
- Different envelope: Replace raised-cosine with exponential, ADSR, etc.
- Vertical analysis: Modify to analyze rows instead of or in addition to columns
- 2D mapping: Use both row and column position for more complex parameter control
- Add effects: Incorporate reverb, delay, filtering in post-processing
- Multi-channel: Extend beyond stereo to surround sound
Troubleshooting Common Issues
Problem: Very quiet or no sound
Cause: Image very dark (low brightness values)
Solution: Increase image brightness in image editor, or modify volume scaling in script
Cause: Image very dark (low brightness values)
Solution: Increase image brightness in image editor, or modify volume scaling in script
Problem: Constant, rapid clicking
Cause: Image very bright or minClickInterval too small
Solution: Increase minClickInterval to 0.15-0.2s
Cause: Image very bright or minClickInterval too small
Solution: Increase minClickInterval to 0.15-0.2s
Problem: No stereo effect (centered sound)
Cause: Grayscale image or equal red/blue values
Solution: Use color image, or modify pan formula to use green channel or luminance
Cause: Grayscale image or equal red/blue values
Solution: Use color image, or modify pan formula to use green channel or luminance
Problem: Clipping/distortion
Cause: Very bright image, volume scaling too high
Solution: Reduce max amplitude in script (change 1.2 to 0.8)
Cause: Very bright image, volume scaling too high
Solution: Reduce max amplitude in script (change 1.2 to 0.8)
Problem: Very long processing time
Cause: Large image dimensions
Solution: Resize image before loading (1000px max dimension recommended)
Cause: Large image dimensions
Solution: Resize image before loading (1000px max dimension recommended)
Future Extensions
Potential enhancements:
- Real-time sonification: Process webcam feed or video
- Interactive exploration: Click on image regions to hear corresponding sonification
- Multi-scale analysis: Sonify different frequency components of image (wavelet-like)
- Gesture control: Use hand/eye tracking to "scan" image auditorily
- Machine learning integration: Train models to recognize image features from sonification
- Cross-modal training: Help visually impaired learn to interpret visual data through sound