Perceptual Synchrony Pipeline — User Guide

Audio feature synchronization: detects similar acoustic gestures in two audio signals, clusters them by perceptual similarity, and applies binding effects to enhance perceived synchrony.

Author: Shai Cohen Affiliation: Department of Music, Bar-Ilan University, Israel Version: 2.2 (2025) License: MIT License Repo: https://github.com/ShaiCohen-ops/Praat-plugin_AudioTools

Contents:

What this does Quick start Synchrony Theory Gesture Detection Clustering Methods Binding Effects

What this does

This script implements Perceptual Synchrony Pipeline — a system for detecting and enhancing acoustic synchrony between two audio signals. Process: (1) Feature extraction: Intensity, spectral centroid, and spectral slope over time. (2) Gesture detection: Identify meaningful acoustic changes as "gestures" (80-2000ms). (3) Gesture tagging: Classify gestures into perceptual categories (brightness rise/fall, noise bloom, spectral drop, accent peak, intensity swell, smooth arc). (4) Clustering: Find matching gestures between signals using temporal proximity and structural similarity. (5) Binding effects: Apply audio processing to enhance perceived synchrony (amplification, timbral stamping, stereo width modulation). Result: Stereo mix where similar gestures are perceptually bound together.

Key Features:

Multi-feature analysis: Intensity + spectral centroid + spectral slope
7 Gesture types: Brightness rise/fall, noise bloom, spectral drop, accent peak, intensity swell, smooth arc
3 Clustering modes: Local window, structural role, hybrid
4 Effect presets: Subtle, Moderate, Aggressive, Extreme
Comprehensive visualization: 6-panel display of analysis and results
Stereo processing: Dynamic width modulation based on synchrony
Confidence scoring: Weighted matching with min_confidence threshold

What is perceptual synchrony? Traditional audio mixing: Manual alignment, equalization, compression. Perceptual binding: Cognitive grouping of simultaneous auditory events. Gesture-based synchronization: Matching similar acoustic contours between signals. Advantages: (1) Content-aware: Responds to acoustic content, not just timing. (2) Perceptual: Based on auditory grouping principles. (3) Automatic: No manual alignment needed. (4) Gradual: Confidence-based effect application. (5) Multi-dimensional: Uses intensity, timbre, spectral shape. Use cases: Audio production (enhancing instrument synchrony), sound design (creating perceptual groups), music analysis (studying ensemble coordination), film sound (binding effects to actions), experimental music (algorithmic synchronization).

Technical Implementation: (1) Frame analysis: 10ms steps, intensity via Praat Intensity object, spectrogram via Praat Spectrogram. (2) Normalization: 30-bin percentile normalization (95th percentile). (3) Gesture detection: Threshold-based on total derivative (intensity + centroid + slope). (4) Gesture characterization: Compute monotonicity, direction, covariation, peakness, salience. (5) Tagging: Apply thresholds to classify gestures. (6) Clustering: Local window (500ms) and/or structural position matching. (7) Confidence calculation: Weighted combination of tag overlap, shape similarity, salience match, duration match. (8) Resynthesis: Apply effect envelopes (attack/release), timbral stamp (2-4kHz boost), differential tilt, stereo width modulation. (9) Visualization: 6-panel display showing features, gestures, clusters, anchors, stereo width.

Quick start

In Praat, select exactly two Sound objects.
Run script… → Perceptual_Synchrony_Pipeline.praat.
Set Frame_step_ms (10ms typical) and Gesture thresholds.
Choose Clustering_mode: Local window, Structural role, or Both.
Set Perceptual_window_ms (500ms typical for local clustering).
Set Min_confidence (0.35 typical) for cluster acceptance.
Choose Effect_preset: Subtle, Moderate, Aggressive, or Extreme.
Enable Play_result to automatically play output.
Click OK — analysis runs, visualization appears, stereo mix created.

Quick tip: Start with Moderate preset and Both clustering mode. Use similar-length sounds for best results. For rhythmic music, use local window clustering (500ms). For structural similarities (e.g., verse-chorus patterns), use structural role clustering. Check min_confidence threshold — lower (0.2) catches more matches but may include false positives; higher (0.5) is stricter. Enable visualization to see detected gestures and clusters. The output stereo mix has enhanced synchrony: anchor regions (clusters) are boosted and share timbral stamp; non-anchor regions have differential processing. Play with headphones to hear stereo width modulation.

Important: TWO SOUNDS REQUIRED — script expects exactly two selected Sound objects. Feature extraction works best with clear acoustic events (not constant noise/tones). Gesture detection depends on derivative thresholds — adjust gesture_threshold for more/fewer gestures. Clustering requires some temporal overlap — if sounds are completely different timing, few clusters found. Effect application is destructive (amplifies/clips possible) — output normalized to 0.95 peak. Stereo width modulation can create extreme panning in extreme preset. Processing time scales with duration — long files (>5min) may be slow (consider downsampling outside script). Visualization creates complex plot — may be slow on older computers.

Synchrony Theory

Perceptual Binding Principles

🧠 Auditory Grouping Mechanisms

Gestalt principles applied to audio:

Common fate: Simultaneous similar changes group together
Similarity: Similar acoustic characteristics group together
Proximity: Temporally close events group together
Good continuation: Smooth, continuous changes group together

Script operationalizes these principles via feature analysis and clustering

Acoustic Feature Space

Three primary features tracked:

INTENSITY (loudness): Computed via Praat Intensity object (75 Hz cutoff) Represents: Energy, amplitude envelope Temporal derivative: dInt/dt (change in loudness) SPECTRAL CENTROID (brightness): Weighted average frequency: Σ(freq × power) / Σ(power) Represents: Perceptual brightness/"sharpness" Temporal derivative: dCentroid/dt (brightening/darkening) SPECTRAL SLOPE (noise-to-toneness): Ratio: highPower(2-5kHz) / lowPower(0-1kHz) Represents: Noise content, spectral tilt Temporal derivative: dSlope/dt (noise blooming/clearing)

Gesture Definition

🎵 What Constitutes an Acoustic Gesture?

Operational definition: A continuous segment where combined feature derivatives exceed threshold

Total change metric: totalChange[i] = |dCentroid[i]| + |dIntensity[i]| + |dSlope[i]| Gesture detection: IF totalChange > gesture_threshold (default: 0.12) THEN gesture starts WHILE totalChange > gesture_threshold × 0.5 AND duration < maxGestureDur (2s) gesture continues ELSE gesture ends Constraints: Minimum duration: 80ms (8 frames at 10ms step) Maximum duration: 2000ms Must contain meaningful change (not constant)

Why this definition? Captures perceptually salient changes while filtering noise and micro-variations

Normalization Strategy

30-bin percentile normalization:

FOR each feature (intensity, centroid, slope) in each sound: 1. Find min and max values 2. Create 30 equally-spaced bins across range 3. Compute histogram of feature values 4. Find 95th percentile bin (where cumulative sum ≥ 0.95×total_frames) 5. Use value at that bin as normalization reference Normalization formula: normalized_value = (raw_value - min_value) / (pct95_value - min_value) Benefits: - Robust to outliers (unlike min-max) - Consistent across different audio levels - Preserves relative dynamics - Works for both tonal and noisy sounds

Synchrony Confidence Model

📊 Multi-factor Confidence Calculation

Four components weighted:

Component	Weight	Calculation
Tag Overlap	30%	Weighted sum of matching gesture tags
Shape Similarity	30%	Direction matching + monotonicity similarity
Salience Match	20%	min(salienceA, salienceB) / max(salienceA, salienceB)
Duration Match	20%	min(durationA, durationB) / max(durationA, durationB)

Total confidence: conf = 0.3×tagScore + 0.3×shapeScore + 0.2×salienceScore + 0.2×durScore Threshold: IF conf ≥ min_confidence (default: 0.35) THEN accept as cluster

Gesture Detection & Tagging

Gesture Shape Analysis

📈 Characterizing Gesture Morphology

Computed for each detected gesture:

CUMULATIVE CHANGES: centChange = Σ dCentroid over gesture intChange = Σ dIntensity over gesture slopeChange = Σ dSlope over gesture DIRECTIONALITY: risingCent = frames where dCentroid > 0.01 fallingCent = frames where dCentroid < -0.01 centDirection = sign(centChange) [1 = rise, -1 = fall] MONOTONICITY: centMonotonic = max(risingCent, fallingCent) / totalFrames intMonotonic = max(risingInt, fallingInt) / totalFrames COVARIATION: covarFrames = frames where (dCentroid & dSlope same sign) covariation = covarFrames / totalFrames PEAKNESS: Find intensity peak within gesture peakContrast = maxIntensity - max(minBefore, minAfter) (Only if risePhase≥2 AND fallPhase≥2 AND contrast>0.15) SALIENCE: salience = √(centChange² + intChange² + slopeChange²)

The Seven Gesture Tags

🏷️ Perceptual Categories

Tag	Symbol	Detection Criteria	Weight	Color
Brightness Rise	BR	centChange>0.2 AND centMonotonic>0.65	1.0	Green
Brightness Fall	BF	centChange<-0.2 AND centMonotonic>0.65	1.0	Blue
Noise Bloom	NB	centChange>0.12 AND slopeChange>0.12 AND covariation>0.4	1.2	Orange
Spectral Drop	SD	centChange<-0.2 AND slopeChange<-0.2 AND covariation>0.4	1.2	Purple
Accent Peak	AP	peakness > 0.15	1.5	Red
Intensity Swell	IS	intChange>0.15 AND intMonotonic>0.65	0.8	Light Green
Smooth Arc	SA	centMonotonic > 0.8	0.5	Gray

Weighted tag score: sum of matching tag weights between gestures

Tag Overlap Calculation

EXAMPLE: Gesture A has tags [BR, AP] (weights: 1.0 + 1.5 = 2.5) Gesture B has tags [BR, NB] (weights: 1.0 + 1.2 = 2.2) Matching tags: BR (both have it) Weighted overlap: wOverlap = 1.0 (weight of BR) Total weights: totalW = 2.5 + 2.2 = 4.7 Tag score: tagScore = wOverlap / (totalW - wOverlap + ε) = 1.0 / (4.7 - 1.0 + 0.001) = 1.0 / 3.701 ≈ 0.27 Interpretation: Higher when gestures share many distinctive tags

Shape Similarity Metrics

DIRECTION MATCHING: IF centDirectionA = centDirectionB: +0.3 IF intDirectionA = intDirectionB: +0.3 MONOTONICITY SIMILARITY: 0.2 × (1 - |centMonotonicA - centMonotonicB|) COVARIATION SIMILARITY: 0.2 × (1 - |covariationA - covariationB|) TOTAL SHAPE SCORE: shapeScore = directionMatch + monotonicitySim + covariationSim Range: [0, 1.0]

Clustering Methods

Mode 1: Local Window Clustering

⏱️ Temporal Proximity

Principle: Gestures occurring in same time window likely related

Algorithm:

Divide timeline into overlapping windows (500ms default)
Window step = 250ms (50% overlap)
Compare all gestures from Sound A and Sound B within same window
Compute confidence score for each pair
Accept if confidence ≥ min_confidence

Best for: Rhythmic synchronization, real-time coordination, phrase alignment

Mode 2: Structural Role Clustering

🏛️ Normalized Position

Principle: Gestures at similar relative positions serve similar structural roles

Algorithm:

Compute normalized position: normPos = midTime / totalDuration
Compare gestures with |normPosA - normPosB| < tolerance (0.15)
Require stronger tag overlap (wOverlap > 0.5)
Weight shape similarity more heavily (0.35 vs 0.30)
Include position similarity in confidence calculation

Best for: Formal structures, verse-chorus patterns, large-scale organization

Mode 3: Hybrid (Both Methods)

🔀 Combined Approach

Principle: Use both temporal proximity and structural similarity

Algorithm:

Run local window clustering (Mode 1)
Run structural role clustering (Mode 2)
Combine candidate lists (union, not intersection)
Apply same confidence threshold to all
Label each cluster with mode ("LOCAL" or "STRUCT")

Best for: General purpose, mixed rhythmic/structural synchronization

Default recommendation: Most versatile, catches most meaningful matches

Candidate Selection & Greedy Matching

INPUT: List of candidate pairs (gA, gB, confidence) CONSTRAINTS: Max clusters per gesture: max_clusters_per_gesture (default: 2) Confidence cutoff: min_confidence + 0.05 (e.g., 0.40 if min=0.35) GREEDY ALGORITHM: 1. Sort candidates by confidence (descending) 2. FOR each candidate in sorted order: IF gestureA not yet used ≥ max_clusters AND gestureB not yet used ≥ max_clusters AND confidence ≥ cutoff THEN: - Accept as cluster - Mark gestures as used - Continue OUTPUT: List of accepted clusters

Cluster Properties Storage

FOR each accepted cluster c: cluster_gA[c] = gesture A index cluster_gB[c] = gesture B index cluster_conf[c] = confidence score (0-1) cluster_mode$[c] = "LOCAL" or "STRUCT" cluster_wOverlap[c] = weighted tag overlap cluster_shapeScore[c] = shape similarity score

Binding Effects & Resynthesis

Effect Presets

🎚️ Four Intensity Levels

Preset	Anchor Boost	Timbral Stamp	Outside Tilt	Anchor Width	Outside Width	Attack	Release
Subtle	+4 dB	+1.5 dB	±1.0 dB	0.25	0.40	20 ms	80 ms
Moderate	+8 dB	+3.0 dB	±2.0 dB	0.15	0.45	15 ms	100 ms
Aggressive	+12 dB	+5.0 dB	±3.0 dB	0.05	0.50	10 ms	120 ms
Extreme	+15 dB	+6.0 dB	±4.0 dB	0.02	0.55	8 ms	150 ms

All effects scaled by confidence² (non-linear emphasis on high-confidence clusters)

Anchor Region Processing

🔊 Amplitude Envelope Shaping

For each anchor region (cluster):

effectBoost = 1 + (boostLin - 1) × conf² WHERE boostLin = 10^(anchorBoostDB/20) ENVELOPE SHAPE: attack phase: linear ramp from 1 to effectBoost over attackMs sustain phase: constant at effectBoost release phase: linear ramp from effectBoost to 1 over releaseMs APPLICATION: partA[startA:endA] *= envelope(t) partB[startB:endB] *= envelope(t)

Also create mask: maskA[startA:endA] = 1 (marks anchor regions)

Timbral Stamping (2-4kHz Emphasis)

🎛️ Shared Spectral Signature

Creates common timbral identity for synchronized gestures:

stampGain = 10^(anchorStampDB/20) - 1 PROCESS: 1. Create band-pass filtered versions (2000-4500 Hz) 2. Scale filtered signals to 0.5 peak (prevent over-emphasis) 3. Mix into anchors only (using mask): partA += filteredA × maskA × stampGain partB += filteredB × maskB × stampGain RATIONALE: 2-4kHz region important for perceptual "presence" and fusion

Differential Tilt Outside Anchors

🎚️ Contrast Enhancement

Creates perceptual contrast between synchronized and unsynchronized regions:

tiltGain = 10^(outsideTiltDB/20) PROCESS: OUTSIDE anchors (mask < 0.5): Sound A: × (1 + (tiltGain - 1) × 0.3) [brighter] Sound B: ÷ (1 + (tiltGain - 1) × 0.3) [darker] INSIDE anchors (mask ≥ 0.5): No change (already processed with anchor boost/stamp) EFFECT: Enhances distinction, makes anchors "pop" more

Stereo Width Modulation

🎧 Dynamic Panning Envelopes

Creates spatial binding for synchronized gestures:

width parameters: anchorWidth = narrow (0.02-0.25) outsideWidth = wide (0.40-0.55) panning envelopes: panEnvA = outsideWidth - (outsideWidth - anchorWidth) × maskA panEnvB = outsideWidth - (outsideWidth - anchorWidth) × maskB STEREO MIX: LEFT = partA × (0.5 + panEnvA) + partB × (0.5 - panEnvB) RIGHT = partA × (0.5 - panEnvA) + partB × (0.5 + panEnvB) RESULT: - Outside anchors: wider stereo image (sounds separated) - Inside anchors: narrower image (sounds fused)

Visualization Output

📊 Six-Panel Display

Panel	Content	Purpose
1. Sound A Features	Centroid (blue), intensity (orange), gestures (colored bars)	Show detected gestures in Sound A
2. Sound B Features	Same as panel 1 but for Sound B	Show detected gestures in Sound B
3. Cluster Connections	Timelines with connecting lines (width = confidence)	Visualize which gestures are linked
4. Anchor Regions	Top: Sound A anchors (blue), Bottom: Sound B anchors (purple)	Show temporal location of synchronized regions
5. Stereo Width	Width envelope over time (narrow in anchors, wide outside)	Show spatial binding modulation
6. Legend & Parameters	Tag colors, cluster types, effect parameters	Reference for interpretation

Color coding: Red=Peak, Orange=Bloom, Green=Rise, Blue=Fall, Purple=Drop, Light Green=Swell, Gray=Smooth Arc

Output Characteristics

Final stereo mix properties:

Format: Stereo, same sampling rate as input A
Duration: Minimum of two input durations (truncated to match)
Peak level: Normalized to 0.95 (-0.4 dBFS)
Naming: PerceptualMix_[SoundA]_[SoundB]
Processing chain: Anchor boost → Timbral stamp → Differential tilt → Width modulation → Stereo mix
Audible effects: Synchronized gestures louder, brighter, more centered; unsynchronized regions contrasting

Parameter Guidelines

For different applications:

Speech synchronization: Frame step 20ms, min gesture 120ms, Moderate preset
Music ensemble: Frame step 10ms, min gesture 80ms, Local window 300ms
Sound design/film: Frame step 5ms (detailed), Structural clustering, Subtle preset
Rhythmic analysis: Local window 200ms, min confidence 0.4
Large-scale form: Structural clustering, posTol 0.2, min confidence 0.3
Experimental: Extreme preset, Both clustering, min confidence 0.25

Advanced Applications

Music Production

Use case: Enhancing drum-vocal synchronization in pop mixes

Settings: Local window 250ms, Moderate preset, min confidence 0.4

Result: Kick/vocal accents bound together, creating tighter rhythm section

Sound Design

Use case: Binding Foley effects to visual actions

Workflow: Sound A = Foley, Sound B = visual impact sound

Settings: Both clustering, Aggressive preset for strong binding

Music Analysis

Use case: Studying ensemble coordination in classical recordings

Method: Analyze violin vs cello parts, examine cluster patterns

Research questions: How much synchronization? Where does it occur?

Algorithmic Composition

Use case: Creating synchronized textures from unrelated sounds

Technique: Process multiple pairs, layer results

Creative potential: Emergent synchrony from disparate materials

Psychoacoustic Research

Use case: Studying perceptual binding thresholds

Experimental design: Vary parameters, measure listener judgments

Research value: Quantitative model of auditory grouping

Troubleshooting

Problem: No clusters found
Causes: Sounds too different, min_confidence too high, no temporal overlap
Solutions: Lower min_confidence to 0.2, try Both clustering, check if sounds have similar events

Problem: Output clipping/distorted
Causes: Effect preset too aggressive, anchor boost too high
Solutions: Use Subtle/Moderate preset, script normalizes to 0.95 peak

Problem: Too many false clusters
Causes: min_confidence too low, gesture_threshold too low
Solutions: Increase min_confidence to 0.4-0.5, increase gesture_threshold to 0.15-0.18

Problem: Processing very slow
Causes: Long files, small frame step, complex visualization
Solutions: Use frame step 20ms, extract shorter sections, disable visualization in script

Future Development

Potential extensions:

Multi-signal processing: Extend to 3+ sounds simultaneously
Temporal warping: Automatically align sounds before processing
Machine learning: Train gesture classifiers on labeled data
Real-time version: Implement as VST/AU plugin
More features: Add pitch, roughness, modulation features
Cross-modal: Synchronize audio with visual/motion data
Interactive control: GUI for adjusting clusters manually