Gesture-Based Hard Quantization — User Guide

Concatenative gesture quantization: segment a reference sound, match each segment to a canonical dictionary atom in a normalized pitch/voicing/intensity feature space, and reconstruct the timeline with a nearest-neighbour leader plus optional stochastic polyphonic layers.

Author: Shai Cohen Affiliation: Department of Music, Bar-Ilan University, Israel Version: 2.2 (2026) License: MIT License Repo: https://github.com/ShaiCohen-ops/Praat-plugin_AudioTools

Contents:

What this does Quick start Quantization Theory Preset Guide Parameters Guide Algorithm Details Applications Complete Workflow

What this does

This script implements gesture-based hard quantization as a concatenative audio-mosaicing process. A reference sound is divided into equal analysis segments. Each dictionary sound is converted into one canonical atom whose duration and samples are exactly those used during rendering. Reference segments and atoms are then compared in a normalized feature space containing log-frequency pitch, an explicit voiced/unvoiced mask, and intensity.

The first output voice is the true hard-quantized leader: for every segment it selects the nearest dictionary atom after the recency penalty is applied. Optional Voices 2–4 add progressively more stochastic alternatives and are mixed across the stereo field. The result therefore ranges from strict nearest-neighbour quantization to a multi-layer polyphonic mosaic.

Key Features:

4 curated presets — Maximum Variety, Balanced, Coherent, Minimal
Explicit reference selection — use Reference_filename, or fall back to the alphabetically first supported audio file
Canonical dictionary atoms — analysis and playback refer to the same audio samples
3-channel feature timeline — log₂(F0), voiced mask, and intensity
Additive 3-step recency penalty — discourages recent reuse without making zero-distance matches immune
1–4 output voices — Leader, Shadow, Wander, Scatter
Fixed-hop assembly — crossfades do not shorten the requested timeline
Reproducible stochastic layers — Random_seed > 0 reproduces selections; 0 is unpredictable
Stereo polyphonic rendering — equal-power panning for multi-voice output
Suite-standard visualization — segment-to-gesture map, per-voice distances, pan positions, and summary strip

What does “hard quantization” mean here? Voice 1 maps each reference segment to one discrete dictionary atom by nearest-neighbour selection. That is the strict hard-quantization layer. Voices 2–4 are optional stochastic extensions: they deliberately choose from larger candidate sets, so the final multi-voice texture should not be described as a single deterministic nearest-neighbour quantizer.

Technical implementation: (1) scan supported audio files and choose the reference; (2) convert all files to mono, resample to 44.1 kHz, and scale intensity to 70 dB; (3) derive segment duration and build one canonical atom per dictionary entry; (4) extract 3N features from atoms and actual reference segments; (5) build normalization ranges from those same vectors; (6) select one atom per segment for each voice using top-k search plus recency penalty; (7) assemble each voice with fixed-hop crossfades; (8) pan and sum the voices; (9) peak-scale final output to 0.95.

Quick start

Prepare a folder containing at least two supported audio files: WAV/WAV-uppercase, AIFF, AIF, or FLAC.
Run Gesture-Based_Hard_Quantization.praat.
Enter Folder_path, or leave it blank to choose the folder interactively.
Enter Reference_filename exactly if you want a specific file to be the reference. If left blank, the alphabetically first file is used.
Choose a preset. Balanced is the default and creates two voices.
For Custom, set Number_of_segments, K_best_matches, Repetition_penalty, Num_voices, Voice_levels_dB, Crossfade_ms, Pitch_floor, Pitch_ceiling, Output_duration, and Random_seed.
Click OK. The script builds canonical atoms, extracts features, selects each voice, assembles the timeline, and mixes the result.

Quick tip: Start with Balanced. If you want the most literal reference-to-dictionary mapping, use one voice or the Minimal/Coherent presets. If you want a denser stereo mosaic, use Balanced or Maximum Variety. Set Random_seed to a positive integer when you want stochastic voices to be reproducible.

Important: The reference is no longer defined by an undocumented “first file” rule. Use Reference_filename when file identity matters. If the field is blank, the fallback is explicitly the alphabetically first supported audio file. Every input is converted to mono before matching and rendering; stereo in the final output comes from the multi-voice pan stage, not from preservation of source-channel imaging.

Quantization Theory

1. Reference Segments and Canonical Atoms

🎯 Match what is actually rendered

The reference duration is divided by Number_of_segments:

segmentDur = referenceDuration / Number_of_segments crossfade = Crossfade_ms / 1000 crossfade is capped to 0.5 * segmentDur canonical atom duration: grainDur = segmentDur + crossfade

For every dictionary entry, the script builds a canonical atom of exactly grainDur. A long gesture is truncated. A short gesture is lengthened as far as practical (factor capped at 8×); if it is still too short, the stretched content is tiled and then cut to the required length. No artificial zero-padding tail is inserted.

This is crucial: the same canonical atom is both measured and played. The matcher does not evaluate a whole dictionary file and later render an unrelated opening fragment.

2. Feature Space: 3N Dimensions

📊 Pitch + voicing + intensity

N = n_time_samples = 50 by default For every sample position i: Channel 1: logPitch[i] = log2(F0[i]) when voiced logPitch[i] = 0 when unvoiced Channel 2: voicedMask[i] = 1 when voiced voicedMask[i] = 0 when unvoiced Channel 3: intensity[i] = dB level Feature vector: v = [logPitch(1..N), voicedMask(1..N), intensity(1..N)] Default dimensionality: 3 * 50 = 150

Sampling is performed at cell centres, not exact segment edges. Pitch and intensity query times are additionally clamped into the valid frame range of their analysis objects, avoiding artificial undefined values at boundaries.

Pitch is stored as log₂ frequency. Therefore equal frequency ratios have comparable pitch distances: for example, 100→200 Hz and 400→800 Hz represent the same one-octave difference.

3. Global Normalization

Normalization ranges are computed from: • all canonical dictionary atoms • all actual reference analysis segments Pitch: normalized only for voiced frames mapped to [0,1] in log2-frequency space Voicing mask: already 0/1 and left unchanged Intensity: mapped to [0,1] If no voiced frames exist anywhere: pitch contributes no information matching proceeds from voicing/intensity information

4. Euclidean Distance

distance(referenceSegment, dictionaryAtom) = sqrt( sum over all 3N normalized dimensions (reference[d] - atom[d])^2 )

Smaller values indicate a closer match in this explicitly defined feature space. The distance is not a direct perceptual metric and is not weighted by musical pitch labels, timbral embeddings, or semantic content.

5. Additive Recency Penalty

🔁 Three-step reuse memory

First compute all raw candidate distances for the current segment. meanCandidateDistance = mean(distance to every dictionary atom) Then add: most recent choice: + repetition_penalty * meanCandidateDistance 2nd-most recent: + repetition_penalty * 0.6 * meanCandidateDistance 3rd-most recent: + repetition_penalty * 0.3 * meanCandidateDistance

The penalty is additive, not multiplicative. This means even a perfect zero-distance match can be pushed away from immediate repetition.

6. Top-K Selection

1. Compute penalized distances. 2. Find the K smallest candidates. 3. If K = 1: choose the nearest candidate. 4. If K > 1: choose UNIFORMLY at random from those K candidates. There is no 1/distance probability weighting inside the top-K set.

Preset Guide

4 Curated Presets

1. Maximum Variety

Segments: 30
K-best: 10
Repetition penalty: 0.8
Voices: 4

Character: Leader + Shadow + Wander + full-dictionary Scatter layer.

2. Balanced

Segments: 20
K-best: 7
Repetition penalty: 0.7
Voices: 2

Character: Nearest-neighbour leader plus one stochastic top-k shadow.

3. Coherent

Segments: 12
K-best: 3
Repetition penalty: 0.3
Voices: 1

Character: Mono leader only. Because Voice 1 always uses k=1, the preset's K value matters only if Num_voices is changed in Custom; built-in Coherent itself renders the nearest-neighbour leader.

4. Minimal

Segments: 8
K-best: 1
Repetition penalty: 0.0
Voices: 1

Character: Coarse, deterministic nearest-neighbour mapping with no recency penalty.

Preset scope: Built-in presets override Number_of_segments, K_best_matches, Repetition_penalty, and Num_voices. Voice_levels_dB, Crossfade_ms, Pitch_floor, Pitch_ceiling, Folder_path, Reference_filename, Output_duration, Random_seed, Draw_visualization, and Play_result remain user settings.

Voice Roles

Voice	Role	K	Penalty behaviour	Meaning
1	Leader	1	1.0×	Nearest-neighbour; the only strict hard-quantized layer
2	Shadow	K_best_matches	1.0×	Uniform random choice from the top-K candidates
3	Wander	2 × K_best_matches, clamped to dictionary size	1.3×	Broader stochastic candidate set with stronger recency pressure
4	Scatter	Entire dictionary	Effectively irrelevant	Uniform random choice over the whole dictionary; ranking and penalty cannot affect which entry is eligible

Parameters Guide

Main Form

Parameter	Default	Behaviour
Preset	Balanced	Built-ins override segment count, K, repetition penalty, and voice count
Number_of_segments	20	Natural number; defines analysis segment duration
K_best_matches	7	Base K used by stochastic voices; clamped to dictionary size per voice
Repetition_penalty	0.5 in form; 0.7 under default Balanced preset	Additive 3-step recency penalty
Num_voices	2	1 = mono leader; 2–4 = stereo polyphonic output
Voice_levels_dB	`0 -3 -5 -7`	Whitespace-separated gains for Voices 1–4; unreadable/missing values fall back to defaults
Crossfade_ms	10	Minimum 0.1 ms; capped to half a reference segment
Pitch_floor	75 Hz	Must remain below Pitch_ceiling; very short segments can force an effective per-segment floor upward
Pitch_ceiling	600 Hz	If floor ≥ ceiling, both reset to 75/600
Folder_path	blank	Blank opens a folder chooser
Reference_filename	blank	Exact file name; blank uses alphabetically first supported file
Output_duration	0	0 = match reference duration; positive value requests a different output duration
Random_seed	0	0 = unpredictable; positive integer = reproducible stochastic selections
Draw_visualization	Yes	Draws the v2.2 suite-standard diagnostic page
Play_result	Yes	Plays final output after completion

Script-Level Constants

These are intentionally no longer form controls in v2.1+, because the old 33-row form did not fit on smaller displays:

target_sample_rate = 44100
n_time_samples = 50
verbose_output = 1

Edit the script directly if these rarely changed values must be altered.

Output Duration

If Output_duration = 0: targetOutputDur = reference duration If Output_duration > 0: targetOutputDur = requested duration segmentsToGenerate = ceil(Output_duration / segmentDur) If more segments are required than exist in the reference analysis: reference segment features repeat cyclically: refSeg = ((outputSeg - 1) mod Number_of_segments) + 1

Algorithm Details

Phase 1: File Discovery and Preprocessing

Supported scan patterns: *.wav *.WAV *.aiff *.aif *.flac Reference: explicit Reference_filename when supplied otherwise alphabetically first file For every file: Read Convert to mono if needed Resample to 44100 Hz if needed Reject silent / near-silent input Scale intensity to 70 dB

Phase 2: Build Canonical Atoms

segmentDur = referenceDuration / Number_of_segments xfadeSec = min(Crossfade_ms / 1000, segmentDur / 2) grainDur = segmentDur + xfadeSec For each dictionary gesture: if gesture >= grainDur: take its first grainDur seconds else: Lengthen toward grainDur (factor capped at 8) rebuild from real sample content tile if still too short cut exactly to grainDur

Phase 3: Feature Extraction and Normalization

Features are extracted from canonical atoms and from the actual reference segments. Each time cell contributes log₂(F0), voiced mask, and intensity. Normalization ranges are then derived jointly from both sets and both sets are normalized in place.

Phase 4: Voice Selection

For every output segment and every voice: 1. Load the appropriate reference-segment feature vector. 2. Compute distance to every dictionary atom. 3. Compute mean candidate distance. 4. Add recency penalty to last 3 choices. 5. Find the top K for this voice. 6. K=1 -> nearest candidate. K>1 -> uniform random pick inside top K. 7. Update the voice's 3-choice history.

Voices 2–4 begin with staggered random history states to avoid identical initial recency conditions across the polyphonic layers.

Phase 5: Fixed-Hop Assembly

Each canonical atom is: segmentDur + crossfade Concatenate with overlap subtracts: crossfade at every internal join Therefore the hop remains exactly: segmentDur After assembly: trim to targetOutputDur apply one head fade + one tail fade only scale each voice by Voice_levels_dB

There are no per-piece fades at internal joins; Praat's overlap crossfade is the only internal envelope, avoiding the old double-fade level dip.

Phase 6: Mono / Stereo Mix

1 voice: mono output, peak-scaled to 0.95.

2 voices: pan positions −0.55 / +0.55.

3 voices: −0.70 / 0 / +0.70.

4 voices: −0.75 / −0.25 / +0.25 / +0.75.

Multi-voice rendering uses equal-power pan gains, sums to stereo, and peak-scales the final stereo Sound to 0.95.

Output Naming

1 voice: gesture_quant__ 2-4 voices: gesture_quant__v_

Applications

Audio Mosaicing

Use a reference whose pitch/voicing/intensity trajectory you want to map onto a discrete gesture corpus. The Leader provides the closest atom-by-atom realization available under the defined feature metric.

Polyphonic Gesture Clouds

Balanced and Maximum Variety add stochastic voices around the Leader. These layers are not “better matches”; they intentionally broaden the candidate set and create simultaneous alternative realizations of the same reference timeline.

Rhythmic / Prosodic Mapping

The feature vector includes intensity contour and voicing state as well as pitch. Speech, vocal gestures, instrumental phrases, and dynamic envelopes can therefore steer the dictionary sequence, but the script does not explicitly model phonemes, semantic content, beat locations, note labels, or timbre embeddings.

Extended Duration

Set Output_duration above the reference duration to continue the mosaic. The reference feature sequence repeats cyclically while stochastic voices can continue choosing different dictionary atoms on each pass.

Controlled Reproducibility

Use a positive Random_seed to recover stochastic Shadow/Wander/Scatter decisions. Voice 1 is deterministic for a fixed reference, dictionary, parameters, and recent-history state, while Voices 2–4 depend on the RNG.

Interpretation caution: “Following a melody” or “reconstructing speech” should be understood only in terms of the script's pitch/voicing/intensity feature trajectory. It does not perform note recognition, transcription, phoneme recognition, semantic analysis, or general perceptual timbre matching.

Complete Workflow

Recommended User Workflow

Put the reference and dictionary material in one folder using WAV, WAV-uppercase, AIFF, AIF, or FLAC.
Type Reference_filename explicitly whenever you do not want to depend on alphabetical ordering.
Start with Balanced.
Listen first to the Leader behaviour: one voice gives the clearest hard-quantization result.
Add Shadow/Wander/Scatter only when you want a polyphonic extension rather than stricter matching.
Use Repetition_penalty to discourage recent reuse; remember it affects the last three choices with decreasing strength.
Use Crossfade_ms for boundary smoothing; it does not change the hop because atoms include the crossfade margin.
Use a positive Random_seed for reproducible stochastic takes.

Visualization

v2.2 Diagnostic Page

Reference waveform with dashed segment boundaries.
Quantized output waveform shown on the same amplitude scale; stereo output is converted to mono only for display.
Quantization Map — Voice 1 cells show the central transformation law, reference segment → chosen dictionary gesture. Voices 2–4 appear as offset coloured markers.
Match Distance by Voice — all active voice-distance trajectories share one y-range; the dashed line is the Voice 1 mean.
Voice Pan — explicit pan position for each voice.
Shared role legend — Leader, Shadow, Wander, Scatter.
Summary strip — material, dictionary size, segment count/duration, crossfade, selection mode, K, repetition penalty, Voice 1 distance statistics, seed, gesture usage, output mode, duration, and peak.

Troubleshooting

Reference file not found: Reference_filename must exactly match a scanned file name. Leave it blank only when the alphabetically first file is intentionally the reference.

Pitch contributes little or nothing: Very short segments may require an effective Pitch floor above the user setting. If no voiced frames are found anywhere, the script reports that pitch carries no information and matching relies on the remaining channels.

Repeated atom despite a high penalty: The penalty discourages the last three selections but does not ban them. If one candidate remains sufficiently better, it may still win in Voice 1 or remain inside a stochastic top-K set.

Voice 4 ignores matching quality: This is intentional. Scatter uses K equal to the dictionary size, making selection uniform across the whole corpus; distance ranking and the repetition penalty cannot change eligibility.

Output is stereo although source files were mono: Multi-voice output is intentionally panned to stereo. One-voice output remains mono.

Info Window Statistics

The detailed summary reports preset, reference, dictionary size, voice count, segment duration, K per active voice, repetition penalty, Voice 1 mean/std/min/max distance, distinct gesture usage, maximum reuse, final output name, duration, and channel count. Distance and usage statistics are explicitly Voice 1 statistics, not an average across all polyphonic layers.