Gesture Convolution Transform — Prosodic Convolution

A gesture-based pitch / time / intensity transformation that builds a 2D prosodic feature map, applies anisotropic convolution that smears and couples prosodic dimensions (accent expansion, pitch-glissando shaping, intensity→time coupling, local time dilation around accents), then rebuilds the PitchTier and DurationTier inside a Manipulation object.

Author: Shai Cohen Affiliation: Department of Music, Bar-Ilan University, Israel Version: 1.4 (2026) License: MIT License Repo: https://github.com/ShaiCohen-ops/Praat-plugin_AudioTools

Contents:

What this does Quick start 6 Presets Feature Map (8 dims) Anisotropic Convolution Applications

What this does

This script implements a prosodic convolution transform — it builds a 2-D feature map (time frames × 8 control parameters) from a sound's pitch, intensity, and accent contours, applies an anisotropic convolution that smears and couples prosodic dimensions, then resynthesises the sound via Manipulation (PitchTier + DurationTier replacement) and post-resynthesis intensity shaping. The result is a transformation that feels like a "gesture" propagating through time.

What is "prosodic convolution"? Prosody = pitch, duration, intensity, and accent. Convolution = blending a function with a kernel to produce a smoothed or transformed version. Here, the convolution is anisotropic (different strengths in time vs. parameter dimensions) and coupled (changing one parameter affects others). Accent expansion widens pitch and intensity; pitch curvature feeds back into pitch; intensity slope drives local time dilation. The result is a sound that breathes, warps, and gestures in ways that feel organic rather than algorithmic.

Key Features:

6 Presets — Subtle, Smear, Accent-Expand, Glissando, Warp, Rupture, Custom
8-Dimensional Feature Map — semitone pitch, pitch slope, pitch curvature, local duration factor, intensity (dB), intensity slope, accent strength, voicing mask
Anisotropic convolution — trailing-bias temporal smear, accent expansion, intensity→time coupling, glissando curvature feedback
Manipulation resynthesis — PitchTier + DurationTier replacement, overlap-add resynthesis (phase-vocoder-free)
Continuous envelope shaping — IntensityTier gain with smoothing, avoids click artefacts
Visualisation — pitch contour (original vs transformed), local duration factor, intensity + accent

v1.4 fix: Sound+IntensityTier Multiply creates a new Sound, now captured correctly. Previous versions left the gain unapplied and orphaned the intensity object. The output now correctly applies the continuous gain envelope.

Quick start

In Praat, select exactly one Sound object (mono or stereo).
Run script… → Gesture_Convolution_Transform.praat.
Choose a preset (Subtle, Smear, Accent-Expand, Glissando, Warp, Rupture, or Custom).
If Custom, adjust Effect_amount, Pitch_influence, Time_smear_amount, Intensity_influence, Accent_sensitivity, etc.
Set pitch range (floor/ceiling) and smoothing parameters.
Click OK — script analyses, convolves, resynthesises, outputs Gesture_Convolution_Transform.

Quick tip: Start with Smear for a dramatic forward prosodic blur. Accent-Expand dwells on and inflates accents — great for emphasising rhythmic stress. Glissando exaggerates sliding contours. Rupture maximises everything for broken-beautiful textures. Enable Draw_visualization to see the feature map before/after.

Important: This script uses Praat's Manipulation object (overlap-add resynthesis), not a phase vocoder. It is well-suited for pitch and duration modification of monophonic sounds (voice, solo instruments). On polyphonic or noisy sounds, the PitchTier extraction may be unreliable. The script converts stereo to mono for analysis but preserves the original channel count through resynthesis. Unvoiced regions are protected — pitch derivatives are masked to 0 at unvoiced frames, so consonants and breath aren't smeared.

6 Presets

Preset	Effect	Pitch Infl	Smear	Int Infl	Accent Sens	Max Shift (st)	Dur Factor	Character
Subtle	0.25	0.6	0.2	0.25	0.7	2.0	0.85–1.2	Gentle polish, barely-there shaping
Smear	0.7	1.0	0.85	0.55	1.0	4.0	0.55–1.8	Heavy prosodic blur / temporal overshoot
Accent-Expand	0.75	0.9	0.55	0.9	2.6	5.0	0.5–2.5	Violently dwell on and inflate accents
Glissando	0.8	2.2	0.5	0.35	1.1	9.0	0.7–1.6	Exaggerated sliding contours
Warp	0.85	0.7	1.0	1.0	2.0	4.0	0.4–3.0	Intensity drives time hard; rhythmic dislocation
Rupture	1.0	2.5	1.0	1.0	3.0	12.0	0.35–3.5	Maximum everything; far-out, audibly broken-beautiful

Feature Map (8 dimensions per frame)

Col 1: Semitone pitch

Log-frequency pitch relative to median F0. Voiced only; unvoiced frames are interpolated between nearest voiced neighbours.

Col 2: Pitch slope

First derivative of pitch (rate of change). Masked to 0 at unvoiced boundaries to avoid false gestures.

Col 3: Pitch curvature

Second derivative of pitch — detects inflection points and bending.

Col 4: Local duration factor

Time dilation/compression factor. Starts at 1.0; modified by intensity slope and accent.

Col 5: Intensity (dB)

Log amplitude from Intensity object. Used for accent detection and loudness shaping.

Col 6: Intensity slope

Rate of change of intensity — attack/decay rate.

Col 7: Accent strength

Product of intensity above baseline and absolute pitch slope. Smoothed across time.

Col 8: Voicing mask

0 = unvoiced, 1 = voiced. Preserved exactly — never convolved.

Derivative calculation (masked):
Pitch slope = (p_{i+1} - p_{i-1}) / 2 if voiced_i AND voiced_{i-1} AND voiced_{i+1}, else 0.
Pitch curvature = p_{i+1} - 2p_i + p_{i-1} under same condition.
Accent = (intensity_i - mean_intensity) × (1 + |pitch_slope_i|).
All dimensions z-score normalised before convolution.

Anisotropic Convolution — The Gesture Core

Kernel: trailing-bias temporal smear

w(k) = exp(-k/halfWin) for k ≥ 0 (trailing samples weighted more)
w(k) = 0.4 × exp(k/halfWin) for k < 0 (leading samples attenuated)
This produces a causal, forward-overshooting smear — gestures propagate into the future, like a physical impulse.

Couplings

Pitch ← curvature feedback: cP1 = smeared_pitch + 0.5 × pitch_influence × smeared_curvature × accent_gain
Pitch slope ← accent: cP2 = slope × (1 + accent_sensitivity × smeared_accent)
Duration ← intensity slope + accent: cP4 = 1 + time_smear × (0.6×intensity_slope + 0.8×accent)
Intensity ← accent expansion: cP5 = intensity + 0.4 × intensity_influence × smeared_accent

Why "anisotropic"? The convolution kernel is asymmetric in time (trailing bias) and across dimensions (pitch, duration, intensity, accent all have different coupling strengths). This creates a transformation where an accent not only gets louder (intensity expansion) but also stretches the surrounding time (duration dilation) and slopes the pitch (glissando feedback) — exactly like a physical gesture in voice or instrument.

Smoothing before audio: All transformed controls are smoothed into continuous breakpoint functions before they touch audio (PitchTier, DurationTier, IntensityTier). This avoids overlap-add clicks that would arise from frame-rate control changes. Pitch is smoothed with a 20 ms box average; duration with 60 ms (slower control); gain with 20 ms before laying down IntensityTier points.

Applications

Expressive voice transformation

Use case: Make a spoken phrase more "gestural" — exaggerate pitch contours, stretch around accented syllables, and smear consonants into the next vowel.

Settings: Smear preset, effect_amount=0.6. Output sounds like the speaker is leaning into the text with exaggerated prosody.

Rhythmic accent expansion

Use case: Emphasise rhythmic stress — make accented syllables longer, louder, and more pitch-inflated.

Settings: Accent-Expand preset. Violently dwells on and inflates accents — great for percussive vocal or instrumental phrases.

Glissando / portamento exaggeration

Use case: Turn small pitch slides into wide, sweeping glissandi.

Settings: Glissando preset (pitch_influence=2.2, max_shift=9 st). Pitch curvature feedback amplifies slides; output sounds like a theremin or slide guitar.

Time-warp / rhythmic dislocation

Use case: Intensity drives time — loud parts stretch, quiet parts compress, creating a "breathing" or "warping" effect.

Settings: Warp preset. The intensity slope pushes local duration (intensity→time coupling), producing rhythmic dislocation.

Workflow: Spoken phrase → Gestural overdrive

Source: Monologue recording (voice).
Settings: Smear preset, effect_amount=0.7.
Result: The voice takes on a leaning, gestural quality — syllables smear into each other, accents are exaggerated, pitch contours overshoot. Sounds like the speaker is physically reaching for the words.

Workflow: Instrumental solo → Glissando exaggeration

Source: Trumpet or saxophone solo (monophonic).
Settings: Glissando preset, output_gain=1.2.
Result: Small pitch bends become dramatic slides — the instrument sounds like it's constantly gliding between notes.

Workflow: Percussive vocal → Rupture texture

Source: Beatboxing or percussive spoken word.
Settings: Rupture preset (max everything).
Result: The vocal becomes broken-beautiful — extreme time dilation around accents, pitch jumps up to an octave, intensity exaggerated. The output is almost unrecognisable but retains rhythmic structure.

Troubleshooting:
• Output has clicks / artifacts: Increase Pitch_smooth_ms or Duration_smooth_ms. Ensure intensity smoothing is active (Gain_smooth_ms > 10).
• Pitch jumps too far / unnatural: Reduce Max_pitch_shift_semitones and Max_pitch_jump_semitones. Use Subtle preset for safer results.
• Unvoiced regions are smeared: The voicing mask (col 8) is never convolved — unvoiced frames are preserved. If you hear smearing, check that the mask is correctly set (it's based on Pitch object voicing).
• Intensity gain not applied: v1.4 fixes this. If you're on an older version, Sound+IntensityTier Multiply creates a new Sound; ensure the final output is selected from that result.
• Duration factor too extreme: Adjust Min/Max_duration_factor. Warp and Rupture presets have wide ranges (0.35–3.5) — reduce for milder time stretching.

Visualisation (8-wide canvas)

When Draw_visualization is enabled, the script generates three data panels:

Pitch contour — original (grey) vs transformed (red). Shows pitch inflation, overshoot, and smoothing.
Local duration factor — blue curve showing time dilation/compression (>1 = stretched, <1 = compressed). Unity reference line at 1.0.
Intensity + accent — intensity (blue) and accent (orange, scaled). Shows how accents drive intensity expansion.
Legend / summary — preset name, frame count, grid resolution, parameter values.

The visualisation helps you understand which aspects of the prosody are being modified.

Comparison to phase vocoder: Praat's Manipulation uses overlap-add resynthesis (PSOLA-like) which preserves phase relationships better than a standard phase vocoder for pitch/time modifications. The duration factor is applied as a time-stretch via the DurationTier, not as a frame-rate conversion. This makes the Gesture Convolution Transform well-suited for sustained, monophonic material.