Corpus Concatenative Codec — Neural Codec Corpus Synthesis

Corpus-based concatenative synthesis using a neural audio codec (EnCodec or DAC) as the matching token space. Four modes: Match Build corpus Draw Gesture rhyme — re-voice abstract kinetic gestures by hashed-bigram token-transition rhyming.

Author: Shai Cohen Affiliation: Department of Music, Bar-Ilan University, Israel Version: 1.6 (2026) License: MIT License Repo: https://github.com/ShaiCohen-ops/Praat-plugin_AudioTools

Contents:

What this does Quick start 4 Modes 4 Match Presets Codec Token Space Gesture Rhyme Mode Applications

What this does

This script implements corpus-based concatenative synthesis using a neural audio codec (EnCodec or DAC) as the matching token space. It offers four modes: Match (synthesise from a selected Sound), Build corpus (encode a corpus once), Draw (draw a brightness contour and the corpus voices it), and Gesture rhyme (re-voice an abstract kinetic gesture using codec-token transition structure).

What makes this different? The Gesture rhyme mode (new in v1.6) re-voices abstract kinetic gestures — accelerating clicks, a bouncing ball, an explosive attack decaying into hiss, a microtonal dive, tremolo flutter — using corpus grains whose codec-token TRANSITION structure (hashed bigrams) rhymes with the source, NOT whose timbre matches. It weights the bigram section high and the histogram/energy sections low, so a click-train can be voiced by speech syllables or field recordings that simply move the same way frame-to-frame. This is "compositional gestural rhyming" — the system searches for grains whose internal movement patterns match the source's kinetic shape.

Key Features:

4 Modes — Match, Build corpus, Draw, Gesture rhyme
4 Match Presets — Rhythmic, Textural, Faithful, Sparse
Gesture rhyme mode — hashed-bigram kinetic matching, re-voice abstract gestures
Sequence context — average source features over preceding sub-windows for sustained transition structure
Codec support — EnCodec, DAC, or mock (no torch dependencies for testing)
Onset-detected segmentation — source rhythm drives output timing
Sub-window subdivision — decaying/evolving segments match new corpus material as they change
Timbre + loudness matching — cosine distance on token features + log-RMS energy term
Optional TextGrid — grain provenance tier

Gesture rhyme vs. Match: Match mode finds grains whose timbre (token histogram) matches the source. Gesture rhyme finds grains whose movement (token bigram transitions) matches the source. A click-train and a speech syllable both have rapid frame-to-frame token changes; gesture rhyme can voice a click-train with speech syllables, field recordings, or instrument noises that simply move the same way. Literal sound identity is deliberately ignored.

Quick start

Prepare a corpus audio folder (sounds to use as source material).
First time: Run in Build corpus index mode — set Corpus_audio to your folder, choose Codec (dac or encodec), set Grain_ms/Hop_ms, click OK.
After index is built, select a Sound object in Praat (for Match or Gesture rhyme mode).
Choose Mode: Match (timbre-based), Draw (brightness contour), or Gesture rhyme (kinetic matching).
For Gesture rhyme, adjust Bigram_weight (high = kinetic matching), Hist_weight (low = de-emphasise timbre), Sequence_context (smooth transitions).
Click OK — Python backend synthesises the output.
Output appears as a new Sound: originalname_concat_codec (Match) or originalname_gesture_rhyme_codec (Gesture rhyme).

Quick tip: Use Gesture rhyme with a high Bigram_weight (4.0–8.0) and low Hist_weight (0.1–0.5) to voice abstract gestures. Try a source that's just a click train or a bouncing ball — the corpus will voice it with grains that move the same way. Sequence_context (1–3) smooths the matching over preceding windows, privileging sustained transition structure over isolated windows.

Important: Python dependencies required: pip install numpy scipy soundfile. For EnCodec: pip install encodec. For DAC: pip install dac. The corpus index is built once and reused. Gesture rhyme requires an existing index — it never builds or reslices the corpus. The "mock" codec works without torch and is useful for testing.

4 Modes

Match (synthesise from corpus)

Select a Sound in Praat. The script detects onsets, subdivides into analysis windows, matches each sub-window against the corpus (timbre + loudness), and reconstructs the output.

Use: Transform any sound using a corpus of your choice.

Build corpus index

One-time encoding of a corpus folder. Slices audio into overlapping grains, encodes each into codec tokens, stores features and metadata.

Use: Prepare a corpus for Match, Draw, and Gesture rhyme modes.

Draw (brightness contour → corpus)

Open a RealTier editor, draw a brightness curve over time. The corpus synthesises the contour by selecting grains that match the target brightness at each time step.

Use: Compose with corpus material using a continuous gesture.

Gesture rhyme NEW

Re-voice an abstract kinetic gesture using codec-token transition (hashed-bigram) rhyming. High bigram weight, low histogram weight. A click-train can be voiced by speech syllables or field recordings that move the same way.

Use: Voice abstract kinetic shapes — accelerating clicks, bouncing balls, explosive attacks, tremolo flutter.

Gesture rhyme feature-compatibility: The stored corpus feature matrix is split into histogram and bigram sections. The script re-weights each section at match time (hist_weight, bigram_weight) — no corpus rebuild is needed. Sequence_context averages source features over preceding sub-windows, privileging sustained transition structure over isolated windows.

4 Match Presets

Preset	Onset Interval (ms)	Analysis Grain (ms)	Analysis Hop (ms)	Energy Weight	Crossfade (ms)	Repeat Penalty	Character
Rhythmic	40	40	20	1.5	10	0.10	Follows transients closely — fast, articulated.
Textural	120	120	60	0.5	60	0.02	Smeared, washed — ambient texture.
Faithful	60	50	25	1.0	20	0.15	Track source closely — balanced.
Sparse	80	100	80	1.0	15	0.30	Distinct granular stutter — sparse.

Codec Token Space — Features for Matching

Token extraction

Each audio grain is encoded by a neural codec (EnCodec or DAC) into tokens: [n_codebooks, n_frames] of integer token IDs.

Token IDs are categorical symbols — we never take Euclidean distance on raw IDs.

Searchable features

Per-codebook token histograms — marginal distribution of each codebook (1024 bins) — timbre
Hashed bigram transitions — local sequence structure: (token_i * 131 + token_{i+1}) % 64 — kinetic movement
Features are L1-normalised and concatenated; compared with cosine distance.

Energy term

Cosine distance on token features captures timbre but not loudness. A near-silent grain and a loud one can have similar token-histogram shapes. The match function adds a separate log-RMS energy term: energy_dist = |log(RMS_corpus) - log(RMS_source)| / 4, weighted by energy_weight.

Gesture rhyme weights

Bigram_weight (default 4.0) — high = kinetic motion drives the match
Hist_weight (default 0.5) — low = literal timbre de-emphasised
Energy_weight (default 0.2) — low = energy can't dominate kinetic match

Why hashed bigrams? Bigram transitions capture how tokens change frame-to-frame — acceleration, deceleration, oscillation, abrupt shifts. A click-train and a speech syllable both have rapid frame-to-frame token changes; gesture rhyme finds grains whose internal movement patterns match the source's kinetic shape, regardless of what they actually sound like.

Gesture Rhyme Mode — Hashed-Bigram Kinetic Matching

Gesture rhyme workflow:

Select a Sound object — the abstract gesture source. This can be a click train, a bouncing ball, an explosive attack decaying into hiss, a microtonal dive, tremolo flutter — anything with a kinetic shape.
Choose Mode = Gesture rhyme.
Set Bigram_weight (high = kinetic matching), Hist_weight (low = de-emphasise timbre), Energy_weight (low = energy de-emphasised).
Set Sequence_context (0–3) to smooth matching over preceding sub-windows.
Click OK — the system searches the EXISTING corpus for grains whose token-transition structure rhymes with the source.
Output is voiced by corpus grains that move the same way, regardless of their literal sound.

Feature weighting:

Bigram_weight 4.0 — strong kinetic matching
Hist_weight 0.5 — timbre de-emphasised
Energy_weight 0.2 — loudness de-emphasised
Sequence_context 1–3 — average over preceding windows for smoother transitions

Kinetic shapes you can voice:

Accelerating clicks — rapid attack → voiced by grains with increasing token-change rate
Bouncing ball — decaying impacts → voiced by grains with damping structure
Explosive attack → hiss — sharp onset, noisy decay → voiced by grains with similar envelope
Microtonal dive — descending pitch glide → voiced by grains with descending token trajectory
Tremolo flutter — rapid amplitude modulation → voiced by grains with oscillating tokens

Compositional gestural rhyming: The source gesture is an ABSTRACT kinetic shape, not a sound you want to hear. The corpus provides the sonic material. The system finds grains whose internal movement (token transitions) rhymes with the gesture. A click-train can be voiced by speech syllables, field recordings, or instrument noises that simply move the same way. This is a new way to compose: design a gesture, then find corpus material that voices it.

Applications

Gesture rhyme — Voice abstract kinetic shapes

Use case: Design a kinetic gesture (accelerating clicks, bouncing ball, explosive attack, microtonal dive) and voice it with corpus grains that move the same way.

Settings: Gesture rhyme mode, Bigram_weight=4.0, Hist_weight=0.5, Sequence_context=1–3.

Match — Rhythmic corpus synthesis

Use case: Replace drum hits or percussive sounds with corpus grains that match the rhythm.

Settings: Rhythmic preset (short analysis windows, high energy weight).

Draw mode — Brightness contour composition

Use case: Compose a piece by drawing brightness contours, using the corpus as a timbral palette.

Settings: Draw mode, Draw_duration_s = 10–60 s, Grain_rate_ms = 50–200 ms.

Workflow: Gesture rhyme — Click train → speech syllables

Gesture source: A click train (rapid, decaying clicks).
Corpus: Folder of speech syllables (vowels, consonants).
Settings: Gesture rhyme mode, Bigram_weight=4.0, Hist_weight=0.3.
Result: The output is voiced by speech syllables whose token-transition structure matches the click train — the rhythm and kinetic shape of the clicks, but filled with vocalic content.

Workflow: Gesture rhyme — Bouncing ball → field recordings

Gesture source: A bouncing ball (decaying impacts, decreasing intervals).
Corpus: Field recordings (doors, footsteps, water drops).
Settings: Gesture rhyme mode, Sequence_context=2.
Result: The output voices the bouncing ball with field recordings that share the same kinetic structure — decaying impacts, decreasing time intervals — but using environmental sounds.

Workflow: Draw mode — Brightness glissando

Corpus: Synthesiser patches (bright to dark).
Settings: Draw mode, 8 s duration, 100 ms grain rate.
Contour: Draw a straight line from 0 to 1 over 8 seconds.
Result: The output glides from the darkest to the brightest corpus grain — a spectral glissando.

Troubleshooting:
• No corpus index found: Build corpus mode first, or run Match once to auto-build. Gesture rhyme never builds the corpus.
• Gesture rhyme output sounds like timbre matching: Increase Bigram_weight (8.0–12.0) and decrease Hist_weight (0.1–0.3). The match should be driven by bigram transitions, not histograms.
• Gesture rhyme output is jerky / discontiguous: Increase Sequence_context (1–3) to smooth the match over preceding windows. This privileges sustained transition structure.
• Codec not installed: For EnCodec: pip install encodec. For DAC: pip install dac. Use "mock" to test without torch.
• Draw mode RealTier editor not opening: Ensure you have a display server running (X11 on Linux, native on macOS/Windows).

Visualisation & TextGrid

When Import_textgrid is enabled, all modes export a TextGrid with an IntervalTier named "grains". In Gesture rhyme mode, the metadata JSON includes gesture_contribution (bigram match), timbre_contribution (histogram match), and energy_contribution — so you can see exactly which component drove each selection.