Corpus Concatenative Codec — Neural Codec Corpus Synthesis
Corpus-based concatenative synthesis using a neural audio codec (EnCodec or DAC) as the matching token space. Four modes: Match Build corpus Draw Gesture rhyme. Draw now uses an interactive Praat Demo-window contour surface with corpus-brightness feedback, built-in gesture generators, audible preview, and reusable contours.
What this does
This script implements corpus-based concatenative synthesis using a neural audio codec (EnCodec or DAC) as the matching token space. It offers four modes: Match (synthesise from a selected Sound), Build corpus (encode a corpus once), Draw (compose a time-varying brightness path in an interactive Demo window and let the corpus voice it), and Gesture rhyme (re-voice an abstract kinetic gesture using codec-token transition structure).
Key Features:
- 4 Modes — Match, Build corpus, Draw, Gesture rhyme
- 4 Match Presets — Rhythmic, Textural, Faithful, Sparse
- Interactive Draw surface — edit a 0..1 brightness contour directly in a Praat Demo window; no RealTier editor is required
- Corpus-aware Draw feedback — grain-density shading, red no-grain bands, and an orange step trace showing the nearest available brightness at each grain-rate step
- Draw generators + transforms — Sine, Triangle, Angular, cycle count, Same wave faster ×2, Same wave slower ÷2, Undo, Reset, and optional range stretching
- Audible Draw preview — Preview uses the same synthesis path and parameters as Render, then returns to editing
- Reusable Draw contours — every rendered Draw gesture is saved as a compact
corpus_draw_contourfile for exact replay and controlled comparisons - Gesture rhyme mode — hashed-bigram kinetic matching, re-voice abstract gestures
- Sequence context — average source features over preceding sub-windows for sustained transition structure
- Codec support — EnCodec, DAC, or mock (no torch dependencies for testing)
- Onset-detected segmentation — source rhythm drives output timing
- Sub-window subdivision — decaying/evolving segments match new corpus material as they change
- Timbre + loudness matching — cosine distance on token features + log-RMS energy term
- Optional TextGrid — grain provenance tier
Quick start
- Prepare a corpus audio folder (sounds to use as source material).
- First time: run Build / replace corpus — set
Corpus_audio, choose DAC or EnCodec, and set the corpus grain/hop values. The analysed corpus is cached and reused. - Choose a mode. Match and Gesture rhyme require a selected Sound object; Draw does not.
- For Draw, choose Interactive (Demo window) or Contour file. Interactive Draw opens the dedicated contour surface; a saved contour replays its stored time→brightness trajectory and duration.
- In Interactive Draw, click empty space to add points; click a point and then click elsewhere to move it; Shift-click a point to delete it. Use the on-screen generators/transforms, Preview (P), and Render (Enter).
- For Gesture rhyme, adjust Bigram_weight (high = kinetic matching), Hist_weight (low = de-emphasise timbre), and Sequence_context (context smoothing).
- The Python backend synthesises the result and Praat imports it as a new Sound. A rendered Draw run also saves the normalized contour actually used, so the gesture can be reproduced later.
pip install numpy scipy soundfile. For EnCodec: pip install encodec. For DAC: pip install dac. The corpus index is built once and reused. Gesture rhyme requires an existing index — it never builds or reslices the corpus. The "mock" codec works without torch and is useful for testing.
4 Modes
Match (synthesise from corpus)
Select a Sound in Praat. The script detects onsets, subdivides into analysis windows, matches each sub-window against the corpus (timbre + loudness), and reconstructs the output.
Use: Transform any sound using a corpus of your choice.
Build corpus index
One-time encoding of a corpus folder. Slices audio into overlapping grains, encodes each into codec tokens, stores features and metadata.
Use: Prepare a corpus for Match, Draw, and Gesture rhyme modes.
Draw (brightness contour → corpus)
Draw and edit a time→brightness contour in a dedicated Praat Demo window. The display shows the corpus's own brightness distribution and previews which available brightness each grain-rate step will select.
Use: Compose a path through the corpus, audition it before rendering, and save the exact contour for later replay.
Gesture rhyme NEW
Re-voice an abstract kinetic gesture using codec-token transition (hashed-bigram) rhyming. High bigram weight, low histogram weight. A click-train can be voiced by speech syllables or field recordings that move the same way.
Use: Voice abstract kinetic shapes — accelerating clicks, bouncing balls, explosive attacks, tremolo flutter.
Draw Mode — Interactive Brightness-Contour Composition
Draw mode treats the analysed corpus as a navigable brightness field. Each corpus grain has a spectral-centroid value; the backend maps the corpus's log spectral-centroid range to a normalized 0..1 brightness axis (0 = darkest, 1 = brightest). The drawn contour is sampled at the chosen grain rate, and each step selects the corpus grain whose available brightness is nearest to the target, with repeat penalty applied during the actual render.
Editing the contour
| Action | Result |
|---|---|
| Click empty space | Add a contour point. |
| Click a point, then click elsewhere | Move the selected point. Interior points move in time and value; the two end points remain fixed in time and move vertically only. |
| Shift-click a point | Delete an interior point. End points cannot be deleted. |
| U / Undo | Undo the last point edit, generator, or time transform. |
| R / Reset | Return to the default dark→bright ramp. |
| S / Stretch | Toggle legacy min/max stretching of the drawn range to 0..1. |
| P / Preview | Render and play the current gesture using the same backend Draw call and synthesis parameters as the final render, then return to editing. |
| Enter / Render | Render the current gesture, import the result, and save the normalized contour actually used. |
| Esc / Cancel | Leave Draw without rendering. |
Built-in gesture generators and time transforms
Sine, Triangle, and Angular rewrite the editable point list across the whole Draw duration. The cycles − / + controls set 1–32 cycles. The generator range is taken from the current drawing's lowest and highest values; if the drawing is essentially flat (range < 0.02), the generator uses the full 0..1 brightness range. Angular creates a new jittered zigzag variant each time it is invoked.
Same wave faster ×2 compresses the current drawing into half the duration and repeats it twice. Same wave slower ÷2 takes the first half of the current drawing and stretches it over the full duration. These operations modify the current contour and remain fully editable and undoable.
Absolute brightness vs. Stretch
By default, the vertical axis is absolute within the current corpus: a point at 0.25 targets the lower quarter of that corpus's normalized log-brightness range, while a point at 0.75 targets the upper quarter. Turning Stretch drawn range on reproduces the older RealTier behaviour: the minimum and maximum values in the current drawing are expanded to 0 and 1 before synthesis; a completely flat drawing maps to 0.5. When Stretch is active, the window also shows the effective stretched contour.
Preview, Render, and reusable contour files
Preview writes a temporary contour and calls the same Draw backend with the same grain rate, crossfade, repeat penalty, duration, and effective stretch result used by Render. It produces a temporary WAV, plays it in full, deletes it, and returns to the editor. Preview does not save a persistent contour copy.
Render writes the gesture in the compact human-readable corpus_draw_contour 1 format and asks the backend to save the normalized time→brightness trajectory actually used. The persistent copy is stored under Praat's preferences directory in corpus/draw_contours/. You can later choose Draw source = Contour file and load that file to reproduce the same gesture exactly; the contour file's stored duration is reused.
--tier path for backward compatibility, but the current Praat front-end writes and uses explicit contour files.
4 Match Presets
| Preset | Onset Interval (ms) | Analysis Grain (ms) | Analysis Hop (ms) | Energy Weight | Crossfade (ms) | Repeat Penalty | Character |
|---|---|---|---|---|---|---|---|
| Rhythmic | 40 | 40 | 20 | 1.5 | 10 | 0.10 | Follows transients closely — fast, articulated. |
| Textural | 120 | 120 | 60 | 0.5 | 60 | 0.02 | Smeared, washed — ambient texture. |
| Faithful | 60 | 50 | 25 | 1.0 | 20 | 0.15 | Track source closely — balanced. |
| Sparse | 80 | 100 | 80 | 1.0 | 15 | 0.30 | Distinct granular stutter — sparse. |
Codec Token Space — Features for Matching
Token extraction
Each audio grain is encoded by a neural codec (EnCodec or DAC) into tokens: [n_codebooks, n_frames] of integer token IDs.
Token IDs are categorical symbols — we never take Euclidean distance on raw IDs.
Searchable features
- Per-codebook token histograms — marginal distribution of each codebook (1024 bins) — timbre
- Hashed bigram transitions — ordered token-pair structure kept separate per codebook; each codebook hashes transitions through a 64-bit avalanche mix into 1021 buckets — local sequence / kinetic structure
- Features are L1-normalised and concatenated; compared with cosine distance.
Energy term
Cosine distance on token features captures timbre but not loudness. A near-silent grain and a loud one can have similar token-histogram shapes. The match function adds a separate log-RMS energy term: energy_dist = |log(RMS_corpus) - log(RMS_source)| / 4, weighted by energy_weight.
Gesture rhyme weights
- Bigram_weight (default 4.0) — high = kinetic motion drives the match
- Hist_weight (default 0.5) — low = literal timbre de-emphasised
- Energy_weight (default 0.2) — low = energy can't dominate kinetic match
Gesture Rhyme Mode — Hashed-Bigram Kinetic Matching
- Select a Sound object — the abstract gesture source. This can be a click train, a bouncing ball, an explosive attack decaying into hiss, a microtonal dive, tremolo flutter — anything with a kinetic shape.
- Choose Mode = Gesture rhyme.
- Set Bigram_weight (high = kinetic matching), Hist_weight (low = de-emphasise timbre), Energy_weight (low = energy de-emphasised).
- Set Sequence_context (0–3) to smooth matching over preceding sub-windows.
- Click OK — the system searches the EXISTING corpus for grains whose token-transition structure rhymes with the source.
- Output is voiced by corpus grains that move the same way, regardless of their literal sound.
- Bigram_weight 4.0 — strong kinetic matching
- Hist_weight 0.5 — timbre de-emphasised
- Energy_weight 0.2 — loudness de-emphasised
- Sequence_context 1–3 — average over preceding windows for smoother transitions
- Accelerating clicks — rapid attack → voiced by grains with increasing token-change rate
- Bouncing ball — decaying impacts → voiced by grains with damping structure
- Explosive attack → hiss — sharp onset, noisy decay → voiced by grains with similar envelope
- Microtonal dive — descending pitch glide → voiced by grains with descending token trajectory
- Tremolo flutter — rapid amplitude modulation → voiced by grains with oscillating tokens
Applications
Gesture rhyme — Voice abstract kinetic shapes
Use case: Design a kinetic gesture (accelerating clicks, bouncing ball, explosive attack, microtonal dive) and voice it with corpus grains that move the same way.
Settings: Gesture rhyme mode, Bigram_weight=4.0, Hist_weight=0.5, Sequence_context=1–3.
Match — Rhythmic corpus synthesis
Use case: Replace drum hits or percussive sounds with corpus grains that match the rhythm.
Settings: Rhythmic preset (short analysis windows, high energy weight).
Draw mode — Brightness contour composition
Use case: Compose a piece by drawing brightness contours, using the corpus as a timbral palette.
Settings: Draw mode, choose Interactive (Demo window), set Draw duration and Grain rate, then shape the contour manually or with Sine / Triangle / Angular generators. Use Preview before Render when auditioning live alternatives.
Workflow: Gesture rhyme — Click train → speech syllables
Gesture source: A click train (rapid, decaying clicks).
Corpus: Folder of speech syllables (vowels, consonants).
Settings: Gesture rhyme mode, Bigram_weight=4.0, Hist_weight=0.3.
Result: The output is voiced by speech syllables whose token-transition structure matches the click train — the rhythm and kinetic shape of the clicks, but filled with vocalic content.
Workflow: Gesture rhyme — Bouncing ball → field recordings
Gesture source: A bouncing ball (decaying impacts, decreasing intervals).
Corpus: Field recordings (doors, footsteps, water drops).
Settings: Gesture rhyme mode, Sequence_context=2.
Result: The output voices the bouncing ball with field recordings that share the same kinetic structure — decaying impacts, decreasing time intervals — but using environmental sounds.
Workflow: Draw mode — Brightness glissando
Corpus: Synthesiser patches (bright to dark).
Settings: Draw mode, Interactive (Demo window), 8 s duration, 100 ms grain rate.
Contour: Start from the default 0→1 ramp or draw a new path; the window immediately shows corpus density and the nearest available brightness sequence.
Result: The output travels from darker to brighter regions of the corpus — a corpus-driven spectral glissando. Use Preview to audition the current path before Render.
• No corpus index found: Build corpus mode first, or run Match once to auto-build. Gesture rhyme never builds the corpus.
• Gesture rhyme output sounds like timbre matching: Increase Bigram_weight (8.0–12.0) and decrease Hist_weight (0.1–0.3). The match should be driven by bigram transitions, not histograms.
• Gesture rhyme output is jerky / discontiguous: Increase Sequence_context (1–3) to smooth the match over preceding windows. This privileges sustained transition structure.
• Codec not installed: For EnCodec:
pip install encodec. For DAC: pip install dac. Use "mock" to test without torch.• Draw window has red horizontal regions: those brightness ranges contain no corpus grains; redraw through denser regions or rebuild the corpus with more varied material.
• Draw reports that brightness data is missing: the corpus index was built by an older version; rebuild / replace the corpus so per-grain spectral-centroid data is stored.
• Preview or Render pauses Praat: this is expected while the backend renders (and while Preview plays the result). Preview uses the same synthesis path as Render. On Praat 7.0, file-writing / subprocess operations may also trigger the normal trust prompt.
• Closing the Draw Demo window: closes the interactive Draw session and stops the script; use Cancel if you want an explicit exit.