Cross-Entropy Concatenative Mosaicing — Information-Theoretic Resynthesis

Sound P (source palette) is sliced into frames, each normalised into a spectral PDF. Sound Q (target template) is sliced similarly. For every Q frame, the script searches the entire P library and selects the frame whose PDF minimises the cross-entropy against Q's PDF. The winning P-frames are concatenated in Q's temporal order, producing a resequencing of P's timbral material that tracks Q's spectral progression.

Author: Shai Cohen Affiliation: Department of Music, Bar-Ilan University, Israel Version: 1.2 (2026) License: MIT License Repo: https://github.com/ShaiCohen-ops/Praat-plugin_AudioTools

Contents:

What this does Quick start 5 Presets Theory Musicality Terms Applications

What this does

This script implements information-theoretic concatenative mosaicing using cross-entropy as the matching criterion. Sound P (the source palette) is sliced into equal-length frames, and each frame's magnitude spectrum is downsampled into nBins bands and normalised into a probability density function (PDF). Sound Q (the target template) is sliced the same way. For every Q frame, the script searches the entire P library and selects the frame whose PDF minimises the cross-entropy against Q's PDF. The winning P-frame's raw audio is extracted and the frames are concatenated in Q's temporal order, producing a resequencing of P's timbral material that tracks Q's spectral progression.

What is cross-entropy? Cross-entropy H(P, Q) = -Σ P(x)·log₂(Q(x)) measures how well a target distribution P is modelled by a candidate distribution Q. In this script, Q is the target PDF (from Sound Q), and P is the candidate PDF (from Sound P). The frame that minimises cross-entropy is the one whose spectral distribution best matches the target. This is equivalent to minimising KL divergence: H(Q, P) = KL(Q||P) + constant. The direction is important — v1.0 reversed it, causing one bass-heavy frame to sweep nearly every slot; v1.1 fixed this to KL(Q||P), where every uncovered bin costs ~-log₂(eps) bits.

Key Features:

5 Presets — Fine Detail, Standard, Coarse Texture, Micro Grain, Spectral Precision, plus Custom
Cross-entropy matching — selects frames that best model the target spectrum
Musicality terms — Continuity_bits rewards source-order runs; Variety_bits penalises repetition
Stereo mode — "best + runner-up" creates stereo output from the two best matches
Crossfaded concatenation — eliminates clicks at frame boundaries
Visualisation — reordering map, entropy trace, P-frame usage, centroid overlay, PDF heatmaps

v1.2 stereo mode: "best + runner-up" uses the two-minimum tracking already performed in the matching loop — no extra search cost. The L/R correlation follows match ambiguity: many near-equal candidates widen the image; one dominant candidate narrows it toward mono. The right channel uses the second-best P-frame for each Q-slot, producing a spatially informed stereo field.

Quick start

In Praat, select exactly two Sound objects (Sound P = source palette, Sound Q = target template).
Run script… → CrossEntropy_Concatenative_Mosaicing.praat.
Choose a preset from the dropdown (5 options, plus Custom).
Set Frame_duration_s, Number_of_bins, and musicality terms (Continuity_bits, Variety_bits).
Select Stereo_mode: Single mosaic (best match only) or Stereo (best + runner-up).
Click OK — script analyses P and Q, performs matching, resynthesises output.
Output Mosaiced_Output appears in the Objects window.

Quick tip: Use Standard (100 ms, 100 bins) for general-purpose mosaicing. Fine Detail (30 ms, 120 bins) for detailed spectral tracking. Micro Grain (15 ms, 80 bins) for granular, stuttering textures. Enable Stereo: best + runner-up for a spatially informed stereo field — the width follows match ambiguity. The visualisation shows the reordering map and spectral centroid overlay to verify the match quality.

Important: The two sounds must share the same sample rate. The complexity is O(numFramesQ × numFramesP × nBins) — for long sounds or fine frames, processing may take several seconds to minutes. The script uses vectorised inner products for speed. Crossfaded concatenation (min(5 ms, frame/4)) eliminates clicks at frame boundaries. The musicality terms are zeroable — set Continuity_bits=0 and Variety_bits=0 for pure per-frame argmin matching.

5 Presets

Preset	Frame (ms)	Bins	Continuity	Variety	Character
Fine Detail	30	120	0.3	0.8	Detailed spectral tracking, fine temporal resolution.
Standard	100	100	0.4	0.8	Balanced general-purpose mosaicing.
Coarse Texture	150	60	0.3	0.6	Coarse, textural — faster, less detailed.
Micro Grain	15	80	0.5	1.2	Granular, stuttering — high variety penalty.
Spectral Precision	100	200	0.2	0.6	High spectral resolution, lower continuity.

Continuity vs Variety: Continuity_bits rewards choosing the palette frame that follows the previous choice in source order (creating coherent phrases). Variety_bits penalises re-choosing the immediately previous frame (preventing stuck notes). Both are zeroable for pure per-frame matching.

Theory — Cross-Entropy and KL Divergence

Cross-entropy

H(Q, P) = −Σ Q(x) · log₂(P(x) + ε)

This is the expected log loss when using P to model Q.

Minimising H(Q, P) is equivalent to minimising KL divergence:

KL(Q||P) = H(Q, P) − H(Q) (where H(Q) is constant across candidates)

The frame that minimises cross-entropy is the one whose spectral distribution best explains the target.

Matching cost with musicality terms:

Base cost: H(Q_j, P_i) = −Σ Q_j(b) · log₂(P_i(b) + ε)
Continuity bonus: if i = prevBest + 1, cost -= continuity_bits
Variety penalty: if i = prevBest, cost += variety_bits

The frame with the lowest total cost is selected for each Q-slot.

Why v1.1 changed the direction: v1.0 minimised H(P_i, Q_j) = −Σ P_i log₂(Q_j), which is won by whichever palette frame concentrates its mass on Q's single biggest bin (almost always the lowest band). This caused one bass-heavy P-frame to sweep nearly every slot. v1.1 minimises H(Q_j, P_i) = KL(Q_j||P_i) + constant — the frame that best explains the whole target spectrum, with every uncovered bin costing ~−log₂(ε) bits.

Musicality Terms — Continuity and Variety

Continuity_bits

Rewards choosing the palette frame that immediately follows the previous choice in source order (P-frame index + 1).

Effect: Creates coherent phrases — the output follows the original order of P for stretches, preserving the source's structure.

Range: 0.0 (off) to 1.0+ (strong).

Variety_bits

Penalises re-choosing the immediately previous frame (same P-frame index).

Effect: Prevents stuck notes — the same frame cannot repeat consecutively without a cost.

Range: 0.0 (off) to 1.5+ (strong).

Interaction: Continuity and Variety work together. Continuity rewards source-order runs; Variety prevents repetition. With Continuity=0.4 and Variety=0.8, the output tends to follow P's order but jumps away when a repeated frame would be selected, creating a "sensible" mosaic that avoids stagnation.

Applications

Timbre transfer / spectral tracking

Use case: Sound P is a corpus of sounds; Sound Q is a target sound. The output resequences P's material to match Q's spectral progression, creating a "timbre-transferred" version of Q.

Settings: Standard preset, Stereo: best + runner-up. The output tracks Q's timbral trajectory using P's sounds.

Corpus-based composition

Use case: Sound P is a long collection of phrases; Sound Q is a melody or rhythmic template. The output is a mosaic of P's phrases following Q's structure.

Settings: Fine Detail preset, Continuity=0.5, Variety=0.6. The output preserves P's phrase order while matching Q's spectral shape.

Texture synthesis

Use case: Sound P is a texture (e.g., crackling, wind); Sound Q is a rhythmic or spectral target. The output is a texture that follows Q's envelope.

Settings: Coarse Texture preset, Stereo mode on. The stereo field widens when there are multiple near-equal matches.

Granular / stutter effects (Micro Grain)

Use case: Create stuttering, glitchy textures from a source sound.

Settings: Micro Grain preset (15 ms frames, high variety penalty). The output is a rapid, stuttering mosaic.

Workflow: Voice → Instrumental mosaicing

Sound P: Corpus of instrumental sounds (piano, strings, brass).
Sound Q: Voice recording (melody).
Settings: Standard preset, Stereo: best + runner-up.
Result: The voice's spectral contour is tracked by the instrumental corpus — each frame of the voice is filled by the best-matching instrument sound, creating a mosaic that follows the voice's melody and timbre.

Workflow: Drum loop → Granular stutter

Sound P: Drum loop (source material).
Sound Q: Same drum loop (target — creates a self-mosaic).
Settings: Micro Grain preset (15 ms frames, variety penalty=1.2).
Result: The drum loop is granulated into a stuttering, glitchy version — the output follows the original's spectral contour but is composed of tiny fragments.

Workflow: Ambient texture → Spectral tracking

Sound P: Long ambient drone (texture).
Sound Q: Short melodic phrase (target).
Settings: Fine Detail preset, Continuity=0.3.
Result: The ambient texture is resequenced to follow the melodic phrase's spectral progression — a textured version of the melody.

Troubleshooting:
• Output is silent or very quiet: Check that Sound P and Sound Q have similar amplitude. The matching is spectral (PDF) — amplitude is not preserved. Use the output's Scale peak command.
• One frame dominates the output: Reduce Variety_bits or increase Continuity_bits. If a single P-frame is the best match for many Q-frames, the output will be repetitive.
• Output has clicks: The script uses crossfaded concatenation (min 5 ms or frame/4). If clicks persist, increase the crossfade duration by editing the script.
• Processing is slow: Reduce Number_of_bins (e.g., 60–80) or increase Frame_duration_s. For long sounds, use Coarse Texture preset.
• Stereo mode produces mono output: Ensure Sound P has at least two distinct near-matches for most frames. If the best and runner-up are the same frame (or very similar), the stereo image will collapse.

Visualisation (8-wide canvas)

When Draw_visualization is enabled, the script generates:

Reordering map — blue line shows which P-frame fills each Q-slot (identity reference = grey diagonal).
Match quality — cross-entropy value per Q-frame (lower = better fit).
P-frame usage histogram — how many times each P-frame was selected.
Spectral centroid overlay — target Q (grey) vs. resynthesised P (orange) over time.
PDF heatmaps — target Q spectral PDF vs. resynthesised P spectral PDF.
Summary strip — frame counts, bins, mean entropy, palette coverage, output duration.

The reordering map and centroid overlay are the key diagnostics: they show whether the output tracks the target's spectral progression and how the source material is being used.