Self-Similarity Resynthesis — User Guide
MFCC self-similarity becomes a compositional control signal: recurring or novel material drives gain, gating and spatial modulation, while Self-Mosaic can reorder full-rate source blocks by acoustic similarity.
What this does
Self-Similarity Resynthesis analyzes how similar each moment of a sound is to other moments and uses that structure as a control signal. MFCC feature vectors are normalized, compared pairwise in a self-similarity matrix (SSM), reduced to a per-frame similarity score, mapped to dB gain, smoothed and applied back to the original full-rate waveform.
Self-Similarity_Spectral_Resynthesis, most modes are not spectral resynthesis. The spectrum is analyzed to obtain MFCC descriptors, but the source is normally rendered by time-varying amplitude modulation of the original waveform. Self-Mosaic is the exception: it substitutes audio blocks from similar source locations.What is a self-similarity matrix?
An SSM compares every analysis frame with every other frame. Here each frame is represented by an MFCC vector and L2-normalized. The matrix cell at row i, column j is then the cosine similarity between the two normalized vectors:
The main diagonal is always strongly self-similar because each frame is being compared with itself. Repeated timbres, gestures or sections can produce additional high-similarity structures away from that diagonal. The script converts those pairwise relations into one score per frame and then into a musical control envelope.
Quick start
- Select exactly one Sound.
- Run
Self-Similarity_Spectral_Resynthesis.praat. - Start with the default Gentle Resynthesis preset and Preserve Stereo.
- Use Glitch Gating or Brutal Novelty for aggressive gating; use Spectral Mosaic for block substitution.
- Open Edit_details only when you need analysis bandwidth, MFCC count, thresholds, smoothing or gain calibration.
Presets
Named presets set their own Creative mode. The Creative-mode menu is used directly only with Custom.
| Preset | Creative mode | Core analysis | Character |
|---|---|---|---|
| Custom | User choice | 10 ms step; 30 ms frame; 8 MFCCs | Uses the details-state defaults unless edited. |
| Gentle Resynthesis | Standard | 10 ms / 30 ms / 8 MFCCs | ±4 dB similarity shaping, 60 ms envelope smoothing, no chaos. |
| Glitch Gating | Inverted | 5 ms / 20 ms / 6 MFCCs | Hard gate, short smoothing, small chaos; emphasizes novelty. |
| Ghost Remix | Standard | 8 ms / 25 ms | Strong +10/−12 dB similarity contrast with modest smoothing. |
| Brutal Novelty | Inverted | 5 ms / 20 ms / 5 MFCCs | Aggressive hard-gated novelty with minimal smoothing. |
| Spectral Mosaic | Self-Mosaic | 20 ms / 50 ms | Copies blocks from the most-similar non-neighbouring source frames. |
| Chaotic Tremolo | Hard Quantized | 3 ms / 15 ms / 4 MFCCs | Three-level gain behaviour with 0.35 random score perturbation. |
Creative modes
| Mode | Meaning |
|---|---|
| Standard (Similarity Boost) | High recurrence produces higher similarity scores and therefore more gain. |
| Inverted (Novelty Extractor) | Uses 1 − score, so dissimilar / novel frames become stronger. |
| Diagonal Recurrence | Blends the normal row score with near-diagonal recurrence within ±10 frames. |
| Hard Quantized | Quantizes normalized control values to three levels: 0, 0.5, 1 before dB mapping. |
| Self-Mosaic | For each frame, finds the strongest similarity outside a ±5-frame neighbourhood and copies that source audio block to the target time at 0.7 gain. Frame-boundary clicks are inherent to this intentionally direct technique. |
Spatial modes
| Mode | Behaviour |
|---|---|
| Mono | Applies the envelope to a mono source. Self-Mosaic also becomes mono. |
| Preserve Stereo | Applies the same envelope to every source channel; despite the label, this path preserves multichannel structure as well. |
| Stereo Sweep | Applies opposite time-varying L/R biases around the shared envelope. Mono is duplicated to stereo; multichannel use is fundamentally L/R-oriented. |
| Rotating | Downmixes to mono and creates stereo amplitude motion with 0.2 Hz cosine/sine factors. |
| Soft Modulation | Multiplies each source channel by 0.5 + 0.5×envelope, preserving a 50% floor. |
| True Mid-Side | For stereo, computes M=(L+R)/2 and S=(L−R)/2, applies the envelope only to M, then reconstructs L/R. Mono falls back to ordinary envelope application. |
Edit details
| Control | Default | Role |
|---|---|---|
| Analysis bandwidth | 16 kHz analysis | Downsamples only the analysis driver when the source rate is higher; output rendering always returns to the untouched full-rate source. |
| Time step / frame length | 0.01 / 0.03 s | Temporal spacing and window size for MFCC analysis. |
| Number of MFCCs | 8 | Dimensionality of each feature vector. |
| Similarity threshold | 0.5 | Only off-diagonal SSM cells above this value contribute to the standard row score. |
| Contrast power | 2.0 | Shapes the normalized score before mapping it to dB. |
| High / low gain | +4 / −8 dB | Configured dB range of the gain map. |
| Mask smoothing | 8 frames | Moving average on the similarity score. |
| Envelope smoothing | 50 ms | Moving average on the 1 kHz render envelope. |
| Add chaos | 0 | Unseeded random perturbation in normalized similarity units. |
| Hard gate / threshold | Off / 0.5 | Optional binary gate before contrast/dB mapping. |
| Output gain | 0 dB | Applied after optional peak normalization. |
| Normalize output | On | Target-normalizes non-silent output to peak 0.95 before Output gain. |
Processing pipeline
- Choose one representative analysis channel: for multichannel input, the highest-RMS channel, avoiding anti-phase cancellation from a fold-down.
- Optionally downsample that analysis copy to 16 kHz or 8 kHz.
- Compute MFCC frames, L2-normalize each frame vector and build the full cosine-similarity SSM.
- Compute one score per frame according to the creative mode; optionally smooth, add chaos, gate or quantize.
- Map the centered score distribution through a contrast curve into the configured dB range.
- For envelope modes, interpolate the dB gain map onto a 1 kHz envelope, optionally smooth it, resample the envelope to the original sample rate and multiply the untouched full-rate source.
- For Self-Mosaic, copy full-rate source blocks from the strongest non-neighbouring SSM match instead of applying a gain envelope.
- Apply the selected spatial rendering.
- Optionally target-normalize to 0.95, apply Output gain, and if post-gain peak exceeds 1.0 apply a final down-only safety scale to 0.999.
Output, channels and randomness
- Output name:
<source>_similarity_<preset>. - Sample rate: original source rate; analysis downsampling never limits the output bandwidth.
- Duration: preserved.
- Randomness: only recipes with
Add chaos > 0vary across runs; there is no exposed random seed. - Normalization: with Normalize output enabled, non-silent output is target-normalized to 0.95 before Output gain. Positive gain can raise the result afterwards; only peaks above 1.0 trigger the 0.999 safety ceiling.
Visualization
- Self-similarity matrix: red = high, blue = low, with the main diagonal visible.
- Score and gain panels: show the per-frame structural score and the mapped dB envelope.
- Output waveform: mono or L/R drawing depending on the result.
- Summary: preset, mode, analysis bandwidth, spatial mode, score statistics, gain range and processing information.
Technological and compositional context
Audio self-similarity matrices became an important tool in music-information retrieval because they make repetition and large-scale form visible without requiring a score. Jonathan Foote's 1999 ACM Multimedia paper showed how pairwise acoustic similarity can reveal recurring sections, rhythmic organization and structural boundaries.
This script turns that analytical representation into a compositional controller: recurrence is no longer only displayed or segmented; it becomes a dynamic-gain, gating or source-reordering mechanism. The analysis remains recognizably MIR-style, while the rendering deliberately treats the SSM as musical control data.
Further reading: Jonathan Foote, “Visualizing Music and Audio using Self-Similarity,” Proceedings of ACM Multimedia '99, pp. 77–80. doi:10.1145/319463.319472.