Self-Similarity Resynthesis — User Guide

MFCC self-similarity becomes a compositional control signal: recurring or novel material drives gain, gating and spatial modulation, while Self-Mosaic can reorder full-rate source blocks by acoustic similarity.

Author: Shai Cohen Affiliation: Department of Music, Bar-Ilan University, Israel Version: 3.4.2 (2026) License: MIT License Repo: https://github.com/ShaiCohen-ops/Praat-plugin_AudioTools
Contents:

What this does

Self-Similarity Resynthesis analyzes how similar each moment of a sound is to other moments and uses that structure as a control signal. MFCC feature vectors are normalized, compared pairwise in a self-similarity matrix (SSM), reduced to a per-frame similarity score, mapped to dB gain, smoothed and applied back to the original full-rate waveform.

Name versus mechanism: despite the filename Self-Similarity_Spectral_Resynthesis, most modes are not spectral resynthesis. The spectrum is analyzed to obtain MFCC descriptors, but the source is normally rendered by time-varying amplitude modulation of the original waveform. Self-Mosaic is the exception: it substitutes audio blocks from similar source locations.

What is a self-similarity matrix?

An SSM compares every analysis frame with every other frame. Here each frame is represented by an MFCC vector and L2-normalized. The matrix cell at row i, column j is then the cosine similarity between the two normalized vectors:

SSM[i,j] = Σc N[c,i] × N[c,j]

The main diagonal is always strongly self-similar because each frame is being compared with itself. Repeated timbres, gestures or sections can produce additional high-similarity structures away from that diagonal. The script converts those pairwise relations into one score per frame and then into a musical control envelope.

Quick start

  1. Select exactly one Sound.
  2. Run Self-Similarity_Spectral_Resynthesis.praat.
  3. Start with the default Gentle Resynthesis preset and Preserve Stereo.
  4. Use Glitch Gating or Brutal Novelty for aggressive gating; use Spectral Mosaic for block substitution.
  5. Open Edit_details only when you need analysis bandwidth, MFCC count, thresholds, smoothing or gain calibration.

Presets

Named presets set their own Creative mode. The Creative-mode menu is used directly only with Custom.

PresetCreative modeCore analysisCharacter
CustomUser choice10 ms step; 30 ms frame; 8 MFCCsUses the details-state defaults unless edited.
Gentle ResynthesisStandard10 ms / 30 ms / 8 MFCCs±4 dB similarity shaping, 60 ms envelope smoothing, no chaos.
Glitch GatingInverted5 ms / 20 ms / 6 MFCCsHard gate, short smoothing, small chaos; emphasizes novelty.
Ghost RemixStandard8 ms / 25 msStrong +10/−12 dB similarity contrast with modest smoothing.
Brutal NoveltyInverted5 ms / 20 ms / 5 MFCCsAggressive hard-gated novelty with minimal smoothing.
Spectral MosaicSelf-Mosaic20 ms / 50 msCopies blocks from the most-similar non-neighbouring source frames.
Chaotic TremoloHard Quantized3 ms / 15 ms / 4 MFCCsThree-level gain behaviour with 0.35 random score perturbation.

Creative modes

ModeMeaning
Standard (Similarity Boost)High recurrence produces higher similarity scores and therefore more gain.
Inverted (Novelty Extractor)Uses 1 − score, so dissimilar / novel frames become stronger.
Diagonal RecurrenceBlends the normal row score with near-diagonal recurrence within ±10 frames.
Hard QuantizedQuantizes normalized control values to three levels: 0, 0.5, 1 before dB mapping.
Self-MosaicFor each frame, finds the strongest similarity outside a ±5-frame neighbourhood and copies that source audio block to the target time at 0.7 gain. Frame-boundary clicks are inherent to this intentionally direct technique.

Spatial modes

ModeBehaviour
MonoApplies the envelope to a mono source. Self-Mosaic also becomes mono.
Preserve StereoApplies the same envelope to every source channel; despite the label, this path preserves multichannel structure as well.
Stereo SweepApplies opposite time-varying L/R biases around the shared envelope. Mono is duplicated to stereo; multichannel use is fundamentally L/R-oriented.
RotatingDownmixes to mono and creates stereo amplitude motion with 0.2 Hz cosine/sine factors.
Soft ModulationMultiplies each source channel by 0.5 + 0.5×envelope, preserving a 50% floor.
True Mid-SideFor stereo, computes M=(L+R)/2 and S=(L−R)/2, applies the envelope only to M, then reconstructs L/R. Mono falls back to ordinary envelope application.
Self-Mosaic has no gain envelope. In Mosaic mode, any spatial choice other than Mono falls back to preserving the source channels.

Edit details

ControlDefaultRole
Analysis bandwidth16 kHz analysisDownsamples only the analysis driver when the source rate is higher; output rendering always returns to the untouched full-rate source.
Time step / frame length0.01 / 0.03 sTemporal spacing and window size for MFCC analysis.
Number of MFCCs8Dimensionality of each feature vector.
Similarity threshold0.5Only off-diagonal SSM cells above this value contribute to the standard row score.
Contrast power2.0Shapes the normalized score before mapping it to dB.
High / low gain+4 / −8 dBConfigured dB range of the gain map.
Mask smoothing8 framesMoving average on the similarity score.
Envelope smoothing50 msMoving average on the 1 kHz render envelope.
Add chaos0Unseeded random perturbation in normalized similarity units.
Hard gate / thresholdOff / 0.5Optional binary gate before contrast/dB mapping.
Output gain0 dBApplied after optional peak normalization.
Normalize outputOnTarget-normalizes non-silent output to peak 0.95 before Output gain.

Processing pipeline

  1. Choose one representative analysis channel: for multichannel input, the highest-RMS channel, avoiding anti-phase cancellation from a fold-down.
  2. Optionally downsample that analysis copy to 16 kHz or 8 kHz.
  3. Compute MFCC frames, L2-normalize each frame vector and build the full cosine-similarity SSM.
  4. Compute one score per frame according to the creative mode; optionally smooth, add chaos, gate or quantize.
  5. Map the centered score distribution through a contrast curve into the configured dB range.
  6. For envelope modes, interpolate the dB gain map onto a 1 kHz envelope, optionally smooth it, resample the envelope to the original sample rate and multiply the untouched full-rate source.
  7. For Self-Mosaic, copy full-rate source blocks from the strongest non-neighbouring SSM match instead of applying a gain envelope.
  8. Apply the selected spatial rendering.
  9. Optionally target-normalize to 0.95, apply Output gain, and if post-gain peak exceeds 1.0 apply a final down-only safety scale to 0.999.

Output, channels and randomness

Visualization

Technological and compositional context

Audio self-similarity matrices became an important tool in music-information retrieval because they make repetition and large-scale form visible without requiring a score. Jonathan Foote's 1999 ACM Multimedia paper showed how pairwise acoustic similarity can reveal recurring sections, rhythmic organization and structural boundaries.

This script turns that analytical representation into a compositional controller: recurrence is no longer only displayed or segmented; it becomes a dynamic-gain, gating or source-reordering mechanism. The analysis remains recognizably MIR-style, while the rendering deliberately treats the SSM as musical control data.

Further reading: Jonathan Foote, “Visualizing Music and Audio using Self-Similarity,” Proceedings of ACM Multimedia '99, pp. 77–80. doi:10.1145/319463.319472.