Latent STFT Decoder β€” User Guide

Trains a convolutional Beta-VAE on log-magnitude STFT patches extracted from event-segmented audio, then navigates the latent space to synthesise new audio via decoded STFT patches. Waveform reconstruction via phase borrowing or Griffin-Lim.

Author: Shai Cohen Affiliation: Department of Music, Bar-Ilan University, Israel Version: 1.0 (2025) License: MIT License Citation: Cohen, S. (2025). Praat AudioTools Repo: https://github.com/ShaiCohen-ops/Praat-plugin_AudioTools
Contents:
What this does Quick start STFT VAE Theory Preset Strategies Parameters & Controls Visualization & Analysis Applications

What this does

This script implements a Latent STFT Decoder β€” a system that trains a convolutional Beta-VAE on log-magnitude STFT patches extracted from event-segmented audio, then navigates the latent space to synthesise new audio via decoded STFT patches. Waveform reconstruction is achieved via phase borrowing from nearest real events or iterative Griffin-Lim phase reconstruction.

πŸŽ›οΈ What is STFT VAE Decoding?

This approach combines several advanced techniques:

  • Event segmentation: Audio is divided into events (200ms–3s)
  • STFT patches: For each event, extract log-magnitude STFT patches (F Γ— T)
  • Downsampled VAE: Patches are block-averaged to a smaller grid (vae_freq Γ— vae_frames) for fast training
  • Beta-VAE: Learns a latent space with adjustable KL weight (Ξ²) for disentanglement
  • Latent navigation: Three modes: interpolate (cyclic through events), random_walk, drift (coherent with momentum)
  • Waveform reconstruction: Phase borrowed from real events or Griffin-Lim iteration
  • Visit penalty: Prevents repetition by inflating distances for recently used events

Key Features:

Technical Implementation: (1) Event Segmentation: Intensity-based detection. (2) STFT Patches: log-magnitude STFT [F Γ— patch_frames] per event. (3) Downsampling: Block average to vae_freq Γ— vae_frames for VAE input. (4) Beta-VAE: Train on downsampled patches with adjustable Ξ². (5) Latent Navigation: Generate trajectory through latent space. (6) Decode: Upsample decoded patches, reconstruct waveform via phase borrow or Griffin-Lim. (7) Assemble: Crossfade segments, normalize, stereo Haas delay.

Quick start

  1. In Praat, select exactly one Sound object (any duration, any content).
  2. Run script… β†’ select LatentSTFTDecoder.praat.
  3. Choose Preset (2-4 for specific strategies, 1 for custom).
  4. Set latent size, duration, seed.
  5. Configure VAE training parameters (beta, epochs, batch size).
  6. Set STFT parameters (n_fft, hop_length, patch_frames).
  7. Set VAE grid size (vae_freq, vae_frames) β€” smaller = faster.
  8. Choose navigation mode and parameters (steps, K, step size, temperature, etc.).
  9. Select phase mode, normalization mode.
  10. Enable Draw_visualization for analysis display.
  11. Click OK β€” engine segments, extracts STFT patches, trains VAE, navigates, reconstructs.
Quick tip: Start with Standard preset on a 10-20 second recording with varied texture. Enable visualization β€” you'll see the original waveform with event boundaries, the reconstructed waveform, and spectrograms. Listen to how the VAE learns to reconstruct and morph between events. The output appears as "source_stftdec" in the Objects window.
Important: PYTHON DEPENDENCIES β€” Requires numpy, scipy, soundfile. VAE GRID SIZE (vae_freq Γ— vae_frames) is the key speed control β€” smaller = faster but lower resolution. PHASE MODE "borrow" is faster but less flexible; "griffinlim" can create novel phases but slower. VISIT PENALTY prevents repetition β€” adjust visit_weight and visit_decay. TELEPORT PROBABILITY helps explore new regions.

STFT VAE Theory

STFT Patch Extraction

For each event, compute STFT with: β€’ n_fft (default 512) β€’ hop_length (default 128) β€’ patch_frames (default 32) fixed time frames per patch Result: patches [N, F, T] where: β€’ N = number of events β€’ F = n_fft//2 + 1 (frequency bins) β€’ T = patch_frames Complex STFTs are stored separately for phase borrowing.

Downsampling for VAE

πŸ“‰ Block Averaging for Speed

To keep VAE weights small and training fast, patches are block-averaged:

vae_freq = desired frequency bins (default 32) vae_frames = desired time frames (default 16) For each target frequency bin i: f0 = round(i Γ— F / vae_freq) f1 = max(f0+1, round((i+1) Γ— F / vae_freq)) vae_patches[:, i, :] = mean(patches[:, f0:f1, :], axis=1) Then similarly for time frames: vae_patches[:, :, j] = mean(vae_patches[:, :, t0:t1], axis=2)

This reduces a typical 257Γ—32 patch (8224 dimensions) to 32Γ—16 (512 dimensions) β€” 16Γ— smaller, enabling fast training.

Beta-VAE Architecture

Encoder: input (vae_freqΓ—vae_frames) β†’ hidden (h) β†’ (ΞΌ, log σ²) Decoder: z ∼ 𝒩(ΞΌ, Οƒ) β†’ hidden (h) β†’ output Loss = MSE_recon + Ξ² Γ— KL[𝒩(ΞΌ,Οƒ) || 𝒩(0,I)] where Ξ² (beta) controls disentanglement β€” higher Ξ² = more factorized latent dimensions.

Latent Navigation Modes

🧭 Three Trajectory Types

ModeAlgorithmEffect
interpolateCyclic interpolation between all eventsSmooth, predictable morphing through all identities
random_walkRandom steps + occasional teleportExploratory, unpredictable
driftCoherent direction with momentum + teleportDirected exploration, gradual evolution

Step size scaled by median nearest-neighbor distance in latent space.

Teleport probability (p_jump) jumps to random event to break out of local regions.

Visit Penalty System

visit_scores[i] tracks recent usage, decaying each step by visit_decay. When selecting nearest events for barycentric mixing: adj_dists = raw_dists Γ— (1.0 + visit_weight Γ— visit_scores) Higher visit_weight = stronger penalty for recently used events. This prevents the same event from being chosen repeatedly.

Phase Reconstruction Modes

borrow: Weighted complex STFT mix from K nearest events Zxx_mixed = Ξ£ w_i Γ— Zxx_i magnitude scaled to match VAE output phase from Zxx_mixed preserved griffinlim: Iterative phase reconstruction Initialize with random phase For n_iter: ISTFT β†’ STFT β†’ replace magnitude with VAE output Converges to magnitude with consistent phase

Preset Strategies

Preset 2: Quick (small, fast)

⚑ Fastest Option

Latent: 6 | Epochs: 15 | n_fft: 256 | Hop: 64

Patch frames: 16 | VAE grid: 16Γ—8 | Nav steps: 20

Character: Small VAE, minimal settings β€” fastest processing, lower quality

Use on: Quick previews, experimentation

Preset 3: Standard

βš–οΈ Balanced

Latent: 8 | Epochs: 30 | n_fft: 512 | Hop: 128

Patch frames: 32 | VAE grid: 32Γ—16 | Nav steps: 30

Character: Balanced speed/quality β€” good for most uses

Use on: General purpose, exploration

Preset 4: High Quality

✨ Maximum Quality

Latent: 16 | Epochs: 60 | n_fft: 1024 | Hop: 256

Patch frames: 64 | VAE grid: 48Γ—24 | Nav steps: 50

Character: High-resolution STFT, larger VAE, more epochs β€” best quality, slower

Use on: Final renderings, high-fidelity material

Parameters & Controls

Core Parameters

ParameterDefaultDescription
Latent_size8VAE latent dimensions (2–64)
Duration (0 = original)0Target output duration (seconds)
Seed42Random seed for reproducibility

VAE Training Parameters

ParameterDefaultDescription
Beta (KL_weight)0.5Weight for KL loss (higher = more disentangled)
Epochs30Training epochs (5–500)
Batch_size8Batch size for training (1–64)

STFT Parameters

ParameterDefaultDescription
N_fft512FFT size (64–2048)
Hop_length128Hop length (1–n_fft)
Patch_frames32Fixed time frames per patch (4–256)

VAE Grid Size (key speed control)

ParameterDefaultDescription
Vae_freq32Frequency bins fed to VAE after downsampling
Vae_frames16Time frames fed to VAE after downsampling

Latent Navigation Parameters

ParameterDefaultDescription
Nav_modeinterpolateinterpolate, random_walk, drift
Nav_steps30Number of navigation steps (4–1000)
K_neighbors4K nearest events for barycentric mixing (1–10)
Step_size0.30Movement step size (0–5)
Temperature0.25Noise level for exploration (0–5)
P_jump (teleport_prob)0.05Probability of teleport to random event (0–1)
Visit_weight2.0Strength of visit penalty (higher = less repetition)
Visit_decay0.92Decay factor for visit scores per step

Output Parameters

ParameterDefaultDescription
Normalize_modermsnone, peak, rms
Phase_modeborrowborrow (phase from real events) or griffinlim

Output

ParameterDefaultDescription
Draw_visualization1Generate 5-panel analysis display
Play_result1Audition after processing

Visualization & Analysis

5-Panel Display

Latent STFT Decoder Visualization: Panel 1: TITLE ‒ Script name, source name, preset, nav mode, latent size, n_fft Panel 2: INPUT WAVEFORM ‒ Gray waveform with red dotted lines = event boundaries ‒ Title: "Original (N events)" Panel 3: OUTPUT WAVEFORM ‒ Blue waveform = STFT-decoded output ‒ Title: "STFT dec" ‒ X-axis: Time (s) Panel 4: INPUT SPECTROGRAM ‒ 0-5000 Hz spectrogram of original ‒ Title: "Original spectrogram" Panel 5: OUTPUT SPECTROGRAM ‒ 0-5000 Hz spectrogram of STFT-decoded output (L channel) ‒ Title: "STFT-decoded output spectrogram (L channel)" Panel 6: CONFIG PANEL ‒ STFT configuration: n_fft, hop, patch_frames, freq_bins ‒ VAE configuration: latent, beta, epochs, loss ‒ Navigation: mode, steps, phase mode ‒ Events, duration, normalize mode ‒ Title: "VAE + STFT configuration:" Panel 7: SUMMARY PANEL ‒ Events, nav steps, mode, phase mode ‒ VAE loss (initial→final), latent, seed ‒ Duration in/out, normalize mode, RMS comparison ‒ STFT parameters summary ‒ Warnings if any

Reading the Config Panel

What the numbers mean:
  • STFT: n_fft (frequency resolution), hop (time resolution), patch_frames (event length), freq_bins (FFT/2+1)
  • VAE: latent size, beta (KL weight), epochs trained, loss reduction %
  • Nav: navigation mode, steps taken, phase mode (borrow/griffinlim)
  • Events: number of events, original duration β†’ output duration, normalize mode

Comparing Spectrograms

What to look for:
  • Original: Detailed spectral structure from source
  • Output: Reconstructed from VAE-decoded patches β€” should preserve broad spectral shape
  • Phase mode: "borrow" may sound more realistic (phase from real events); "griffinlim" may have artifacts but more flexibility

Applications

Electroacoustic Composition

Use case: Creating spectral morphs and variations from source material

Technique: Standard preset with interpolate navigation

Workflow:

Sound Design for Media

Use case: Creating evolving textures, spectral transformations

Technique: Quick preset for preview, High Quality for final

Applications:

Music Production

Use case: Creating spectral variations, morphing between sounds

Technique: Different navigation modes on same source

Examples:

Research & Education

Use case: Studying VAE latent spaces, spectral reconstruction, phase reconstruction

Technique: Compare phase modes on same source

Learning outcomes:

Practical Workflow Examples

🎬 Film Scene: Spectral Morph

Goal: Create 60-second spectral morph from 30-second ambient recording

Settings:

  • Source: 30-second ambient
  • Preset: High Quality
  • Nav: interpolate, steps=60
  • Phase: borrow (realistic)

Result: Smooth spectral evolution through all events β€” continuous ambient morph

🎚️ Electronic Music: Random Texture

Goal: Create unpredictable texture from synth stab

Settings:

  • Source: 8-second synth stab
  • Preset: Quick
  • Nav: random_walk, temp=1.0, p_jump=0.2

Result: Unpredictable spectral variations β€” glitchy texture

πŸŽ™οΈ Voice Processing: Spectral Exploration

Goal: Explore spectral space of vocal recording

Settings:

  • Source: 10-second vocal phrase
  • Preset: Standard
  • Nav: drift, step_size=0.4, temp=0.3

Result: Coherent drift through vocal spectral space β€” explores variations

Troubleshooting Common Issues

Problem: Python not found or missing packages
Cause: Python not installed, or packages missing
Solution: Install Python and required packages: pip install numpy scipy soundfile
Problem: VAE loss not decreasing
Cause: Too few epochs, too small latent, or data too complex
Solution: Increase epochs, increase latent_size, or use smaller vae_grid
Problem: Output has clicks
Cause: Crossfade insufficient at segment boundaries
Solution: Increase xfade_sec in assemble_output() (currently 10ms)
Problem: Output repetitive (looping)
Cause: Visit penalty too low, or teleport probability too low
Solution: Increase visit_weight, increase p_jump, reduce visit_decay
Problem: Griffin-Lim artifacts (phasiness)
Cause: Too few iterations, or inconsistent magnitude
Solution: Increase gl_n_iter, or use borrow mode

Advanced Techniques

Custom VAE architecture:

In NumpySTFTVAE class, modify hidden layer size calculation or add more layers for deeper networks.

Block averaging modification:

In _block_avg(), change from mean to max or median for different downsampling characteristics.

Visit penalty tuning:

Adjust visit_weight and visit_decay to control repetition vs. variety. Higher visit_weight = more variety, but may lose coherence.

Multi-channel input:

Script extracts mono for analysis; output is stereo with Haas delay (0.3ms). For true stereo processing, modify to process each channel separately.