Self-Attention Latent Navigation — User Guide

VAE encode → self-attention → navigation plan → output. No external models, no internet, pure NumPy. The system segments audio into events, trains a lightweight VAE, builds a self-attention matrix, generates a navigation plan with four phases (drift, mutate, return, settle), and executes it to produce a new timeline.

Author: Shai Cohen Affiliation: Department of Music, Bar-Ilan University, Israel Version: 1.0 (2025) License: MIT License Citation: Cohen, S. (2025). Praat AudioTools Repo: https://github.com/ShaiCohen-ops/Praat-plugin_AudioTools

Contents:

What this does Quick start Self-Attention Navigation Theory Preset Strategies Parameters & Controls Visualization & Analysis Applications

What this does

This script implements a Self-Attention Latent Navigation engine — a system that segments audio into events, trains a lightweight VAE, builds a self-attention matrix, generates a navigation plan with four phases (drift, mutate, return, settle), and executes it to produce a new timeline. The mixing weights combine geometric distance and attention similarity.

🧠 What is Self-Attention Navigation?

This approach combines several advanced concepts:

VAE encoding: Each event is encoded to a latent mean vector μ via a Variational Autoencoder
Self-attention matrix: Multi-head attention over event latents captures similarity relationships
Four-phase navigation: Drift (geometry-dominant), Mutate (attention-grows), Return (attention-dominant), Settle (geometry-returns)
Hybrid mixing weights: wⱼ ∝ (1/distⱼᵖ) × (A[i,j]ᵠ) — α balances geometry vs. attention
No external plan files: Everything happens in memory, only output.wav persists

Key Features:

7 Preset Strategies — Gentle Drift to Chaotic Plasma, plus Custom
VAE Encoding — Pure numpy VAE with 60-300 training iterations
Multi-Head Self-Attention — Fixed He-init projections (not trained), layer-norm + residual
4 Navigation Phases — drift, mutate, return, settle with smooth parameter transitions
Hybrid Mixing Weights — wⱼ ∝ (1/distⱼᵖ) × (A[i,j]ᵠ) with α controlling balance
Attention-Guided Return — Return phase anchors chosen by cosine similarity in attention space
Smart Resampling — Sinc FFT for 0.7-1.4×, phase vocoder for larger stretches
Comprehensive Visualization — Phase bar, waveforms, spectrograms, summary panel

Technical Implementation: (1) Segmentation: Silence-based or equal intervals. (2) Mel Patches: 64×32 log-mel patches per event. (3) VAE: Train 60-300 iterations, encode to latent μ. (4) Attention: Multi-head (heads × dh) with fixed projections, layer-norm + residual. (5) Plan Generation: 4 phases with smooth parameter transitions. (6) Execution: For each step, compute weights, mix clips, crossfade, append. (7) Time-normalization: Resample to target duration, stereo output.

Quick start

In Praat, select exactly one Sound object (any duration, any content).
Run script… → select self_attention_latent.praat.
Choose Preset (2-8 for specific strategies, 1 for custom).
Set segmentation method and parameters.
Set VAE / latent parameters (latent size, random seed).
Set attention heads (auto-adjusted to divide latent size).
Set navigation plan parameters (steps, duration/energy scale/jitter).
Choose target duration, pitch mode, normalize mode.
Enable Show_visualization for analysis display.
Click OK — engine segments, trains VAE, computes attention, generates plan, executes, reconstructs.

Quick tip: Start with Gentle Drift preset on a 10-20 second recording with varied texture. Enable visualization — you'll see the phase bar showing the four phases (drift, mutate, return, settle). Listen to how the sound evolves through each phase. The output appears as "source_sal" in the Objects window.

Important: PYTHON DEPENDENCIES — Requires numpy and soundfile (scipy optional, unused). ATTENTION HEADS must divide latent size evenly — script auto-adjusts. PHASE DURATIONS are fixed percentages: drift 33%, mutate 33%, return 25%, settle 9%. NO EXTERNAL PLAN FILE — everything in memory. CROSSFADE between segments (12ms) prevents clicks.

Self-Attention Navigation Theory

VAE Encoding

Input: log-mel patch (64 mel bands × 32 frames = 2048 features) Encoder: input (2048) → hidden (h) → (μ, log σ²) Decoder: z ∼ 𝒩(μ, σ) → hidden (h) → output where h = max(z_dim×2, min(256, √(2048×z_dim))) Training: 60-300 iterations, Adam optimiser, β=0.05 for KL.

Self-Attention Matrix

📊 Multi-Head Attention (Fixed Projections)

Z ∈ ℝᴺˣᵈ (event latents) Layer-norm: Zn = (Z - μ_z) / σ_z Residual: Zr = Zn + Z / max(|Z|) For each head h (d/h dimensions): Q = Zr · Wq, K = Zr · Wk scores = (Q·Kᵀ) / √(d_h) A_h = softmax(scores) A = (1/heads) Σ_h A_h Row-normalised: A[i] = A[i] / Σⱼ A[i,j] Attention entropy: H = - Σᵢⱼ A[i,j] log(A[i,j]) / N (nats)

Fixed projections: Wq, Wk are He-initialised random matrices (not trained). This captures a fixed notion of similarity.

Layer-norm + residual: Stabilises the computation and preserves identity.

Four-Phase Navigation Plan

🧭 Phase Proportions and Characteristics

Phase	Portion	α (geometry weight)	Step size	Temperature	K neighbors
drift	33%	0.90 → 0.70	0.05 → 0.25	0.05 → 0.20	2 → 4
mutate	33%	0.70 → 0.30	0.30 → 0.70	0.40 → 0.80	4 → 8
return	25%	0.20 → 0.35	0.30 → 0.10	0.20 → 0.10	4 → 2
settle	9%	0.90	0.20 → 0.05	0.15 → 0.05	3 → 2

Smoothness constraints: |Δ step_size| ≤ 0.15, |Δ temperature| ≤ 0.20, |Δ k| ≤ 2 per step.

Return anchors: In return phase, anchor step chosen from drift phase with highest attention cosine similarity to current step.

Mixing Weight Formula

At each step i, with latent position z, K nearest neighbors at positions z_j, and attention row A[i, j]: p_eff = p × α q_eff = q × (1 - α) geom_j = 1 / (‖z - z_j‖₂²)^(p_eff/2) attn_j = A[i, j]^q_eff w_j = (geom_j × attn_j) / Σ (geom_k × attn_k) Default: p = 2.0, q = 1.0 α controls geometry vs. attention balance: • α = 1 → pure inverse-distance (geometry dominates) • α = 0 → pure attention power

Resampling Strategy

When mixing clips to target length, three methods are used: ratio = target_len / current_len • 0.90 ≤ ratio ≤ 1.10: pad/trim with short fade (no resampling) • 0.70 ≤ ratio ≤ 1.40: sinc FFT resampling (high quality, fast) • otherwise: phase vocoder (preserves pitch, slower)

Preset Strategies

Preset 2: Gentle Drift

🌊 Subtle Evolution

Segmentation: Silences, thresh=25dB, min=0.05s

Latent: 8 | Heads: 4 | Steps: 24

Scale/Jitter: dur=1.0±0.05, eng=1.0

Pitch: preserve_f0 | Norm: rms

Character: Subtle, slow evolution with pitch preserved

Use on: Ambient, gradual transformations

Preset 3: Vocal Texture

🗣️ Fine-Grain Vocal

Segmentation: Silences, thresh=20dB, min=0.03s

Latent: 12 | Heads: 8 | Steps: 48

Scale/Jitter: dur=0.8±0.1, eng=1.0±0.05

Pitch: preserve_f0 | Norm: rms

Character: Fine-grain vocal texture, high attention

Use on: Speech, vocal material

Preset 4: Fragmented Echo

🔊 Scattered, Echoic

Segmentation: Silences, thresh=15dB, min=0.02s

Latent: 8 | Heads: 4 | Steps: 96

Scale/Jitter: dur=0.5±0.3, eng=0.8±0.2

Pitch: preserve_f0 | Norm: rms

Character: Short events, many steps, scattered feel

Use on: Glitch, percussive material

Preset 5: Deep Mutation

🧬 Strong Morphing

Segmentation: Silences, thresh=25dB, min=0.05s

Latent: 24 | Heads: 8 | Steps: 72

Scale/Jitter: dur=1.2±0.15, eng=1.1±0.1

Pitch: preserve_spectral_envelope | Norm: rms

Character: Large latent, high heads, strong morphing

Use on: Experimental, transformative material

Preset 6: Rhythmic Grid

🥁 Pulsed, Rhythmic

Segmentation: Equal intervals, size=0.125s

Latent: 8 | Heads: 4 | Steps: 64

Scale/Jitter: dur=1.0±0.02, eng=1.0±0.05

Pitch: off | Norm: peak

Character: Equal-interval grid, tight jitter, pulse feel

Use on: Rhythmic material, loops

Preset 7: Spectral Smear

🎨 Spectral Transformation

Segmentation: Silences, thresh=30dB, min=0.1s

Latent: 16 | Heads: 4 | Steps: 36

Scale/Jitter: dur=1.5±0.2, eng=0.9±0.1

Pitch: preserve_spectral_envelope | Norm: rms

Character: Large steps, spectral envelope preserved

Use on: Spectral transformations, timbral smearing

Preset 8: Chaotic Plasma

⚡ Maximum Variation

Segmentation: Silences, thresh=10dB, min=0.02s

Latent: 32 | Heads: 16 | Steps: 120

Scale/Jitter: dur=1.0±0.5, eng=1.0±0.4

Pitch: off | Norm: rms

Character: Maximum variation, chaotic, plasma-like

Use on: Extreme transformations, chaos

Parameters & Controls

Segmentation Parameters

Parameter	Default	Description
Segmentation_method	Silences	Silences or Equal_intervals
Silence_threshold (dB)	25.0	dB below which is considered silence
Min_event_duration (s)	0.05	Minimum event length
Min_silence_duration (s)	0.03	Minimum silence length for segmentation
Interval_size (s)	0.25	Equal interval size (for method 2)

VAE / Latent Parameters

Parameter	Default	Description
Latent_size	8	VAE latent dimensions (2–32)
Random_seed	42	Seed for reproducibility

Self-Attention Parameters

Parameter	Default	Description
Attention_heads	4	Number of attention heads (auto-adjusted to divide latent size)

Navigation Plan Parameters

Parameter	Default	Description
Plan_steps	72	Number of navigation steps (8–500)
Duration_scale	1.0	Base duration multiplier per step (0.85–1.15 after clamping)
Duration_jitter	0.0	Random variation in duration scale
Energy_scale	1.0	Base energy multiplier per step (0.80–1.20 after clamping)
Energy_jitter	0.0	Random variation in energy scale

Output Parameters

Parameter	Default	Description
Target_duration (s)	0.0	0 = original duration
Pitch_mode	preserve_f0	off, preserve_f0, preserve_spectral_envelope
Normalize_mode	rms	none, peak, rms
Load_output	1	Import output into Praat
Show_visualization	1	Generate 5-panel analysis display
Play_result	1	Audition after processing

Visualization & Analysis

5-Panel Display

Self-Attention Latent Navigation Visualization: Panel 1: TITLE • Script name, source name, heads, steps, latent size Panel 2: PHASE BAR • Colored bar showing proportion of steps in each phase: - Blue = drift (33%) - Yellow = mutate (33%) - Orange = return (25%) - Green = settle (9%) • Labels with step counts if segment > 8% • Title: "Navigation plan: X steps / Y executed" Panel 3: ORIGINAL WAVEFORM • Gray waveform • Title: "Original" Panel 4: OUTPUT WAVEFORM • Green waveform • Title: "Output" • X-axis: Time (s) Panel 5: ORIGINAL SPECTROGRAM • 0-5000 Hz spectrogram of original • Title: "Original spectrogram" Panel 6: OUTPUT SPECTROGRAM • 0-5000 Hz spectrogram of output • Title: "Output spectrogram" Panel 7: SUMMARY PANEL • Events, steps, executed, output duration • Attention heads, entropy, VAE loss • Pitch mode, normalize mode, RMS comparison

Reading the Phase Bar

What the colors mean:

Blue (drift): Geometry-dominant, small coherent steps — smooth evolution
Yellow (mutate): Attention grows, larger steps — exploratory
Orange (return): Attention-dominant, pull toward anchors — creates tension/resolution
Green (settle): Geometry returns, cool-down — convergence
Labels show step counts when segment > 8% of total

Interpreting Attention Entropy

What entropy means:

High entropy (>2.0): Attention spread evenly — many events considered similar
Low entropy (<1.0): Attention focused on few events — strong clustering
Entropy is reported in nats (natural units) in the summary panel

Applications

Electroacoustic Composition

Use case: Creating structured narratives through four-phase navigation

Technique: Gentle Drift or Deep Mutation presets

Workflow:

Select a 20-60 second recording with varied acoustic states
Run with Gentle Drift preset
Listen to the four-phase arc: drift → mutate → return → settle
Export and use as movement in larger work

Sound Design for Media

Use case: Creating evolving textures, transitions, narrative arcs

Technique: Vocal Texture or Spectral Smear on appropriate sources

Applications:

Voice transformation: Vocal Texture with preserve_f0
Atmospheres: Gentle Drift with long steps
Chaotic moments: Chaotic Plasma with high jitter

Music Production

Use case: Creating evolving pads, rhythmic variations

Technique: Rhythmic Grid for pulse, Fragmented Echo for glitch

Examples:

Pad evolution: Gentle Drift with preserve_spectral_envelope
Rhythmic variations: Rhythmic Grid with peak normalization
Glitch textures: Fragmented Echo with high duration jitter

Research & Education

Use case: Studying self-attention, latent navigation, phase transitions

Technique: Compare presets on same source, examine phase bar

Learning outcomes:

Understand how attention captures event similarity
See how α balances geometry vs. attention in mixing
Observe four-phase navigation structure
Explore relationship between entropy and perceptual clustering

Practical Workflow Examples

🎬 Film Scene: Narrative Arc

Goal: Create 60-second cue with narrative arc

Settings:

Source: 30-second ambient recording
Preset: Gentle Drift
Plan steps: 72, duration_scale=1.2

Result: Four-phase narrative: drift (stable) → mutate (explore) → return (tension) → settle (resolution)

🎚️ Electronic Music: Glitch Breakdown

Goal: Create glitch breakdown from drum loop

Settings:

Source: 8-second drum loop
Preset: Fragmented Echo
duration_jitter=0.5, energy_jitter=0.3

Result: Highly varied, echoic glitch texture

🎙️ Voice Processing: Vocal Journey

Goal: Transform voice through four-phase journey

Settings:

Source: 10-second vocal phrase
Preset: Vocal Texture
Pitch: preserve_f0

Result: Voice drifts, mutates, returns, settles — vocal narrative

Troubleshooting Common Issues

Problem: Python not found or missing packages
Cause: Python not installed, or packages missing
Solution: Install Python and required packages: pip install numpy soundfile

Problem: Too few events detected
Cause: Silence threshold too high/low, or source has no clear events
Solution: Adjust silence_threshold, min_event_duration, or use equal intervals

Problem: Attention heads not dividing latent size
Cause: latent_size not divisible by heads
Solution: Script auto-adjusts heads downward until division works

Problem: Output has clicks
Cause: Crossfade insufficient at splice points
Solution: Increase XFADE_SEC in Python script (currently 12ms)

Problem: Phase bar not showing labels
Cause: Phase segment too small (<8% of total)
Solution: Accept that small phases don't get labels, or adjust plan_steps

Advanced Techniques

Custom phase proportions:

In generate_plan(), modify DRIFT_END, MUTATE_END, RETURN_END to change phase percentages.

Attention matrix modification:

In build_attention(), experiment with different projection initialisations or add training (requires backprop).

Mixing weight exponents:

In _mix_weights(), modify p and q to change sensitivity to distance vs. attention.

Multi-channel input:

Script preserves original channel count — for stereo input, output will be stereo.