Neural Resynthesis Vocoder — HuBERT-Soft to HiFi-GAN

Speech-trained neural resynthesis pipeline: HuBERT-Soft unit extraction → latent operations → acoustic model → HiFi-GAN vocoder. Transform any sound through a neural voice synthesis model with extreme creative controls.

Author: Shai Cohen Affiliation: Department of Music, Bar-Ilan University, Israel Version: 2.6 (2026) License: MIT License Repo: https://github.com/ShaiCohen-ops/Praat-plugin_AudioTools

Contents:

What this does Quick start 8 Presets Latent Unit Operations Pipeline Applications

What this does

This script implements a speech-trained neural resynthesis vocoder using the bshall HuBERT-Soft → acoustic model → HiFi-GAN pipeline. It extracts HuBERT-Soft units from the input audio, applies creative latent operations (temperature scaling, quantization, noise injection), then resynthesises through a neural acoustic model and HiFi-GAN vocoder. The result is a "speech-ified" version of any sound — voices become instrument-like, instruments become voice-like, and extreme parameter settings produce alien textures.

What is HuBERT-Soft? HuBERT is a self-supervised speech representation model. "HuBERT-Soft" refers to the continuous (soft) units extracted from the hidden layers — a timbral representation that captures phonetic and prosodic content. This script lets you manipulate these units directly: scale their magnitude (temperature), round them (quantization), or add noise. The acoustic model then maps these units to a mel-spectrogram, and HiFi-GAN renders audio at 16 kHz.

Key Features:

8 Extreme Presets — Standard Resynthesis, Unit Gain x1.4, Soft-Unit Rounding, Pushed + Heavy Rounding, Coarse Rounding, Extreme Unit Gain x2.5, Noise Injection, Combined Extreme
Temperature scaling — scales unit vector magnitude (1.0 = unchanged, 0.3–0.9 = softer/darker, 1.1–1.5 = brighter/pushed, 2.0+ = extreme/unstable)
Quantization — rounds continuous units to q levels (0 = off, 4 = aggressive crush, 64–256 = subtle rounding)
Noise injection — adds Gaussian noise to units (0.02–0.1 = subtle texture, 0.3–0.6 = stormy)
House-style colour accents — each preset has a fixed accent colour for waveforms and borders (preset-coded, no more black-and-white)
Model caching — models download once to Praat prefs root, then run offline
CPU patching — automatically routes to CPU if GPU unavailable

Speech-trained behaviour: The pipeline is trained on speech (LibriTTS). Non-speech input (music, field recordings, noise) is "speech-ified" — it takes on vocal characteristics. This is a feature, not a bug: creative resynthesis that turns anything into a vocal-like texture.

Quick start

In Praat, select exactly one Sound object.
Run script… → NeuralResynthesisVocoder.praat.
Choose a preset from the dropdown (8 options).
Or select "Custom Settings" and adjust Temperature, Codebook_Quantization_Steps, Noise_injection_scale manually.
Click OK — Python engine runs HuBERT extraction, latent operations, acoustic model, HiFi-GAN resynthesis.
Output originalname_neuralResynth appears in the Objects window.

Quick tip: Start with Standard Resynthesis for a clean neural reconstruction. Unit Gain x1.4 pushes units brighter — good for adding presence. Coarse Rounding (q=4) crushes units aggressively, producing robotic, glitchy textures. Noise Injection adds stormy texture — great for ambient or experimental sounds. Enable Draw_visualization to see the resynthesised waveform and spectrogram with preset-colour accents.

Important: Python dependencies required: pip install torch torchaudio soundfile. The first run downloads ~500 MB of models from torch.hub — requires internet. Subsequent runs work offline from cache. The internal sample rate is 16 kHz, so output is band-limited to 8 kHz regardless of input. The script is CPU-safe (GPU optional) and model caching is located in Praat prefs root (.../Praat/torch_hub_cache).

8 Presets

Preset	Temperature	Quantization	Noise	Character
Standard Resynthesis	1.0	0	0.0	Clean neural reconstruction — speech-trained vocoder.
Unit Gain x1.4	1.4	0	0.02	Brighter/pushed units, subtle noise texture.
Soft-Unit Rounding (q=64)	0.8	64	0.0	Subtle rounding, softer character.
Pushed + Heavy Rounding (q=128)	1.2	128	0.05	Bright + rounded + light noise — pushed texture.
Coarse Rounding (q=4, crush)	1.0	4	0.0	Aggressive unit crush — robotic, glitchy.
Extreme Unit Gain x2.5	2.5	0	0.0	Off-manifold — alien, unstable textures.
Noise Injection	0.5	0	0.4	High noise, reduced unit gain — stormy.
Combined Extreme	0.1	128	0.5	Gain 0.1 + rounding + noise — extreme dark/stormy.

Latent Unit Operations — What the Parameters Actually Do

Temperature (unit-gain scaling)

units ← units × temperature

Scales the magnitude of the HuBERT-Soft unit vectors. This is NOT sampling temperature — the pipeline is deterministic.

< 1.0 (0.3–0.9): pulls units toward origin → softer, darker timbre
= 1.0: unchanged — standard resynthesis
> 1.0 (1.1–1.5): pushes units brighter/pushed
> 2.0: off-manifold — unstable, alien textures

Quantization (soft-unit rounding)

units ← round(units × q) / q

Rounds continuous soft units to q discrete levels. This is a bitcrush on the units, not real codebook/VQ quantization (HuBERT-Soft has no codebook).

0: off — continuous units
4–16: aggressive crush — robotic, glitchy
64–128: subtle rounding — slight texture change
256+: almost imperceptible

Noise injection

units ← units + N(0, σ²)

Adds Gaussian noise to the units.

0.0: off
0.02–0.1: subtle texture — adds grain
0.3–0.6: stormy — high noise, chaotic

Honest naming: The script parameters are named for what they actually do (scaling magnitude, rounding continuous values, adding noise) — not borrowed from diffusion/sampling terminology. The pipeline is deterministic (same input + same params = same output) and does not involve stochastic sampling.

Pipeline: HuBERT-Soft → Acoustic Model → HiFi-GAN

Stage 1: HuBERT-Soft unit extraction
Input audio (resampled to 16 kHz) → HuBERT-Soft → units (continuous latent representation, shape [T, D])

Stage 2: Latent operations (applied in order)
1. Temperature scaling: units ← units × temp
2. Noise injection: units ← units + N(0, σ²)
3. Quantization: units ← round(units × q) / q

Stage 3: Acoustic model → HiFi-GAN vocoder
units → acoustic model → mel-spectrogram → HiFi-GAN → 16 kHz audio

Stage 4: Upsample
Resampled back to original sample rate (44.1 kHz, 48 kHz, etc.)

Model caching: The first run downloads models from bshall's torch.hub repositories (hubert, acoustic-model, hifigan) to .../Praat/torch_hub_cache. Subsequent runs are completely offline. The cache is shared across all AudioTools neural scripts.

Applications

Voice transformation

Use case: Singing or spoken voice → neural resynthesis with creative latents.

Settings: Unit Gain x1.4 for brighter, pushed voice. Soft-Unit Rounding for a smoother, darker character.

Instrument → vocalisation

Use case: Turn any instrument into a vocal-like texture.

Settings: Standard Resynthesis. A piano or guitar will sound like a "speech-ified" version — the model maps instrumental timbres onto vocal acoustic space.

Glitch / robotic textures

Use case: Coarse rounding creates robotic, bit-crushed vocal textures.

Settings: Coarse Rounding (q=4) preset — extreme unit crush, output sounds like a robotic voice.

Extreme / alien textures

Use case: Off-manifold exploration — units pushed far beyond training distribution.

Settings: Extreme Unit Gain x2.5 preset — units scaled to values never seen during training, producing unstable, alien textures.

Workflow: Voice → Robotic glitch

Source: Spoken word or sung melody.
Settings: Coarse Rounding (q=4) preset.
Result: The voice becomes robotic, with stepped pitch and timbral jumps — like a vocoder with a very low bit depth.

Workflow: Piano → Vocalised piano

Source: Piano recording.
Settings: Standard Resynthesis preset.
Result: The piano takes on vocal formants — it sounds like a choir of pianos singing. Non-speech material is "speech-ified" (a creative feature).

Workflow: Field recording → Stormy texture

Source: Field recording (wind, water, traffic).
Settings: Noise Injection preset (noise=0.4, temp=0.5).
Result: The recording becomes a stormy, vocal-like texture — the noise units produce turbulent, organic sound.

Troubleshooting:
• Model download fails: First run requires internet. Check that torch.hub can reach GitHub. If behind a proxy, set environment variables. The cache path is printed in the error log.
• Output is silent or very quiet: The pipeline outputs at 16 kHz. If input is extremely quiet, the model may produce near-silence. Normalise input before processing.
• CPU memory error: The HuBERT model is ~300 MB. If running on low-memory CPU, the script may crash. Use smaller input duration or run on GPU.
• Non-speech material sounds "speech-ified": This is intentional. The model was trained on speech; it maps any input to speech-like acoustic space. If you want a general-purpose vocoder, use a different model.
• Visualisation colours match presets: Each preset has a fixed colour accent (house style). The waveform and borders use this colour, making the preset identity visible at a glance.

Visualisation (House-style colour accents)

When Draw_visualization is enabled, the script generates:

Title bar — accent-coloured stripe, preset name, model info
Waveform — preset-colour curve (e.g., blue for Standard, orange for Unit Gain x1.4)
Spectrogram — with preset-colour frame and title, showing the 8 kHz band-limited output
Summary panel — accent-coloured heading, latent operations (temperature, quantization, noise), source name, output rate

The colour coding makes preset identity immediately obvious and ties the visualisation to the AudioTools house style (see Bayesian Drone Weaver, etc.).

CPU deserialisation patch: The script forces all model loading to map_location='cpu' even if the model was saved on GPU. This makes the pipeline robust across machines — no CUDA errors on CPU-only systems.