DDSP Neural Revoicing — Timbre Transfer

Renders the selected Sound through a pretrained Magenta DDSP timbre-transfer model (Violin, Flute, Flute2, Trumpet, Tenor_Saxophone). Preserves the input's pitch contour and loudness gesture and re-renders them with the chosen instrument model.

Author: Shai Cohen Affiliation: Department of Music, Bar-Ilan University, Israel Version: 0.4 (2026) License: MIT License Repo: https://github.com/ShaiCohen-ops/Praat-plugin_AudioTools

Contents:

What this does Quick start 5 Target Models Controls Theory Applications

What this does

This script implements DDSP neural timbre transfer using pretrained Magenta DDSP models. It renders the selected Sound through one of five instrument models (Violin, Flute, Flute2, Trumpet, Tenor_Saxophone), preserving the input's pitch contour and loudness gesture and re-rendering them with the chosen instrument's timbre. The heavy DSP is offloaded to a Python engine that downloads a model once, then caches it for offline use.

What is DDSP? DDSP (Differentiable Digital Signal Processing) is a neural audio synthesis framework from Google Magenta. Unlike black-box neural networks, DDSP models are built from interpretable DSP components (oscillators, filters, noise) with learnable parameters. The timbre-transfer models used here are autoencoders trained on solo instrument recordings — they learn to separate pitch/loudness (the "gesture") from timbre (the instrument identity). You provide the gesture (your input), choose the timbre (the target model), and the model re-synthesises it.

Key Features:

5 Target Models — Violin, Flute, Flute2, Trumpet, Tenor_Saxophone
Honest conditioning controls — pitch shift, loudness shift, autotune, quiet attenuation, confidence gating, note-gap deepening
Output level control — match input loudness, peak normalise, or raw model level
Model caching — downloads once, then runs offline
Diagnostic analysis — pitch + intensity CSV for reference

Honest limitations: This is neural revoicing / DDSP timbre transfer, not literal instrument conversion and not guaranteed-natural timbre. Output is stylised. Works best on clean monophonic input. Polyphonic / noisy material is unstable. The model synthesises at 16 kHz; matching the input sample rate upsamples the output but adds no content above ~8 kHz.

Quick start

In Praat, select exactly one Sound object (monophonic works best).
Run script… → DDSPNeuralRevoicing.praat.
Choose a Target_model (Violin, Flute, Flute2, Trumpet, Tenor_Saxophone).
Adjust conditioning controls: Note_detection_threshold_dB, Pitch_shift_octaves, Loudness_shift_dB, Autotune_amount_0to1, Quiet_attenuation_dB, Confidence_gate_0to1, Note_gap_depth_dB.
Set output options: Output_name, Match_input_sample_rate, Output_level, Play_result.
Click OK — Python engine extracts f0/loudness, runs the DDSP model, renders output.
Result appears as a Sound object with your chosen name.

Quick tip: For a clean, natural-sounding violin transfer, use low autotune (0.0–0.1) and moderate confidence gate (0.15). For a more "stylised" effect, try Trumpet or Tenor_Saxophone on vocal input. Note_gap_depth_dB (15–25) helps sustained models (Trumpet, Tenor_Saxophone) stop slurring across short staccato gaps. Enable Keep_temp_files to inspect the diagnostic analysis CSV.

Important: Python dependencies required: tensorflow ddsp crepe gin-config numpy soundfile. The first run downloads the selected model from Google Cloud Storage (requires internet). Subsequent runs use the cached model. The script expects a working Python environment with DDSP installed — the python_exe$ in the script points to a venv path; update it if your setup differs.

5 Target Models

Model	Character	Best for
Violin	Smooth, expressive, sustained	Voice, melodic lines, sustained tones
Flute	Bright, airy, breathy	High-pitched sources, vocal fry, breathy material
Flute2	Brighter, more articulate	Fast passages, articulated lines
Trumpet	Brass, bright, powerful	Melodic material, percussive articulation
Tenor_Saxophone	Warm, rich, expressive	Voice, mid-range material, expressive lines

Model source: These are the standard Magenta DDSP timbre-transfer checkpoints from the public GCS bucket (gs://ddsp/models/timbre_transfer_colab/2021-01-06/). The engine downloads the solo_<model>_ckpt folder (gin config + checkpoint shards) over plain HTTPS — no gs:// filesystem plugin required.

Honest Conditioning Controls

Pitch & loudness

Pitch_shift_octaves — transposes the pitch contour (octaves).
Loudness_shift_dB — shifts the amplitude envelope (dB).
Autotune_amount_0to1 — snaps pitch toward the chromatic grid (0 = off, 1 = fully quantised).

Gating & articulation

Note_detection_threshold_dB — note-on gate, dB below the loudness peak.
Quiet_attenuation_dB — dB of attenuation in note-off regions.
Confidence_gate_0to1 — CREPE-confidence floor; frames below it are silenced (fixes wandering pitch on decaying tails).
Note_gap_depth_dB — deepens inter-note loudness gaps (staccato preservation). Helps sustained models stop slurring across short gaps. Try 15–25 for staccato.

Output

Output_level — Match input loudness (scales RMS to source), Peak normalize (~0.95), or Raw (model's native level).
Match_input_sample_rate — resample the 16 kHz output up to the input rate (adds no content above ~8 kHz).

Confidence gate behaviour: Silences sustained, low-loudness, low-confidence runs (genuine note-off tails) with tapered edges. It does not hard-silence every isolated low-confidence frame mid-note, so the amplitude curve remains smooth and undistorted.

Theory: DDSP Timbre Transfer Pipeline

DDSP Autoencoder architecture

The pretrained model is an autoencoder with an interpretable bottleneck:

Encoder: Extracts f0 (pitch) and loudness (amplitude) from the input.
Decoder / Synthesizer: Uses harmonic oscillators, filtered noise, and a learned timbre embedding to re-synthesise audio from the pitch/loudness controls.
The timbre embedding is fixed to the target instrument's identity during inference.

Pipeline (inference):

Input WAV → mono → 16 kHz
Extract f0 (CREPE) + loudness (A-weighted, DDSP)
Apply adjustments: pitch shift, loudness shift, autotune, quiet, confidence gate, note-gap deepening
Pretrained DDSP Autoencoder(features) → audio
Optional resample to input sample rate, output level normalisation

Why 16 kHz? The DDSP models were trained at 16 kHz. This is the model's bandwidth ceiling (~8 kHz). Resampling up to 44.1/48 kHz only changes the file's container rate — no new high-frequency content is added. The stats file reports both the output_sample_rate and the synthesis_rate (16000) to keep this honest.

Applications

Voice → instrument timbre transfer

Use case: Turn a sung melody into a violin, flute, trumpet, or saxophone line.

Settings: Target_model = desired instrument, Pitch_shift_octaves = 0, Loudness_shift_dB = 0, Confidence_gate = 0.15.

Instrument → instrument cross-synthesis

Use case: Play a melody on one instrument and re-render it with another's timbre.

Settings: Input = piano, Target_model = Violin. The output preserves the piano's pitch and loudness gesture but sounds like a violin.

Stylised / experimental textures

Use case: Push the model into unstable, glitchy territory with extreme settings.

Settings: Pitch_shift_octaves = 1.0 (octave up), Autotune = 1.0 (fully quantised), Note_gap_depth = 30 dB. The output becomes highly stylised and "inhuman".

Workflow: Voice → Trumpet

Source: Sung melody (monophonic).
Settings: Target_model = Trumpet, Confidence_gate = 0.15, Note_gap_depth = 20 dB.
Result: The voice is re-rendered as a trumpet with the same pitch and phrasing — the note-gap depth helps articulate short staccato notes.

Workflow: Guitar → Flute

Source: Guitar arpeggio.
Settings: Target_model = Flute, Output_level = Match input loudness.
Result: The arpeggio sounds like a flute — the pitch contour is preserved, but the timbre is replaced with the flute's breathy character.

Workflow: Vocal → Autotuned tenor sax

Source: Expressive vocal line (with pitch bends).
Settings: Target_model = Tenor_Saxophone, Autotune = 0.3 (gentle pitch correction), Confidence_gate = 0.2.
Result: The vocal is rendered as a saxophone with subtle pitch quantisation — a "vocal sax" hybrid.

Troubleshooting:
• Python venv not found: Update python_exe$ in the Praat script to point to your DDSP environment (e.g., C:/Users/User/praat_ddsp_env/Scripts/python.exe).
• Model download fails: The first run requires internet to download the model from GCS. If the bucket listing is disallowed, the engine falls back to known-filename fetch. Check the Python log for details.
• Output is quiet: Set Output_level = Match input loudness or Peak normalize. DDSP renders at a quiet internal level by default.
• Pitch wanders at note tails: Increase Confidence_gate (0.2–0.3) to silence low-confidence frames at the end of notes.
• Sustained models slur across gaps: Increase Note_gap_depth (15–25 dB) to deepen inter-note loudness gaps and preserve staccato articulation.

Diagnostic analysis

The script writes a CSV analysis file (temp_ddsp_revoice_analysis.csv) containing the input's f0 and intensity over time. This is a diagnostic record of what Praat sees in the input, independent of the DDSP engine's own extraction. Enable Keep_temp_files to inspect it.