Hidden Markov Model Timbre Sequencer — User Guide

A 5D diagonal-Gaussian Hidden Markov Model for timbre-state resequencing: k-means initialization, Viterbi hard-EM training, stochastic state generation, Gaussian or uniform within-state frame selection, and fixed-hop normalized Hann overlap-add synthesis.

Author: Shai Cohen Affiliation: Department of Music, Bar-Ilan University, Israel Version: 2.1 (2026) License: MIT License Repo: https://github.com/ShaiCohen-ops/Praat-plugin_AudioTools

Contents:

What this does Quick start HMM Theory Timbre Features Algorithm Details Parameters Guide Applications Complete Workflow Troubleshooting

What this does

This script implements a first-order Hidden Markov Model over learned timbre states. Frames from the selected Sound are represented by five normalized observations, clustered by k-means, and then refined by repeated log-space Viterbi decoding plus re-estimation of diagonal-Gaussian emissions and transition probabilities. New state paths are sampled from the learned chain, and source frames belonging to each visited state are selected for resynthesis.

HMM Components:

Hidden States: timbre classes initialized by k-means
Observations: 5D vectors — intensity, log₂ pitch, voiced fraction, spectral centroid, spectral balance
Emission Model: diagonal Gaussian per state
Transition Model: first-order state-to-state probabilities learned from the decoded path
Initial Distribution: proportional to occupancy across active states
Decoding: log-space Viterbi
Training: Viterbi hard-EM repeated until the path stops changing or Max_HMM_iterations is reached
Generation: sample states → choose source frames by Gaussian-observation matching or uniform within-state sampling

Key Features:

5D diagonal-Gaussian HMM
Viterbi hard-EM training to convergence
Empty-state pruning — unused states cannot be entered during generation
Gaussian or Uniform frame selection
Exact shared time base — one HMM state = Frame_hop_ms in both training and synthesis
Fixed-hop normalized Hann OLA — no free crossfade control
Three output-length modes — match input, target duration, fixed frame count
Random_seed for reproducible k-means, generation, and frame selection
Synthetic stereo — shared state path, independent within-state frame realization per channel
Suite-standard visualization of decoded path, generated path, 5D features, transitions, emissions, waveforms, and summary

What “true HMM” means here: the model has hidden states, explicit observation distributions, transition probabilities, an initial-state distribution, Viterbi decoding, and iterative hard-EM re-estimation. The training is Viterbi / hard EM, not Baum-Welch soft EM.

Technical implementation: (1) convert source to mono; (2) extract 5D frame observations; (3) min-max normalize each feature to [0,1]; (4) initialize K states with k-means; (5) estimate diagonal-Gaussian emissions and transition probabilities; (6) run Viterbi hard-EM until convergence or iteration limit; (7) prune empty states from the chain; (8) sample a generated state path; (9) select source frames inside each state by Gaussian-observation matching or uniform sampling; (10) reconstruct at exactly Frame_hop_ms using normalized Hann OLA; (11) trim/pad to the requested duration; (12) optionally create synthetic stereo by independent frame choices within the same generated state sequence.

Quick start

Select exactly one Sound object in Praat.
Run HMM_Timbre_Sequencing.praat.
Choose a preset or Custom.
For Custom, set Frame_size_ms, Frame_hop_ms, Number_of_states_K, and the two iteration limits.
Choose Frame_selection:
- Gaussian: draw a synthetic 5D observation from the selected state's Gaussian, then use the nearest real frame inside that state
- Uniform: choose any frame belonging to that state with equal probability
Choose an Output_mode:
- Match input duration
- Target duration (seconds)
- Fixed number of frames
Set Random_seed if you need reproducible output.
Choose mono or synthetic stereo, visualization, Info reporting, and playback.

Quick tip: The default Output_mode is Target duration (seconds) with an 8.0 s target. The preset's Output_length_frames value matters only when you explicitly select Fixed number of frames.

Important: The source is converted to mono before analysis and resynthesis. Stereo_output does not preserve source stereo. It creates a new stereo image by selecting independent source frames for left and right while both channels follow the same generated HMM state sequence.

HMM Theory

Model Structure

λ = (A, B, π)

A: learned first-order transition probabilities between states
B: diagonal-Gaussian emission model in the normalized 5D observation space
π: initial-state distribution derived from state occupancy, over active states only

Diagonal Gaussian Emission

For state s and normalized observation x: log b_s(x) ∝ -0.5 * Σ_d ((x_d - μ_s,d) / σ_s,d)^2 - Σ_d ln(σ_s,d) Dimensions are treated independently within each state (diagonal covariance). Regularization: σ = sqrt(variance + 0.02^2) hard floor = 0.01

Viterbi Hard-EM Training

INITIALIZATION: k-means labels estimate emissions and transitions REPEAT up to Max_HMM_iterations: 1. Compute log emission likelihood for every frame/state. 2. Run log-space Viterbi. 3. Backtrack best hidden-state path. 4. Count assignment changes. 5. Re-estimate emissions and transitions from the decoded path. 6. Stop when the path no longer changes.

This is hard EM: one best state assignment per frame is used at each re-estimation step. It is not Baum-Welch, which would retain soft posterior state probabilities.

Transition Learning and Empty States

For active target states: A[i,j] = (count(i→j) + 0.01) / rowSum For empty target states: A[i,j] = 0 If a row has no usable outgoing counts: distribute uniformly over active states

States with zero occupancy are structurally unreachable during generation. The initial state is also sampled only from active states, in proportion to their occupancy.

Generation

1. Draw initial state from active-state occupancy. 2. For each next position: sample next state from current transition row. 3. For each generated state: Gaussian mode: draw a 5D observation from the state's Gaussian choose nearest source frame inside that state Uniform mode: choose any source frame inside that state equally

Timbre Features

The HMM observation is a 5-dimensional feature vector. Each dimension is min-max normalized to [0,1] across the source frames.

1. Intensity

Measurement: Praat Intensity object; mean energy over the analysis frame.

Role: differentiates quieter and louder regions without pretending to be a raw waveform RMS formula.

2. Log₂ Pitch

Measurement: mean log2(F0) over voiced Pitch frames that fall inside the analysis frame.

Role: octave relationships are represented consistently across frequency.

Unvoiced handling: if a frame has no voiced pitch samples, the pitch dimension is parked at the global mean voiced log-pitch (or log₂200 if the whole source is unvoiced). Voicing itself is represented separately.

3. Voiced Fraction

Measurement: proportion of Pitch frames inside the analysis frame that contain valid F0.

Range: 0–1 before normalization.

Role: separates “is pitched?” from “how high is the pitch?”.

4. Spectral Centroid

Measurement: centre of gravity of a Spectrum computed from a Hann-windowed analysis copy of the frame.

Role: broad brightness descriptor.

5. Spectral Balance

Measurement:

balance = ln((energy[1000..5000 Hz] + ε) / (energy[0..1000 Hz] + ε))

This replaces the old and misleading “spectral slope” label. It is a log high/low band-energy ratio, not a regression slope.

Normalization

for each feature dimension d: min_d = minimum over all frames max_d = maximum over all frames range_d = max_d - min_d normalized_d = (raw_d - min_d) / range_d If range is effectively zero: range is replaced by 1

The current script uses min-max normalization, not z-score normalization.

Algorithm Details

Step 1: Frame Extraction

Analysis frames begin every Frame_hop_ms. Number of frames:

floor((duration - frame_size) / frame_hop) + 1

The HMM time unit is the hop itself.

Step 2: Feature Extraction

Global Intensity and Pitch objects are queried in real time coordinates. Spectral centroid and balance are measured from a Hann-windowed copy of each frame. The raw audio segment used later for synthesis remains unwindowed until the OLA stage.

Step 3: K-means Initialization

Centroids are seeded from distinct frames where possible. Frames are assigned by Euclidean distance in normalized 5D space, centroids are re-estimated, and the loop stops when no assignments change or Max_kmeans_iterations is reached.

Step 4: Hard-EM HMM Training

After initial emission and transition estimation, Viterbi decoding and re-estimation repeat until the path stabilizes or Max_HMM_iterations is exhausted.

Step 5: Generate State Sequence

The first state is drawn from occupancy across active states. Later states are sampled from the learned transition matrix. Empty states are not reachable.

Step 6: Choose a Source Frame for Each State

Gaussian: sample a synthetic 5D observation from the selected state's Gaussian and choose the nearest real source frame belonging to that state.

Uniform: choose one of that state's real source frames uniformly.

Step 7: Fixed-Hop Hann OLA

For generated frame i: source segment: raw frame from original mono source render window: one Hann window output position: (i - 1) * Frame_hop_s accumulate: outBuf += windowed frame envBuf += Hann window after all frames: outBuf /= max(envBuf, 0.15 * peak(envBuf))

The overlap is therefore:

overlap = Frame_size - Frame_hop

There is no independent Crossfade_ms control because changing the synthesis advance would change the temporal scale of the learned state transitions.

Step 8: Exact Output Length

For duration-based modes, the buffer is trimmed or padded to the exact requested duration, followed by a short boundary fade. In Fixed number of frames mode:

duration = N * hop + (frame_size - hop)

Step 9: Stereo

With Stereo_output enabled, the generated state path is shared. Left and right channels independently call the frame-selection procedure inside each state, are rendered separately by the same Hann OLA process, and are then combined to stereo.

Parameters Guide

Main Form

Parameter	Default	Meaning
Preset	Custom	Custom / Fine Grain / Coarse Grain / Textural / Rhythmic / Experimental
Frame_size_ms	80	Analysis and resynthesis grain length; minimum 10 ms
Frame_hop_ms	40	HMM state time unit; if larger than frame size, reset to half the frame
Number_of_states_K	8	Initial number of timbre states
Max_kmeans_iterations	50	K-means iteration cap
Max_HMM_iterations	10	Viterbi hard-EM iteration cap
Frame_selection	Gaussian	Gaussian observation → nearest frame in state, or Uniform within state
Output_mode	Target duration	Match input / Target duration / Fixed number of frames
Target_duration_s	8.0	Used only by Target duration mode; non-positive value falls back to input duration
Output_length_frames	200	Used only by Fixed number of frames mode
Random_seed	0	0 = unpredictable; positive integer = reproducible run
Stereo_output	On	Synthetic stereo from independent within-state frame choices
Draw_visualization	On	Draw v2.1 suite-standard HMM page
Show_info	On	Print detailed final model summary
Play_result	On	Play final output

Presets

Preset	Frame	Hop	K	Fixed-frame value
Fine Grain	80 ms	40 ms	12	400
Coarse Grain	100 ms	50 ms	5	80
Textural	80 ms	40 ms	16	600
Rhythmic	100 ms	50 ms	8	200
Experimental	64 ms	32 ms	24	1000

Preset scope: Presets override only Frame_size_ms, Frame_hop_ms, Number_of_states_K, and Output_length_frames. They do not change Output_mode, Target_duration_s, iteration limits, Frame_selection, Random_seed, stereo, visualization, Info, or playback settings.

Applications

Algorithmic Timbre Resequencing

Generate new temporal arrangements that follow transition statistics learned from the selected source's timbre-state path.

Texture and Gesture Variation

Use Fine Grain/Textural for denser state vocabularies or Coarse Grain/Rhythmic for broader state categories. Gaussian frame selection tends to choose source examples consistent with a sampled state observation; Uniform intentionally ignores within-state Gaussian typicality.

Rhythmic / Temporal Reorganization

Because one HMM state corresponds to exactly one Frame_hop interval in both training and rendering, learned transition timing is no longer stretched by an unrelated synthesis crossfade.

Synthetic Stereo

The shared state sequence keeps left and right channels in the same timbre-state syntax while independent within-state frame selection can create decorrelated source detail.

Model Inspection and Education

The v2.1 visualization makes the script useful for teaching HMM components: state path, observation trajectories, transition probabilities, Gaussian emissions, stochastic generation, and audio reconstruction.

Scope of the current script: It is a single-source offline resequencer. It is not a real-time HMM engine, instrument classifier, genre classifier, cover-song detector, or general model-import/export system. Blending matrices from different sources or using the model for classification would require external modification.

Complete Workflow

Recommended Workflow

Select one Sound with enough timbral variation for the desired K.
Choose a preset or Custom frame/hop/K values.
Keep Gaussian frame selection for state-conditioned stochastic matching; compare with Uniform for a rougher within-state texture.
Choose the output-length mode explicitly; do not assume a preset's Output_length_frames controls duration unless Fixed number of frames is selected.
Use a positive Random_seed if you need repeatable clustering and generation.
Inspect the visualization before interpreting the result: empty states may have been pruned, and hard-EM may converge to fewer active states than the requested K.

v2.1 Visualization

Decoded Input States — final Viterbi path over time.
Generated States — sampled HMM state path over output time.
Observation Trajectories — all five normalized dimensions: intensity, log₂ pitch, voiced fraction, centroid, balance.
Feature legend — separate visual key for the five dimensions.
Learned Transition Matrix — darker cells = higher transition probability.
Diagonal-Gaussian Emissions — mean ± std for all five features in each state.
Input / Output Waveform — source and HMM result on a shared amplitude scale.
Summary strip — requested/active K, k-means iterations, hard-EM convergence, frame/hop/overlap, frame-selection mode, seed, generated state count, stereo mode, duration, peak, and normalized Hann OLA.

Troubleshooting

“Input too short” or fewer frames than states:
The source must be at least 0.1 s, provide at least four analysis frames, and contain at least as many frames as the requested K. Reduce K and/or use a shorter frame/hop.

Several states are inactive:
This is not automatically an error. Hard-EM may leave some states empty. They are then removed structurally from the transition graph and cannot be generated.

Output duration differs from what you expected:
Check Output_mode. Match input and Target duration are trimmed/padded to an exact time. Fixed number of frames uses N × hop + (frame_size - hop).

Stereo does not preserve the source image:
Correct. The source is converted to mono. Synthetic stereo uses the same generated state sequence for both channels but chooses source frames independently within each state.

Output is too repetitive:
Fewer active states, a strongly diagonal transition matrix, or a homogeneous source can all cause repetition. Increasing K is useful only if the material actually supports additional stable states.

Clicks / rhythmic level modulation:
The current renderer already uses normalized Hann OLA at the learned state hop. There is no Crossfade_ms parameter to increase. If artifacts remain, inspect the source material, very short frames, or abrupt frame content rather than trying to change an independent crossfade.

Model limitations:

First-order state transitions only.
Five fixed observation dimensions.
Diagonal Gaussian emissions; feature covariance within a state is not modeled.
Viterbi hard-EM rather than soft Baum-Welch training.
Single-source offline analysis/resynthesis.