AI Conductor Mix — v3.0 User Guide

Algorithmic ensemble conductor with sample‑level rendering. Select 2+ Sound objects – the conductor assigns dynamic roles (leader, shadow, resonance, noise fringe, pulse carrier, interruption, sustain bed, contrast voice, memory trace, silence) and states (sparse, balanced, agitated, saturated, suspended, released) to segments of each file. A stereo mix is rendered continuously in Python – click‑free, no segmentation artefacts.

Author: Shai Cohen Affiliation: Department of Music, Bar‑Ilan University, Israel Version: 3.0 (2026) – Python‑side rendering License: MIT License Repo: GitHub

Contents:

What it does Quick start The 10 roles The 6 states Pipeline (5 stages) Sample‑level rendering Parameters Visualization FAQ / troubleshooting

What this does

AI Conductor Mix treats a selection of 2+ sounds as an ensemble. A descriptor‑driven algorithm analyses each file’s internal structure (RMS, ZCR, spectral centroid, bandwidth, flux, roughness, pitch salience, HNR, attack slope, temporal stability) and segments them into time regions. For every segment, the conductor assigns a role (its function in the mix) and a state (the current global tension/density). A dynamic gain, pan position, and layer group are computed. The stereo mix is then rendered continuously in Python – by building smooth gain/pan envelopes and applying them to the unsegmented source files – eliminating all boundary clicks and segmentation artefacts.

Conductor intelligence: Role assignment is based on percentile‑ranked descriptors, style‑weighted, and refined by temporal rules: leader alternation, masking avoidance, nonlinear interruptions, and a diversity cap. The result is a musically coherent, dynamic mix where each file contributes according to its acoustic character.

Quick start

In Praat, select 2 or more Sound objects (order matters for default pan assignment).
Run script… → AI_Conductor_Mix.praat.
Choose Segment_mode: Fixed frames, Onset‑based, or Hybrid.
Select a Conductor_style:
- Neutral (balanced – uses custom params below), Dramatic (high contrast), Minimal (sparse), Dense (layered).
Adjust Memory_weight (how much past energy influences decisions) and Tension_sensitivity.
Click OK. Praat exports WAVs and descriptors, runs Python conductor, and imports the rendered stereo mix as ConductedMix.

Quick tip: Start with Dramatic style and Onset-based segmentation. Listen to how the AI brings out different files as leader, pulse carrier, or interruption.

Important: Python dependencies: numpy, soundfile. librosa is optional for better onset detection. The conductor is purely descriptor‑driven – no neural network weights are used. The mix is rendered in Python as a 16‑bit PCM WAV.

The 10 roles

Each segment receives a role – its functional identity in the mix. Roles are assigned by scoring each segment against descriptor‑based formulas, then applying temporal rules, style weighting, and a diversity cap.

Role	Tag	Acoustic profile	Base gain	Pan offset
leader	foreground	High energy, bright, stable	0.90	+0.3
shadow	background	Low energy, stable, complementary	0.45	+1.0
resonance	texture	High HNR, stable, low noise	0.60	+0.8
noise_fringe	edge	Noisy, low pitch salience	0.35	+1.2
pulse_carrier	rhythmic	Strong attack, moderate energy	0.70	+0.6
interruption	surprise	Sudden high energy after quiet	0.88	+0.1
sustain_bed	drone	Stable, low flux, low energy	0.40	+0.9
contrast_voice	counterpoint	Bright, pitched, moderate noise	0.75	+1.1
memory_trace	echo	Low energy, stable, low flux	0.30	+0.7
silence	rest	Very low energy – may be omitted	0.00	–

Final gain is modulated by prominence (energy‑ and tension‑derived), polyphony (gain reduced when many segments overlap), and layer group.

The 6 states (global tension curve)

The conductor maintains a global tension curve shaped by the chosen style. Tension at a given moment influences role selection, density, and state transitions. Each segment receives a state label based on tension and its own energy/stability.

State	Tension range	Typical behaviour
sparse	very low	Few active layers, large gaps, quiet.
balanced	low–medium	Regular activity, moderate density.
agitated	medium–high	High energy, bright, fast changes.
saturated	high	Dense layering, many voices, near‑peak gain.
suspended	low, stable	Sustained tones, low flux, held tension.
released	falling	Decay, sparse, dissolving.

Pipeline — five stages

Stage 1 – Export & descriptors (Praat) – export each selected sound to WAV, extract per‑file macro descriptors (pitch, intensity, HNR, centroid).
Stage 2 – Python conductor – load audio, segment (fixed/onset/hybrid), extract per‑segment descriptors (10‑dim), compute scores, assign roles/states/gains/pan, apply temporal rules, diversity cap.
Stage 3 – Praat polling – waits for sentinel file.
Stage 4 – Sample‑level rendering (Python) – build smooth gain/pan envelopes per file using FFT convolution, multiply original streams, accumulate to stereo.
Stage 5 – Visualisation (Praat) – waveforms, role timeline, spectrogram, summary.

Segment descriptors (10 dims)

rms – root‑mean‑square energy
zcr – zero‑crossing rate
centroid – spectral centroid (Hz)
bandwidth – spectral spread
flux – spectral flux (positive changes)
roughness – high‑frequency energy ratio
pitch_salience – autocorrelation peak prominence
hnr_proxy – cepstral harmonic‑to‑noise estimate
attack_slope – energy rise in first 20%
temporal_stability – 1 – normalised std of short‑term energy

Sample‑level rendering engine (v3.0)

Continuous envelope method

For each source file, build a gain envelope (array of length = file samples). For every segment belonging to this file, set gain = segment's gain over its exact sample range. Use np.maximum to keep the highest gain where segments overlap (no dips).
Smooth the gain envelope with a Hanning kernel (150 ms window) via FFT convolution. This replaces hard segment boundaries with smooth, click‑free transitions.
Build pan envelopes for left and right channels similarly, using the segment's pan value (converted to equal‑power L/R gains).
Multiply the original (unsegmented) stereo audio by the smoothed gain and pan envelopes, and accumulate into the output buffer.
Trim silence at the end, normalise peak to 0.95, and write 16‑bit PCM WAV.

Result: Zero clicks, zero amplitude dips – the mix sounds as if it were performed live.

The renderer also performs polyphony‑aware gain scaling: when the number of overlapping segments exceeds 2, the gain of foreground roles is reduced proportionally to avoid saturation. A diagnostic check for amplitude dips at segment boundaries confirms smoothness.

Parameters & defaults

Segmentation

Parameter	Default	Description
Segment_mode	Onset-based	Fixed frames, onset‑based, or hybrid (onset + fixed sub‑division).
Frame_size_ms	500 ms	Frame length for fixed/hybrid.
Frame_hop_ms	250 ms	Hop between frames.
Min_onset_gap_ms	50 ms	Minimum gap between onset boundaries.

Conductor behaviour

Parameter	Default	Description
Conductor_style	Neutral	Neutral, Dramatic, Minimal, Dense – presets for memory/tension and role weights.
Memory_weight	0.4	How much past energy influences current decisions (0–1).
Allow_silence	yes	If unchecked, silence role is never assigned.
Nonlinear_reactions	yes	Enable sudden interruption when high energy follows quiet region.
Tension_sensitivity	0.6	Multiplier for style‑derived tension curve.

Output

Parameter	Default	Description
Draw_visualization	yes	Show waveforms, role timeline, spectrogram, and summary in Praat picture.
Export_mix_plan	no	Save the CSV plan (with segment boundaries) to plugin folder for inspection.

Visualization (Praat picture)

When Draw_visualization = 1, the script draws:

Input waveforms (up to 4 files, shaded grey).
Conducted mix waveform (teal).
Role timeline – a coloured bar for each file showing which role is active over time (leader = red, shadow = grey, resonance = blue, etc.). This is read directly from the plan CSV.
Mix spectrogram (0–5 kHz).
Stats box with segment count, duration, style, memory, tension.
A compact legend for the role colours.

Tip: The role timeline gives an immediate visual summary of the conductor's decisions. You can see at a glance which file is leading, which is providing sustain, and when interruptions occur.

FAQ / troubleshooting

“Python not found” or missing packages

Install: pip install numpy soundfile (librosa optional). On Windows, the script uses py.

Mix is silent or very quiet

Check the Info window – “Plan: X events”. If X is 0, the plan may have been empty. Increase segmentation density (smaller frames, shorter onset gap). Also ensure Allow_silence is not unchecked unless you want silence segments omitted.

I still hear clicks at segment boundaries

The v3.0 continuous envelope renderer is designed to eliminate clicks. If you hear any, it may be a bug – please report it with the debug log. The renderer includes a diagnostic “dip” check; if dips are detected, they are printed in the log.

“Timed out waiting for conductor”

The Python process may have crashed. Run the command shown in the Info window manually in a terminal to see errors. Common issues: missing librosa (falls back to energy‑based onset, but may be slow for large files).

Plan CSV format (v3.0)

The first line is #max_dur=... (canvas duration). Then tab‑separated columns: time, file_index, file_name, seg_index, src_start, src_dur, gain, role, layer_group, entry_time, exit_time, transform, state, density, priority, pan. The exact sample positions (src_start_samp, src_end_samp, src_sr) are used by the renderer but not included in the CSV for Praat.