Audio Word Similarity & Reordering — User Guide
Auto-segmentation, feature extraction, and reordering by similarity: automatic word detection and reorganization based on formant, pitch, or MFCC similarity with MDS visualization.
What this does
This script implements automatic word segmentation and similarity-based reordering. It detects word/syllable boundaries based on silence, extracts acoustic features (formants, pitch, or MFCCs), computes similarity between segments, and reorders them by similarity or MDS (Multidimensional Scaling) dimension.
Key Features:
- Auto-Segmentation: Automatic word/syllable detection via silence threshold
- Three Similarity Metrics: Formants (vowel quality), Pitch (F0), MFCCs (timbre)
- MDS Visualization: 2D similarity map of all detected words
- Multiple Ordering Methods: Nearest neighbor chain, MDS dimension sort, original order
- Automatic Concatenation: Reordered words stitched with optional silence
- Mono Conversion: Automatic stereo-to-mono conversion if needed
Complete pipeline: (1) Preprocessing: Convert to mono if needed. (2) Segmentation: Use intensity threshold to detect silent intervals, label sounding segments as "words". (3) Feature Extraction: Extract chosen feature (formants F1/F2, pitch mean, or MFCC vector) from each word. (4) Distance Matrix: Compute pairwise distances between all words. (5) MDS: Reduce to 2D for visualization. (6) Reordering: Reorder words by chosen method. (7) Concatenation: Stitch reordered words with optional silence padding. (8) Output: New sound object, MDS plot, info window summary.
Technical Implementation: Segmentation uses Praat's intensity-based silence detection. Formant analysis uses Burg method at word midpoint. Pitch analysis uses mean F0 over word. MFCC analysis extracts 12 coefficients and uses Euclidean distance over mean vectors. MDS uses monotone transformation for 2D embedding. Reordering options: nearest neighbor chain (greedy similarity), MDS dimension 1 sort (linear projection), or original order. Concatenation uses Praat's concatenate function with optional silence insertion.
Quick start
- In Praat, select exactly one Sound object (speech with clear word/syllable boundaries).
- Run script… →
word_similarity_reordering.praat. - Set Segmentation parameters: Silence threshold (25 dB), Minimum silent interval (0.1 s), Minimum sounding interval (0.1 s).
- Choose Similarity metric: Formants (vowel quality), Pitch (F0), or MFCC (timbre).
- Set feature-specific parameters (Max formant, Number of formants, or MFCC coefficients).
- Choose Ordering method: Nearest neighbor chain, MDS Dimension 1, or Original order.
- Set Silence between words (0.1 s default).
- Click OK — script runs, outputs new sound, MDS plot, and info summary.
Quick tip: Start with Formants for vowel-based similarity (e.g., grouping similar vowel sounds). Use Pitch for intonation similarity. Use MFCCs for general timbre similarity. For segmentation: adjust Silence threshold based on recording noise floor. For fast speech: reduce Minimum silent interval to 0.05 s. Use Nearest neighbor chain for smooth transitions between similar words. Use MDS Dimension 1 sort to create gradient from one acoustic extreme to another.
Important considerations: SEGMENTATION ACCURACY — Works best with clear pauses between words/syllables. Continuous speech without pauses may not segment correctly. FEATURE EXTRACTION — Formants may be undefined for unvoiced/consonantal segments. Pitch extraction fails for unvoiced sounds. MFCCs work for all sounds but are sensitive to noise. MINIMUM 2 WORDS — Need at least 2 detected segments for analysis. STEREO HANDLING — Automatically converts to mono by averaging channels. COMPUTATIONAL LOAD — MFCC analysis slower than formants/pitch for many segments.
Auto-Segmentation
Intensity-Based Silence Detection
Algorithm
Praat's intensity threshold method:
Steps:
1. Convert sound to Intensity object (100 Hz smoothing)
2. Threshold = -X dB relative to maximum intensity
3. Label intervals below threshold as "silent"
4. Merge silent intervals shorter than minimum_silent_interval
5. Merge sounding intervals shorter than minimum_sounding_interval
6. Remaining "sounding" intervals = words
Parameters:
Silence_threshold_dB = X (default 25)
Higher = more sensitive (detects quieter as sound)
Lower = stricter (requires louder sound)
Minimum_silent_interval_s = min silent duration (default 0.1)
Shorter silent gaps merged with sounding intervals
Minimum_sounding_interval_s = min word duration (default 0.1)
Shorter sounding intervals discarded as noise
Output:
TextGrid with tier 1: "silent" / "sounding" intervals
"sounding" intervals relabeled as word_1, word_2, ...
Parameter Tuning
For different recording conditions:
| Condition | Silence_threshold_dB | Minimum_silent_interval_s | Minimum_sounding_interval_s |
|---|---|---|---|
| Clean studio recording | 30-35 | 0.05 | 0.05 |
| Noisy environment | 20-25 | 0.15 | 0.15 |
| Fast speech | 25-30 | 0.03-0.05 | 0.05 |
| Slow, deliberate speech | 25 | 0.2 | 0.2 |
| Music/tonal sounds | 15-20 | 0.1 | 0.1 |
Segmentation Output
Example segmentation output:
Found 5 intervals (using -25 dB threshold)
word_1: 0.000-0.452s
word_2: 0.652-1.123s
word_3: 1.323-1.789s
word_4: 1.989-2.456s
word_5: 2.656-3.123s
Interpretation:
Each "word" = continuous sounding segment
May be actual words, syllables, or phonetic chunks
Silent gaps typically 0.2s in this example
Troubleshooting Segmentation
Problem: Too many segments (over-segmentation)
Causes: Threshold too low, minimum intervals too short, noisy recording
Solutions: Increase Silence_threshold_dB, increase Minimum_sounding_interval_s, pre-filter noise
Causes: Threshold too low, minimum intervals too short, noisy recording
Solutions: Increase Silence_threshold_dB, increase Minimum_sounding_interval_s, pre-filter noise
Problem: Too few segments (under-segmentation)
Causes: Threshold too high, minimum intervals too long, continuous speech
Solutions: Decrease Silence_threshold_dB, decrease Minimum_silent_interval_s, speech may lack clear pauses
Causes: Threshold too high, minimum intervals too long, continuous speech
Solutions: Decrease Silence_threshold_dB, decrease Minimum_silent_interval_s, speech may lack clear pauses
Problem: Inconsistent segment lengths
Causes: Varying loudness, background noise fluctuations
Solutions: Normalize audio first, use noise gate preprocessing, adjust threshold dynamically
Causes: Varying loudness, background noise fluctuations
Solutions: Normalize audio first, use noise gate preprocessing, adjust threshold dynamically
Feature Extraction
Method 1: Formants (Vowel Quality)
🎤 Vowel Space Analysis
Extracts: First two formants (F1, F2) at word midpoint
Represents: Vowel quality/timbre (F1 = height, F2 = frontness)
Best for: Vowel-heavy speech, grouping similar vowel sounds
Formant Extraction Details
Praat Burg method:
To Formant (burg): time_step, max_formant, window_length, pre-emphasis
Script uses:
time_step = 0 (all frames)
max_formant = Max_formant_Hz (default 5500)
window_length = 0.025
pre-emphasis = 50 Hz
For each word:
t_mid = (start + end) / 2
F1 = Get value at time: 1, t_mid, "hertz", "Linear"
F2 = Get value at time: 2, t_mid, "hertz", "Linear"
Undefined values set to 0 (unvoiced/consonants)
Distance metric:
Euclidean distance: d = √[(F1_i - F1_j)² + (F2_i - F2_j)²]
Interpretation:
Similar F1/F2 → similar vowel articulation
Distance in Hz → perceptual vowel similarity
Method 2: Pitch (F0)
📈 Fundamental Frequency
Extracts: Mean pitch (F0) over word duration
Represents: Intonation, voice pitch, tonal center
Best for: Tonal languages, pitch-based grouping, intonation patterns
Pitch Extraction Details
Praat pitch tracking:
To Pitch: time_step, pitch_floor, pitch_ceiling
Script uses:
time_step = 0.0 (auto)
pitch_floor = 75 Hz
pitch_ceiling = 600 Hz
For each word:
mean_pitch = Get mean: start, end, "Hertz"
Undefined values set to 0 (unvoiced segments)
Distance metric:
Absolute difference: d = |pitch_i - pitch_j|
Interpretation:
Similar pitch → similar intonation/tonality
Works only for voiced segments
Unvoiced segments get distance 0 to all
Method 3: MFCCs (Timbre/Spectral Shape)
🎚️ Mel-Frequency Cepstral Coefficients
Extracts: 12 MFCC coefficients (mean over word)
Represents: Overall spectral shape, timbre, phonetic quality
Best for: General sound similarity, phonetic grouping, any audio type
MFCC Extraction Details
Praat MFCC computation:
To MFCC: number_of_coefficients, window_length, time_step, f1, f2, f3
Script uses:
number_of_coefficients = 12
window_length = 0.015
time_step = 0.005
f1 = 100 Hz, f2 = 100 Hz, f3 = 0
For each word:
1. Extract word segment
2. Compute MFCC (all frames)
3. Convert to TableOfReal
4. Compute mean of each coefficient (c1..c12)
Distance metric:
Euclidean distance over 12D vector:
d = √[∑(c1_i - c1_j)² + (c2_i - c2_j)² + ... + (c12_i - c12_j)²]
Interpretation:
Comprehensive spectral representation
Works for all sounds (voiced/unvoiced)
Captures timbral differences
Feature Comparison
| Feature | Dimensions | Voiced Only? | Speed | Best For | Limitations |
|---|---|---|---|---|---|
| Formants | 2 (F1, F2) | Mostly | Fast | Vowel similarity, phonetics | Fails on unvoiced, needs good formant tracking |
| Pitch | 1 (F0 mean) | Yes | Fastest | Intonation, tonal grouping | Useless for unvoiced, ignores spectral shape |
| MFCCs | 12 (c1..c12) | No | Slowest | General timbre, any sound | Computationally heavy, less interpretable |
Similarity & MDS
Distance Matrix Computation
Pairwise Distance Calculation
For N words, create N×N symmetric matrix:
Distance matrix D:
D[i,j] = distance(word_i, word_j)
D[i,i] = 0 (self-distance)
D[i,j] = D[j,i] (symmetric)
Stored as TableOfReal object:
Rows = words, Columns = words
Values = distances
Example (Formants, 3 words):
Word1: F1=500, F2=1500
Word2: F1=400, F2=2000
Word3: F1=600, F2=1200
D[1,2] = √[(500-400)² + (1500-2000)²] = √[10000+250000] = √260000 ≈ 510
D[1,3] = √[(500-600)² + (1500-1200)²] = √[10000+90000] = √100000 ≈ 316
D[2,3] = √[(400-600)² + (2000-1200)²] = √[40000+640000] = √680000 ≈ 825
Multidimensional Scaling (MDS)
Dimensionality Reduction
Convert distances to 2D coordinates:
MDS goal:
Find 2D points whose distances approximate original distances
Preserve relative similarities in lower dimension
Praat implementation:
dissimilarity = To Dissimilarity (from TableOfReal)
config = To Configuration (monotone mds): dimensions, criterion, tolerance, max_iterations, random_starts
Script uses:
dimensions = 2
criterion = "Primary approach" (Kruskal's stress-1)
tolerance = 1e-05
max_iterations = 50
random_starts = 1
Output:
Configuration object with coordinates (x,y) for each word
x = MDS dimension 1, y = MDS dimension 2
Visualization:
Plot points in 2D space
Similar words cluster together
Dissimilar words far apart
📐 MDS Interpretation
Dimension 1 (x-axis): Primary acoustic gradient
For formants: often F2 (front-back dimension)
For pitch: low vs high pitch
For MFCCs: first principal component of spectral shape
Dimension 2 (y-axis): Secondary acoustic gradient
For formants: often F1 (high-low dimension)
For pitch: may represent pitch stability/variation
For MFCCs: second principal component
MDS Visualization
Example MDS plot (Formants):
word_1 (500,1500) → MDS: (-0.8, 0.2)
word_2 (400,2000) → MDS: (-1.2, -0.5)
word_3 (600,1200) → MDS: (0.5, 0.8)
word_4 (450,1800) → MDS: (-1.0, -0.3)
word_5 (550,1300) → MDS: (0.2, 0.6)
Interpretation:
Left side (negative x): higher F2 (front vowels)
Right side (positive x): lower F2 (back vowels)
Top (positive y): higher F1 (open vowels)
Bottom (negative y): lower F1 (close vowels)
Clusters show vowel similarity
Gradient shows phonetic continuum
Reordering Methods
Method 1: Nearest Neighbor Chain
🔗 Greedy Similarity Chain
Algorithm: Start with first word, repeatedly add most similar remaining word
Result: Smooth transitions between acoustically similar words
Use: Creating fluid sequences, gradual acoustic changes
Algorithm Details
Steps:
1. Start with word 1 as current
2. Mark word 1 as used
3. For position = 2 to N:
a. Find unused word with minimum distance to current
b. Add that word to chain
c. Mark as used, set as new current
4. Output chain order
Example (distances from above):
Start: word_1
Nearest to word_1: word_3 (distance 316)
Nearest to word_3: word_5 (distance 200)
Nearest to word_5: word_4 (distance 350)
Last: word_2
Order: word_1 → word_3 → word_5 → word_4 → word_2
Properties:
- Greedy (local optimization)
- Creates smooth transitions
- Starting word affects entire chain
- Not globally optimal
Method 2: MDS Dimension 1 Sort
📊 Linear Projection Order
Algorithm: Sort words by their MDS Dimension 1 coordinate
Result: Gradient from one acoustic extreme to another
Use: Creating acoustic continua, systematic variation
Algorithm Details
Steps:
1. Compute MDS coordinates for all words
2. Extract Dimension 1 values (x-coordinates)
3. Sort words ascending by Dimension 1 value
4. Output sorted order
Example MDS Dimension 1 values:
word_1: -0.8
word_2: -1.2
word_3: 0.5
word_4: -1.0
word_5: 0.2
Sorted ascending:
word_2 (-1.2) → word_4 (-1.0) → word_1 (-0.8) → word_5 (0.2) → word_3 (0.5)
Interpretation:
Moves along primary acoustic gradient
From one extreme to other
Shows systematic variation in feature space
Method 3: Original Order
⏯️ No Reordering
Algorithm: Keep original temporal order
Result: Words concatenated as originally spoken
Use: Testing, comparison baseline, simple concatenation
Concatenation with Silence
Final Audio Assembly
Process:
1. Extract each word segment in new order
2. Create silence of duration silence_between_words_s
3. Concatenate: word1 + silence + word2 + silence + ... + wordN
Praat commands:
Extract part: start, end, "rectangular", 1, "no"
Create Sound from formula: "silence", 1, 0, duration, sr, "0"
Concatenate: combines selected sounds
Output:
New sound object: originalname_reordered
Duration = sum(word_durations) + (N-1)×silence_duration
Example: 5 words × 0.5s each, 0.1s silence
Total = 2.5s + 0.4s = 2.9s
Ordering Method Comparison
| Method | Complexity | Result Character | Best For | Starting Point Sensitive? |
|---|---|---|---|---|
| Nearest neighbor | O(N²) | Smooth local transitions | Fluid sequences, gradual change | Yes (first word matters) |
| MDS Dimension 1 | O(N log N) | Systematic global gradient | Acoustic continua, extremes contrast | No (global ordering) |
| Original order | O(1) | Preserves original sequence | Baseline, simple concatenation | N/A |
Applications
Speech/Sound Poetry
Use case: Create sound poetry by reordering words phonetically
Technique: Use Formants similarity, nearest neighbor chain
Example: Record words, reorder by vowel similarity for phonetic patterns
Audio Montage/Composition
Use case: Create musical sequences from spoken words
Technique: Use Pitch similarity for tonal sequences, MFCCs for timbral sequences
Workflow:
- Record diverse words/sounds
- Extract features (pitch for melody, MFCCs for texture)
- Reorder to create rising/falling patterns
- Add silence for rhythmic spacing
Phonetic Analysis
Use case: Analyze vowel space or phonetic similarity
Technique: Use Formants, examine MDS plot
Examples:
- Map vowel production of speaker
- Compare phonetic realizations across contexts
- Identify vowel mergers/contrasts
Language Teaching/Training
Use case: Create graded pronunciation examples
Technique: Record words, reorder by similarity to target pronunciation
Application:
- Start with words similar to learner's L1 vowels
- Gradually move to more distant vowels
- Create continuum from easy to difficult
Sound Design
Use case: Create evolving sound textures
Technique: Use MFCC similarity, MDS dimension sort
Examples:
- Record various environmental sounds
- Reorder by timbral similarity
- Create smooth transitions between sounds
- Use for film/game soundscapes
Practical Workflow Examples
🗣️ Vowel Continuum Creation
Goal: Create sequence from front to back vowels
Settings:
- Similarity metric: Formants
- Max formant: 5500 Hz
- Ordering: MDS Dimension 1 sort
- Silence between: 0.2s
Input: Words: "see", "say", "sat", "saw", "so"
Output: Sequence ordered by F2 (front→back): "see" → "say" → "sat" → "saw" → "so"
🎵 Tonal Word Sequence
Goal: Create rising pitch sequence
Settings:
- Similarity metric: Pitch
- Ordering: MDS Dimension 1 sort
- Silence between: 0.15s
Input: Words spoken at different pitches
Output: Sequence from low to high pitch
Use: Musical composition from spoken words
🔊 Timbre Gradient
Goal: Create smooth timbral transition between sounds
Settings:
- Similarity metric: MFCC (12 coefficients)
- Ordering: Nearest neighbor chain
- Silence between: 0.05s (quick transitions)
Input: Various sound objects (click, whoosh, buzz, ring)
Output: Chain where each sound similar to previous
Use: Sound design transitions, audio mosaicing
Advanced Techniques
Hybrid similarity measures:
- Weighted combination: Combine formant and pitch distances
- Hierarchical: First group by voicing, then by formants
- Custom distance: Modify script to implement Mahalanobis or cosine distance
- Feature selection: Use only specific MFCC coefficients (e.g., c1-c4 for spectral envelope)
Preprocessing for better results:
- Normalize loudness: Scale all segments to same RMS
- Noise reduction: Apply noise reduction before segmentation
- High-pass filter: Remove low-frequency rumble (50 Hz cutoff)
- Voice activity detection: More sophisticated than simple threshold
- Manual correction: Edit TextGrid after auto-segmentation
Troubleshooting Common Issues
Problem: Only 1 segment detected
Causes: Threshold too low, continuous speech, minimum intervals too long
Solutions: Decrease Silence_threshold_dB, check audio has pauses, decrease Minimum_silent_interval_s
Causes: Threshold too low, continuous speech, minimum intervals too long
Solutions: Decrease Silence_threshold_dB, check audio has pauses, decrease Minimum_silent_interval_s
Problem: Formants undefined for many words
Causes: Unvoiced segments, weak voicing, children's speech (higher formants)
Solutions: Increase Max_formant_Hz, try MFCC instead, pre-select voiced segments
Causes: Unvoiced segments, weak voicing, children's speech (higher formants)
Solutions: Increase Max_formant_Hz, try MFCC instead, pre-select voiced segments
Problem: MDS plot shows all points clustered
Causes: Low feature variation, distance matrix has small range
Solutions: Use different similarity metric, record more diverse sounds, check feature extraction
Causes: Low feature variation, distance matrix has small range
Solutions: Use different similarity metric, record more diverse sounds, check feature extraction
Problem: Reordered sequence sounds abrupt/jarring
Causes: Too much dissimilarity between consecutive words, silence too short
Solutions: Use nearest neighbor chain, increase silence_between_words_s, normalize segment loudness
Causes: Too much dissimilarity between consecutive words, silence too short
Solutions: Use nearest neighbor chain, increase silence_between_words_s, normalize segment loudness
Problem: MFCC analysis very slow
Causes: Many long segments, high sampling rate
Solutions:
Causes: Many long segments, high sampling rate
Solutions:
Technical Deep Dive
Distance Metrics Mathematics
Euclidean Distance
For formants and MFCCs:
Given vectors x = (x₁, x₂, ..., xₙ) and y = (y₁, y₂, ..., yₙ)
Euclidean distance: d(x,y) = √[∑(xᵢ - yᵢ)²]
Properties:
- Always non-negative: d(x,y) ≥ 0
- Symmetric: d(x,y) = d(y,x)
- Triangle inequality: d(x,z) ≤ d(x,y) + d(y,z)
- Zero self-distance: d(x,x) = 0
For formants (n=2):
d = √[(F1ᵢ - F1ⱼ)² + (F2ᵢ - F2ⱼ)²]
For MFCCs (n=12):
d = √[∑_{k=1}¹² (cᵢₖ - cⱼₖ)²]
Normalization considerations:
Features on different scales (F1 ~200-1000 Hz, F2 ~800-3000 Hz)
Could normalize or use weighted Euclidean
Current script uses raw Hz (formants) or raw coefficients (MFCCs)
Absolute Difference (L1 Distance)
For pitch:
For scalar values (pitch):
d(x,y) = |x - y|
Properties:
- Simpler than Euclidean
- Less emphasis on large differences
- Works for 1D feature
Alternative could use:
Ratio distance: d = |log(x/y)|
Perceptually more appropriate for pitch
But absolute difference simpler
Pitch in Hz not perceptually linear
Better: use semitones: d = |12 × log₂(x/y)|
Could modify script for perceptual distance
MDS Algorithm Details
Kruskal's Non-metric MDS
Praat's implementation:
Goal: Find configuration X that minimizes stress:
Stress = √[∑(dᵢⱼ - δᵢⱼ)² / ∑dᵢⱼ²]
Where:
dᵢⱼ = Euclidean distance in configuration X
δᵢⱼ = disparity (monotonic transformation of input distances)
Monotone transformation:
Find f such that f(input_distance) ≈ configuration_distance
f must be monotonic (preserve order)
Algorithm steps:
1. Initial random configuration
2. Calculate distances dᵢⱼ in current configuration
3. Find monotonic transformation f to minimize stress
4. Update configuration via gradient descent
5. Repeat until convergence (tolerance 1e-05)
"Primary approach" = Kruskal's stress formula 1
Normalized by sum of squared distances
Computational Complexity
Time Requirements
Let:
N = number of words detected
D = audio duration (seconds)
SR = sampling rate (Hz)
Segmentation:
Intensity computation: O(D × SR)
Silence detection: O(D × 100) (intensity sampled at 100 Hz)
Feature extraction:
Formants: O(N × window_analysis) ≈ O(N × D_word × SR)
Pitch: O(N × D_word × pitch_analysis)
MFCCs: O(N × D_word × SR × 12) (most expensive)
Distance matrix:
O(N² × F) where F = features (2, 1, or 12)
MDS:
O(N² × iterations) ≈ O(50N²) worst case
Total complexity dominated by:
- MFCC extraction if used
- Distance matrix for large N
- MDS for N > 50
Typical performance:
N=10-20: seconds to minutes
N=50+: minutes to longer
Memory Considerations
Object Creation
Key objects in memory:
1. Original sound: D × SR samples
2. Intensity object: D × 100 points
3. TextGrid: N intervals
4. Feature objects: N × (formant/pitch/MFCC)
5. TableOfReal: N × N distances
6. Configuration: N × 2 points
7. Segment sounds: N × D_word samples
8. Final sound: concatenated
Peak memory ~2-3× original sound size
Plus N²/2 distance matrix (for N=50, ~1250 floats)
For large N or long audio:
- Consider downsampling
- Limit N via segmentation parameters
- Clear intermediate objects sooner