Mel-Spectrogram Processing

Frequency-domain representation of audio signals that converts time-domain waveforms into mel-scale spectrograms, essential for speech recognition and audio analysis applications. Forms the preprocessing foundation for most modern voice AI systems.

Core Concepts

Mel Scale Transformation

Mel Scale Properties:

• Perceptually linear frequency scale
• Better aligns with human auditory perception
• Compresses higher frequencies more than lower ones
• Formula: mel = 2595 * log10(1 + hz/700)

Frequency Binning:

• Standard configurations: 32, 64, 80, or 128 mel bins
• Each bin represents a frequency range
• Lower frequencies get more bins (higher resolution)
• Higher frequencies get fewer bins (compression)

Spectrogram Computation

Time-Frequency Analysis:

• Short-Time Fourier Transform (STFT) applied to audio windows
• Hop length determines temporal resolution
• Window size affects frequency resolution
• Overlapping windows for smooth transitions

Processing Pipeline:

1. Windowing: Apply Hann/Hamming window to audio segments
2. FFT: Compute frequency spectrum for each window
3. Mel Filtering: Apply mel-scale filter bank
4. Log Transform: Convert to logarithmic scale (dB)

Implementation Patterns

ONNX Model Integration

Model Input/Output:

Input:  [batch_size, audio_samples] (PCM16 audio)
Output: [time_frames, mel_bins, 1, 1] (mel-spectrogram)

Typical Configurations:

• Audio: 16 kHz mono, 1280 samples per chunk (80ms)
• Output: 32 mel bins, variable time frames
• Context: Minimum 400 samples for stable computation

Real-Time Processing

Streaming Considerations:

• Overlapping audio windows for continuity
• Buffer management for context windows
• Memory-efficient frame extraction
• Padding strategies for startup/shutdown

Buffer Management:

# Conceptual streaming pattern
class MelSpectrogramStreamer:
    def __init__(self):
        self.audio_buffer = []
        self.context_size = 1760  # samples for full context
        
    def process_chunk(self, new_samples):
        self.audio_buffer.extend(new_samples)
        
        # Extract with context
        if len(self.audio_buffer) >= self.context_size:
            window = self.audio_buffer[-self.context_size:]
        else:
            # Pad with zeros on startup
            window = [0.0] * (self.context_size - len(self.audio_buffer)) + self.audio_buffer
        
        return self.compute_mel_spectrogram(window)

Audio Preprocessing Requirements

Sample Rate and Format

Standard Configurations:

• 16 kHz mono: Most common for speech recognition
• 8 kHz: Telephony applications
• 44.1/48 kHz: High-fidelity audio applications

Format Conversion:

• PCM16 → Float32 normalization
• Multi-channel → mono downmixing
• Resampling for rate conversion
• DC offset removal

Frame Processing

Temporal Parameters:

• Frame size: Usually 25ms (400 samples at 16kHz)
• Hop size: Usually 10ms (160 samples at 16kHz)
• Context window: May require 1-2 seconds for stable features
• Overlap: Typically 50-75% between consecutive frames

Feature Engineering

Post-Processing Transformations

Normalization Strategies:

# Common transformation patterns
def normalize_mel_features(mel_frames):
    # Method 1: Per-utterance normalization
    mean = np.mean(mel_frames, axis=0)
    std = np.std(mel_frames, axis=0) + 1e-8
    normalized = (mel_frames - mean) / std
    
    # Method 2: Global statistics (from training)
    # normalized = (mel_frames - global_mean) / global_std
    
    # Method 3: Min-max scaling
    # normalized = (mel_frames - min_val) / (max_val - min_val)
    
    return normalized

Domain-Specific Transformations:

• Delta features (velocity): First-order time derivatives
• Delta-delta features (acceleration): Second-order derivatives
• Mean subtraction for robustness
• Variance normalization across time/frequency

Feature Augmentation

Training-Time Enhancements:

• SpecAugment: Frequency/time masking
• Noise addition for robustness
• Speed perturbation (time stretching)
• Volume augmentation

openWakeWord Integration

Pipeline Architecture

Three-Stage Processing:

1. Mel-Spectrogram: Raw audio → frequency features
2. Embedding: Mel frames