kNN-TTS: kNN Retrieval for Simple and Effective Zero-Shot Multi-speaker Text-to-Speech

Input

Generated Audio

kNN-TTS Technical Details

kNN-TTS uses self-supervised learning (SSL) features and kNN retrieval to achieve robust zero-shot multi-speaker TTS.

Key Components

1. Feature Extraction: We extract discrete representations from target speaker speech using a pre-trained SSL encoder. We use the 6th layer of WavLM Large.
2. Text-to-SSL: We train a lightweight TTS model to predict the same representations from Text. For simplicity, we train on a single speaker dataset.
3. Retrieval Mechanism: We use kNN to find for each unit in the generated features its closest matches in the target voice unit database
4. Voice Morphing: By linearly interpolating the source and selected target speaker features, we can morph the two voices. The interpolation parameter λ controls the balance between source and target characteristics
5. Vocoder: We use a pre-trained vocoder to convert the converted features to waveform.

Performance

Our simple and efficient model achieves comparable results to sota models while being trained on 100 to 1000× less transcribed data. This framework is therefore particularly well-suited for low-resource domains.

For more details, please refer to our paper (https://arxiv.org/abs/2408.10771).