I’m not an ML engineer. I build CI pipelines and yell at Bazel. But today I was testing MAI-Transcribe-1 (our inhouse speech-to-text model) and comparing it to my local recording tool, echolab, which uses Qwen3-ASR plus pyannote for speaker diarization. Two things clicked that I want to write down before I forget.
Diarization is not what I thought it was
I assumed diarization was some kind of LLM thing. You feed it audio, it figures out who said what, probably with a prompt somewhere. Nope.
Diarization answers one question: who spoke when. Not what they said. Just which chunks of audio belong to which speaker. And the way it works is pure signal processing:
- Segmentation. Cut the audio into small speech chunks, throw away silence.
- Embedding. For each chunk, extract a voice fingerprint. A vector that captures what that voice sounds like.
- Clustering. Group similar vectors together. Each cluster = one speaker.
No prompts. No attention heads. No chat completions. It’s closer to k-means than it is to GPT.
echolab uses pyannote, a French open-source library that does exactly this pipeline. MAI-Transcribe doesn’t do diarization at all. It transcribes, full stop. If you want “Speaker A said X, Speaker B said Y,” you need to run diarization separately and stitch the results together.
There’s a research frontier where one model does both transcription and diarization end-to-end. But the mainstream approach today is still the pipeline: separate diarization model, separate ASR model, merge at the end.
The four stages of ML training
This came up because I was trying to understand what “fine-tuning” means for a diarization model versus an LLM. Turns out there’s a whole progression, and the education analogy actually holds up.
Pre-training = school. Kindergarten through college. You throw massive amounts of data at the model (text, audio, images, whatever the modality is) and it learns the basic structure of the world. “What does language sound like?” “What patterns exist in speech?” This takes months of compute on thousands of GPUs.
Mid-training (continued pre-training) = grad school. You’ve got a general model, now you specialize it. Feed it more domain-specific data. A speech model might get more medical audio, or more Mandarin. Still unsupervised, still big, but narrower.
Fine-tuning (post-training) = job onboarding. Small amount of labeled data, specific tasks. “Here are 10,000 audio clips where we’ve marked exactly when each speaker starts and stops. Learn to do that.” Hours to days of compute instead of months.
RLHF = apprenticeship with a mentor. A human scores the model’s outputs, and the model adjusts to match human preferences. This is mainly an LLM thing. You need it when the task is generative and subjective (“write a helpful response”). Diarization doesn’t need this because it’s a classification task with a clear right answer.
The key insight: diarization models only need pre-training + fine-tuning. They’re solving “which person does this voice belong to,” not “generate a plausible continuation of this text.” No RLHF, no preference optimization. Just learn what voices sound like, then learn to tell them apart.
It’s a good reminder that “AI” is a very big tent. The thing that makes ChatGPT feel magical (RLHF, instruction following, conversational flow) is a tiny slice of what ML models actually do. Most of the interesting work in speech is closer to traditional signal processing than it is to anything involving a chat prompt.