Building ElevenLabs What a $600M voice company is actually made of — and which parts you could buy.

Two sentences from the CEO, one of which is load-bearing

Asked on stage why OpenAI and Anthropic — both openly targeting his category — had not taken the business, Mati Staniszewski gave a two-part answer in about fifteen seconds:

Both claims are testable against the published literature, and they do not survive equally.

The architecture claim was true and is decaying. The field's actual pivot, between roughly 2021 and 2023, was learning to make audio look like text. Neural codecs turned waveforms into sequences of integers, at which point speech synthesis became next-token prediction — which is precisely the problem that scale is good at, and precisely the apparatus the frontier labs already own. Saying “architecture, not scale” is also exactly what you say when the counterparty's advantage is scale and the audience is investors. It is the self-serving half.

The data claim is right, for a reason he does not give. Transcription is not the bottleneck: a fully automated pipeline that separates music, diarises speakers, segments, transcribes, and quality-filters 101,000 hours of in-the-wild audio is open source and free 2407.05361. If Whisper transcripts were the moat, you would not need a thousand people. What those people produce is the set of judgments that ASR structurally cannot emit — and, more importantly, an evaluation that does not otherwise exist. He is describing an eval moat and calling it a data moat.

And there is a third constraint he does not mention, which is the strongest of the three. Human conversational turn-taking has a mode offset between 0 and 200 ms, measured across ten languages on five continents (Stivers et al., PNAS 2009). That is a perceptual constant, not a product decision, and a time-to-first-audio budget in the tens of milliseconds forecloses design choices five layers upstream. Streaming requires causality, so it rules out non-causal architectures and any form of global prosody planning — the model must commit to the delivery of word one before it has seen word twenty. It also puts batching, the main lever on serving cost, in permanent tension with the latency target.

This is the thing his two claims cannot account for, which is why it belongs up front rather than buried in an engineering section. It explains why a lab that matches you on naturalness still cannot take the conversational business. Quality is a training problem and responds to a bigger cluster. Latency is a structural problem, decided at architecture time, and it does not improve when a competitor buys more GPUs. The sections below on latency and on serving economics are both consequences of this one constraint.

The pipeline, and the four jobs that never go away

Every text-to-speech system, from 1980s formant synthesis to whatever shipped this morning, does four things:

1. Text to linguistic features. Normalise the writing system into something pronounceable, then convert to phonemes.
2. Duration and prosody. Decide how long each sound lasts, where pitch rises, where emphasis falls, where the speaker breathes.
3. Acoustic modelling. Produce an intermediate representation of the sound — historically a mel spectrogram, now often a sequence of discrete tokens.
4. Waveform synthesis. Turn that representation into samples a speaker can move air with.

The architectural question is never whether to do these. It is whether they are separate modules with human-legible interfaces between them, or absorbed into one model's hidden states. That single choice determines almost everything downstream: how you debug, what you can expose to a customer as a control, and where errors compound.

The unglamorous part that actually breaks in production

Stage one has no interesting papers and consumes a startling share of real engineering. “Dr. Smith lives on Sunset Dr.” “$1.5B in Q3.” “The 1969 model” versus “1969 units.” “I read the book” versus “I read books.” Homographs, currency, dates, ordinals, abbreviations, units, addresses — each is a per-language, per-locale rule set with a long tail that never ends.

Modern token-LM systems increasingly skip explicit phonemisation and feed characters or subword tokens straight in, learning grapheme-to-phoneme implicitly from data. This works remarkably well and it introduces a genuinely bad failure mode: the system is now wrong in ways you cannot fix. A rule-based front-end that mispronounces your customer's brand name is a one-line lexicon patch. A model that mispronounces it has to be prompted around, retrained, or overridden by a mechanism you had to build anyway.

Five generations, and what each one broke

The lineage is worth walking honestly, because each generation fixed something real and broke something else real, and the breakages explain the shape of today's systems.

The pattern is a pendulum. Cascaded systems are controllable and inspectable, and every interface between modules is a place where errors compound and information is thrown away — a duration predictor cannot recover what the phonemiser lost. End-to-end systems train jointly on the actual objective and sound better, and offer nowhere to intervene when a customer complains.

One durable lesson for anyone evaluating a vendor: if a team tells you their vocoder is the differentiator, they are describing a layer that was solved six years ago.

The pivot: making audio look like text

This is the section that explains why everything changed, and it is the one place where Staniszewski's architecture claim has real content — pointing in the opposite direction from the one he intends.

A mel spectrogram is a continuous-valued matrix. You model it with regression or diffusion. There is no vocabulary, so there is no cross-entropy, no next-token prediction, no perplexity, and none of the machinery, scaling recipes, or infrastructure that the field had spent a decade building for text.

Neural codecs removed that barrier. SoundStream 2107.03312 and EnCodec 2210.13438 pair a convolutional encoder/decoder with residual vector quantisation: quantise the latent against a codebook, take the residual, quantise that against a second codebook, and repeat N times. The first codebook carries the coarse signal; later ones carry progressively finer detail. Because the quantiser layers can be dropped during training, one model serves a range of bitrates. Descript's DAC 2306.06546 improved the reconstruction substantially with multi-band discriminators and better codebook regularisation.

Audio is now a sequence of integers. Everything the field knows about language models applies.

Two kinds of token, because one is not enough

AudioLM 2209.03143 found the structure that most systems still use. Acoustic tokens alone give you beautiful audio that gradually stops saying anything coherent. Semantic tokens alone — taken from an intermediate layer of a self-supervised speech model — give you coherent structure that reconstructs at poor fidelity. So you model both, hierarchically: semantic tokens for long-range structure, then coarse acoustic tokens conditioned on them, then fine acoustic detail.

Why three seconds of reference audio is enough

VALL-E 2301.02111 put it together: treat TTS as conditional language modelling over codec tokens, train on 60,000 hours, and an unexpected property appears. Three seconds of a stranger's voice, supplied as a prompt, is enough to synthesise arbitrary new text in that voice.

This is widely misread as a cloning algorithm. It is in-context learning. The reference audio is a prompt and the model is continuing it. That framing explains all of the behaviour, including the parts that look like bugs: the output inherits the reference's room reverberation, its microphone, its emotional state, and its background noise, because a continuation continues everything, not just identity. It also explains why so little audio suffices — the base model has already learned the manifold of human voices from tens of thousands of hours, and the prompt only has to locate a point on it. Same reason a three-shot prompt works on a text LLM.

Frame rate is the most consequential number in the system

Mimi, the codec inside Moshi 2410.00037, compresses 24 kHz audio to 12.5 Hz at 1.1 kbps using 8 codebooks, streaming, with an 80 ms frame latency. That is 100 tokens per second of speech. A codec running 8 codebooks at 50 Hz produces 400. Sequence length sets compute, compute sets cost and latency, and you fix all of it the day you choose the codec. Ask any team what their frame rate is; the answer tells you whether they have thought about serving at all.

Where the field genuinely disagrees

Having made audio tokenisable, the field split, and has not reconverged.

Voice cloning, and what actually determines similarity

Three mechanisms, commonly presented as a quality ladder. They are better understood as three different products with different serving characteristics.

There is a genuinely counterintuitive result here worth knowing before you pick an embedder. ECAPA-TDNN is among the strongest speaker-recognition models available, and a 2025 study found it does not necessarily improve speaker similarity in zero-shot multi-speaker TTS 2506.20190. The reason is structural: verification embeddings are trained to be invariant to prosody, emotion, and recording channel, because a verification system must recognise you whether you are shouting or whispering. Those are exactly the axes you need to reproduce. The best identity detector is trained to discard the thing you are trying to synthesise.

What actually drives perceived similarity

• Channel match dominates length. A clone from a phone call sounds like a phone call. Users hear this as “the model is bad,” not “my microphone was bad.”
• Coverage beats duration. Thirty seconds spanning varied phonetic and prosodic territory beats five minutes of monotone reading.
• In-distribution or not. Accented English, low-resource languages, children, elderly, and atypical or disordered speech degrade sharply. This is a data problem, not an architecture problem, and no clever conditioning fixes it.
• Prompt leakage — the failure nobody warns you about. Clone from a sad clip and neutral text comes out sad. Clone from a clip with a fan running and the fan comes too. Because in-context cloning is continuation, not extraction, everything in the reference is a conditioning signal, whether or not you wanted it to be.

The data problem, and why a thousand people exist

Start by establishing what is free, because it is a great deal more than most people assume. Emilia-Pipe, the open preprocessing pipeline released with the Emilia dataset 2407.05361, runs six stages end to end and processes an hour of raw audio in minutes:

That pipeline produced 101,000 hours across six languages at 24 kHz, publicly downloadable, and the extended release exceeds 216,000 hours. Libri-Light's 60,000 hours — the corpus VALL-E trained on — has been public since 2019. Raw transcribed audio at scale is commodity. If transcription were the moat, a thousand contractors would be an absurd way to buy it.

So what are the thousand people producing?

Everything in the list below shares a property: an ASR system cannot emit it, because ASR is trained to discard it. Whisper normalises away disfluencies by design. Every quality that makes speech sound alive is precisely what a transcription model is built to throw out.

• Pronunciation ground truth in context. Whisper writes “Nguyen.” It does not tell you how that is said by an Australian speaker in a customer-service register, or which of three readings your customer prefers.
• Prosody and paralinguistics. Where the emphasis lands. Whether an unpunctuated clause is a question. Breath, laughter, hesitation, hedging. Emotion labels with an intensity, applied consistently by trained annotators rather than crowdworkers.
• Alignment correction. Forced aligners give phone-level boundaries cheaply and are systematically wrong on overlapped, noisy, expressive speech — which is exactly the speech you want most.
• Pairwise preference data. A judges B. This is the RLHF analogue for audio, and it is the only mechanism that can optimise the qualities you cannot write a metric for. It is inherently human, inherently ongoing, and it compounds.
• Consent and provenance records. Who, when, under what licence, in which jurisdiction, revocable how. Unglamorous, and increasingly the thing that decides whether you can sell to an enterprise at all.

Where the claim is self-serving

Notice what Staniszewski does not claim: an advantage in hours of audio. He cannot, because Emilia and Libri-Light make hours a commodity. He claims an advantage in labels — which happens to be the one asset class that is entirely unfalsifiable from outside the company. That is a convenient place to locate a moat when you are on stage explaining why your much larger competitors cannot reach you.

It is also probably true. Both things can hold at once, and a CTO should hold them at once.

There is a general test hiding in that, and it is the most portable thing in this article. When you are comparing competing explanations of why something is fast, or good, or defensible, the useful discriminator is not who has a vendor arguing for them — everything has a vendor. It is which claim you can falsify before you commit to it. An open codec, an open checkpoint, and a published eval can each be run by a third party on hardware you already own; a claim about a closed system's training data can only be taken on trust. ElevenLabs has published no architecture, no corpus, and no evaluation methodology, which means every claim about their internals — including both of the ones examined here — is unfalsifiable by construction. That is not an accusation. It is the reason this piece separates published from inferred on every line, and it is worth noticing that unfalsifiability is itself a component of the moat: a competitor cannot copy what a vendor has not described, and cannot disprove it either.

The counter-evidence on the architecture half is harder for him. Frontier labs did arrive: OpenAI's Realtime line collapsed recognition and synthesis into a single speech-to-speech model, Google shipped native audio output in Gemini, and Kyutai — a lab of roughly ten people — demonstrated full-duplex spoken dialogue with a published paper and open weights 2410.00037. Third-party roundups in 2026 place open-weight models at or above ElevenLabs on some preference benchmarks; I have not verified those against primary sources and would not lean on them, but the direction is not in dispute. weakly sourced

Architecture converged on the transformer. That convergence is scale becoming applicable. What has not converged is the product — voice libraries with cleared rights, consent infrastructure, per-vertical workflows, pronunciation lexicons in forty languages, and latency commitments. Which is, notably, what Staniszewski describes three sentences later, when he lists verticalisation and ecosystem as the rest of the answer. The strongest version of his case is the one he gives second.

Latency, where audio differs hardest from text

A text model that takes an extra 300 ms is slightly annoying. A voice agent that takes an extra 300 ms is a different product — it stops passing as a conversation. The spec is set by human perception and cannot be negotiated: across ten languages on five continents, response-timing distributions are unimodal with the mode between 0 and 200 ms (Stivers et al., PNAS 2009).

Here is where that budget actually goes in a cascaded agent. Every figure below is from a published source; the endpointing and LLM ranges are typical values I have not sourced to a specific paper and are marked as such.

Read that bar again. The audio player's buffer is larger than the model. The industry competes publicly on the 75 ms segment and ships a client that adds several times that. If you are evaluating a vendor, model latency is close to irrelevant — measure mouth-to-ear on your own network, from your own client, at p99.

Moshi takes the constraint seriously in a way cascades cannot: 12.5 Hz frames, 80 ms frame latency, a multi-stream architecture that models both speakers simultaneously without an explicit turn boundary, and a reported theoretical latency of 160 ms — 200 ms in practice 2410.00037. That number lands exactly on the human mode. It is not a faster cascade; it removed the turn.

How the latency budget reaches back up the stack

• Streaming forbids lookahead. A causal model must commit to the prosody of word one before seeing word twenty. Phrasing and emphasis genuinely suffer, on identical weights, purely because you streamed. This is a real quality cost of low latency and no vendor advertises it.
• Non-autoregressive models want the whole utterance. Which is hostile to an LLM still generating it. This is the concrete mechanism by which a tokenisation choice becomes a latency ceiling.
• Batching is the cost lever, and the SLO takes it away. Every millisecond you spend collecting a batch is a millisecond of time-to-first-audio. Cost per hour and latency are in direct tension, always.
• Barge-in is the hard part. When the user interrupts, you kill generation, flush the buffer, and then reconcile what was actually heard against what was generated — the LLM's context must reflect the truncated utterance, not the full one. Almost nobody handles this correctly and no offline metric detects it.

Evaluation, and why audio is harder than text

Mean Opinion Score — listeners rate naturalness on a five-point scale — has been the field's primary metric for thirty years. It has four separate problems, and they compound.

1. Not comparable across studies. Researchers use the term inconsistently and under-report methodology, making cross-paper MOS comparisons unreliable (Kirkland et al., SSW 2023, “Stuck in the MOS pit”). Every leaderboard built by collecting MOS numbers out of papers is measuring nothing.
2. Ceiling compression. As systems approach human, differences collapse into rater noise. Le Maguer et al. put it bluntly in Computer Speech & Language: evaluating only overall quality with MOS may be the end of a cul-de-sac.
3. Listener-pool drift. Crowdworker demographics, headphones, and attention all vary. Your score moves when your system did not.
4. Absolute ratings hide relative differences. Listeners judge each sample independently, so subtle differences between similar systems vanish. CMOS and MUSHRA fix this by presenting side by side — at higher cost, and only between two things you already have.

Whether this is fixable is an open argument, and worth knowing which side a team is on. The reformist position is that the instrument is sound and the practice is not: Kirkland et al. found that most authors fail to report scale labels, increments, or participant instructions, and showed experimentally that those very choices moved MOS for some systems — which is an argument for standardised protocols and honest reporting, not for abandonment. The abolitionist position is that no amount of methodological hygiene rescues a single scalar once systems cluster near the ceiling, and that the field needs protocols built for analysing modern synthesis rather than ranking it. Both readings are held by serious people and the field has not settled it.

The practical consequence for a CTO does not depend on who wins. Any team whose quality story reduces to one number is exposed either way, because both camps agree that number cannot carry the weight.

The objective proxies, and precisely how each lies

What cannot be measured at all

• Prosodic appropriateness. Is this the right emphasis for this sentence in this context? There is no reference. Many readings are correct and infinitely many are wrong, and nothing distinguishes them automatically.
• Long-form consistency. Does the voice at minute thirty-nine match minute one? Does chapter seven match chapter one?
• Pronunciation of things outside your test set. By construction.
• Interaction quality. Barge-in handling, turn-taking, backchannelling. No offline metric touches these.

Which brings the argument back to the thousand contractors, with a stronger justification than the one offered on stage: you cannot automate an evaluation you cannot define, and you cannot improve past an evaluation you cannot run. The labelling organisation is the evaluation organisation. That is the actual moat, and it is a moat specifically because it is an operational cost centre that a research lab has no appetite to build.

Serving economics: what an hour of speech costs

Start from published list prices, which are the only hard numbers available. From ElevenLabs' API pricing page: Flash and Turbo at $0.05 per 1,000 characters, Multilingual v2 and v3 at $0.10; conversational agents at $0.080 per minute; Scribe v2 speech-to-text at $0.22 per hour. Their own rule of thumb puts 1,000 characters at roughly one minute of speech.

The striking figure is the last one. Generating a voice lists at roughly fifteen to thirty times the price of recognising one. ASR and TTS are comparable-sized neural networks doing comparable amounts of arithmetic. The gap is not compute. The gap is that Whisper is free and excellent, so recognition is commoditised, and synthesis quality is not. For a CTO this is the single most useful number on the page: you are being charged for the moat, not the FLOPs.

Audio answers the model-hardware-serving question differently

For text, the evidence points hard at the serving stack. MIT FutureTech's Gundlach, Lynch, Mertens and Thompson put algorithmic efficiency progress at around 3× per year 2511.23455 — and essentially every technique that gets counted as “algorithmic” in that accounting is a serving-stack technique rather than a new model: speculative decoding, paged attention, KV-cache compression, sparse attention. The industry's answer to scale points the same way. vLLM's continuous batching improves latency with the model and the silicon both held fixed, and NVIDIA Dynamo and llm-d both respond to scale by disaggregating prefill from decode — a serving topology change, not a new architecture or a new chip. The same diagnosis recurs at four independent levels of the memory hierarchy — FlashAttention on HBM against on-chip SRAM, LMCache across GPU, CPU, disk and remote storage, KV replication across parallelism strategies, and wafer-scale designs that delete the level entirely — which is about as strong as convergent evidence gets: four scales, four remedies, one bottleneck.

Audio inherits the diagnosis and loses the cure. The dominant serving-stack lever is batching, and the latency SLO is precisely what takes batching away from you. Every millisecond spent accumulating a batch is a millisecond of time-to-first-audio, and the latency budget above is set by a perceptual constant you cannot negotiate. The same weights serving a batch-of-64 offline dubbing job and a batch-of-1 interactive agent differ by an order of magnitude in cost per second of audio. So the lever is still the serving stack — but a large part of the headroom that text workloads spend it on is unavailable to anything conversational, and it was foreclosed upstream, on the day someone picked a codec frame rate.

The practical consequence: “cost per hour of generated audio” is a meaningless number unless a latency SLO is attached to it. A vendor quoting one without the other is quoting their batch price for your interactive workload. Ask for both.

What I would build first

A genuine minimum viable path, entirely from open components. Nothing here is trained from scratch, because training a codec or a base model from zero is a research project and not a product.

What it sounds like

Honestly: very good. In 2026, a competent fine-tune of an open model on a few thousand clean hours produces speech that most listeners will not identify as synthetic on a read-aloud sentence in a quiet room. You will get there in weeks and it will be genuinely impressive.

That is the trap. Demo quality is a solved problem, which means demo quality tells you nothing about the team you are evaluating, including your own.

The five gaps to production, each of them months

1. The tail. Autoregressive models hallucinate, repeat, and truncate at a rate of a few percent. At a million requests a day, a 2% failure rate is twenty thousand broken calls. You need a WER-round-trip guard with automatic regeneration, constrained decoding, and a fallback path. Nobody demonstrates this because it is invisible when it works.
2. Pronunciation. Per language, per locale, per customer, with a runtime override. A permanently staffed function.
3. A prosodic control surface. Customers will ask to emphasise a specific word or read a line with a specific emotion. Exposing that requires exactly the prosody labels you decided to defer. The deferral is the reason you cannot ship the feature.
4. Tail latency. p50 is easy. The product is p99, under barge-in, with a cold voice, on a bad network.
5. Provenance and safety, which is now a compliance deadline. EU AI Act Article 50 transparency obligations apply from 2 August 2026, with penalties up to €15M or 3% of worldwide turnover. Tennessee's ELVIS Act has been in force since July 2024 and the federal NO FAKES Act has advanced out of the Senate Judiciary Committee. You need per-generation traceability, a watermark — AudioSeal 2401.17264 is open, robust, and fast enough for real time — a detection classifier, and a consent record attached to every cloned voice.

That last item is worth a moment of clear sight. Staniszewski describes exactly this stack — trace everything generated, moderate at both voice and text level, publish a public detector that works on competitors' models too — and frames it as safety leadership. It is safety leadership. It is also a compliance product and a barrier to entry that gets more expensive for a new entrant every legislative session. Both readings are correct, and a CTO should hold both.

Build, buy, and eight questions

On “architecture, not scale”

Was true. Is decaying. Codec tokenisation converted speech into a next-token problem, which is the thing scale is best at. The evidence that it is decaying is that the frontier labs shipped native speech-to-speech, open weights are competitive on third-party preference evals, and a ten-person lab shipped full-duplex dialogue with a paper. What remains genuinely architecture-specific is streaming causality and frame rate — real constraints, but engineering, not magic.

On the thousand labellers

True, and more important than the reason he gives. Transcription is free. What is not free is the judgment layer — pronunciation in context, prosody, alignment on hard speech, preference data, consent provenance — and above all the fact that those labels constitute an evaluation that does not otherwise exist. It is an eval moat wearing a data moat's costume, and the costume undersells it.

On the claim he did not make

Latency is architecture, and it is the most durable of the three. The 200 ms human turn gap is a perceptual constant. It forecloses design choices five layers upstream, it puts batching and cost in permanent tension, and — unlike model quality — it does not improve when a competitor buys more GPUs.

The build-versus-buy rule

• Buy if voice is a feature. At $3–6 per hour of generated audio and $4.80 per agent-hour, list price is a rounding error against the cost of owning the tail, the pronunciation function, and the compliance stack. Those three, not the model, are what you would actually be signing up to build.
• Build — meaning fine-tune, never pretrain — if you have a narrow domain where the pronunciation long tail is bounded and yours; or volume high enough that list price becomes a real line item; or a data-residency or latency constraint no API satisfies.
• Never build the codec, and never train a base model from scratch, unless the model itself is the product you sell.

Eight questions for a team that says it can build this

1. What is your codec frame rate, and what did picking it cost you in quality? Tests whether they understand that they chose their compute budget on day one.
2. Autoregressive or not? What is your babble and truncation rate per thousand utterances, and how do you measure it? If there is no number, there is no guard, and the tail is unowned.
3. Show me mouth-to-ear p50 and p99, on a real network, with barge-in. Not model latency. Model latency is the segment they optimise and the smallest one in the budget.
4. What is in your evaluation beyond MOS and WER? Who runs it, and how often? The answer describes their ceiling, because you cannot improve past an eval you cannot run.
5. How does a customer fix a mispronounced proper noun, in production, today? “Next training run” means they have no override and will lose their first enterprise account over it.
6. Where did the training audio come from, and can you produce a consent record for any voice on demand? From 2 August 2026 this is a regulatory question in the EU, not a values question.
7. What is your cost per hour of generated audio at your latency SLO, and how much of it is the batching you cannot do? The gap between their batch price and their interactive price is their real unit economics.
8. On a forty-minute audiobook, does the voice at minute thirty-nine match minute one? Prove it. There is no standard metric for this, so the answer reveals whether they have built their own.