Gemini TTS Voice Consistency Benchmark: A system evaluation based on 5,400 API calls 

From business pain points, evaluation methods, key findings to production deployment, this article systematically breaks down Gemini TTS’s consistent performance across 30 voices, 3 models, and 6 languages.

Abstract

This experiment centers around a highly engineering-oriented yet often underestimated issue by product teams: when TTS is genuinely deployed in long-form text, audiobooks, digital human broadcasting, variable template synthesis, and multilingual content production, can the model stably maintain the same person’s voice identity? To answer this question, this article builds an automated evaluation pipeline based on TitaNet Speaker Embedding to systematically measure 30 preset voices, 3 model versions, and 6 languages of Gemini TTS, executing a total of 540 experimental groups and 5,400 speech generations.

The results show that Gemini 2.5 Flash significantly outperforms Gemini 2.5 Pro in voice consistency; the overall stability of Spanish, Brazilian Portuguese, and Russian is notably stronger than that of US English; Autonoe and Vindemiatrix are the most stable global voices, while voices like Gacrux and Achernar pose severe lower-bound risks in certain weak combinations. These conclusions not only help teams make more accurate selections but, more importantly, prove that voice consistency can be completely automated, standardized, and integrated into the quality governance system of modern TTS infrastructure.

TL;DR

If you only look at the conclusions, here are the six most memorable points from this experiment: First, 2.5 Flash is currently the most stable primary model in Gemini TTS; Second, 3.1 Flash is a viable second choice, but with slightly larger fluctuations; Third, 2.5 Pro is not suitable as a default baseline for high-consistency audio production; Fourth, Spanish is the most stable language across the entire experiment, while US English is the most obvious weak link; Fifth, Autonoe and Vindemiatrix are the global voices most suitable for production whitelists; Sixth, speaker embedding solutions like TitaNet are highly suitable for automated Benchmarks, without relying on large-scale manual listening tests.

1. Why do this

If today is just about making a ten-second Demo voice, voice consistency may not be the core issue. But once entering real business scenarios, the situation changes rapidly. 

For instance, a long audiobook requires the same narrator to speak continuously for tens of minutes; a digital human system needs to maintain the same virtual character’s identity across different paragraphs and knowledge points; customer service broadcasting and transaction notification systems will continuously insert variable content such as names, amounts, locations, and order numbers. 

In these scenarios, even if the model’s pronunciation is very natural, as long as the same voice sounds like a different person in the next sentence, the user experience will immediately collapse.

This is also why voice consistency and naturalness are two different dimensions. Naturalness answers whether a sentence sounds human, while consistency determines whether the same voice still sounds like the same person throughout an entire production workflow. 

As AI-powered video and speech generation become mainstream, platforms such as VMEG increasingly rely on consistent speech synthesis to support long-form narration, multilingual localization, and production-grade voice generation.

The direct goal of this experiment is to transform this originally subjective, hearing-biased issue into a metric that can be automatically measured at scale and continuously reused in engineering systems. 

Only in this way can voice governance no longer rely on the empirical judgment of sounds okay, but become a truly regressible, monitorable, and decision-making infrastructure capability.

2. How to evaluate

This experiment covers all 30 preset voices, 3 model versions (Gemini 2.5 Pro, Gemini 2.5 Flash, Gemini 3.1 Flash), and 6 languages (Hindi, English/US, Russian, Japanese, Portuguese/BR, Spanish) of Gemini. 

For each voice × model × language combination, 10 sentences are uniformly generated, forming a total of 540 experimental groups and 5,400 TTS API calls.

To define consistency clearly enough, the evaluation adopts a fixed intra-group comparison strategy: the 1st sentence of each group acts as the Reference, and the subsequent 9 sentences act as Candidates. The implication here is not to compare whether different voices are similar, but only to ask one question: Under different text content, does the same voice still maintain the same speaker identity?

From an engineering perspective, this design is very suitable for large-scale Benchmarks. It does not require humans to listen one by one, nor does it rely on subjective scoring, but can directly run through all combinations in parallel, and uniformly use speaker embedding to calculate the similarity between the candidate audio and the reference audio. 

The higher the similarity, the more the model can lock the voice of the same person across different content and sentences.

Figure 1: Benchmark Pipeline. It is recommended to place this figure after the “How to evaluate” section in the published version to help readers establish an overall understanding first.

3. Why TitaNet

In this type of evaluation, what truly matters is not how to calculate a distance, but whether what is calculated is indeed the speaker’s identity. If simple spectral distance, energy differences, or traditional acoustic statistical features are used, the model might mix text content, pause rhythm, local emotions, and even recording amplitude variations into the similarity, causing the metric to be impure.

TitaNet’s value lies exactly here. It is not an ordinary audio encoder, but a neural network architecture specifically designed for Speaker Representation / Speaker Verification. It captures local acoustic patterns through 1D Depth-wise separable convolutions, then combines with Squeeze-and-Excitation modules to model global context, and finally compresses variable-length speech into a fixed-length speaker embedding through statistical pooling. In other words, it focuses on who is speaking, rather than what was spoken.

For automated Benchmarks, this characteristic is particularly important. Because the experiment does not care whether the prosody of each sentence is perfectly identical, nor whether the content expression is exactly the same, but whether the speaker’s timbre and identity features are stably retained under cross-content conditions. TitaNet precisely maps this problem into a numerical, parallelizable, and comparable engineering task.

Figure 2: TitaNet schematic diagram. Recommended to appear grouped with the methodology section to emphasize “why this metric is credible.”

4. Experiment overview

Among the 540 experimental groups, the number of successful groups was 504, and the number of failed groups was 36; in the corresponding 5,400 TTS calls, there were 5,036 successful calls. The overall average similarity was 0.6804, with the highest score reaching 0.8150 and the lowest score being 0.3651. This result itself already illustrates a core fact: Gemini TTS is not simply overall good or overall bad, but has very obvious structural stratification.

Looking further at the distribution, there are 238 groups in Tier A (≥0.70), 193 groups in Tier B (≥0.60), 62 groups in Tier C (≥0.50), 9 groups in Tier D (≥0.40), and 2 groups in Tier F (<0.40). In other words, a large number of combinations have actually reached a usable or even excellent level, but there are also genuinely a few severely unstable combinations. The real engineering problem is not whether there are high-scoring combinations, but whether those combinations that can sporadically collapse can be identified and avoided.

5. Finding #1: Why flash surpasses pro in consistency

The finding most worth emphasizing is that Gemini 2.5 Flash not only outperforms 2.5 Pro in voice consistency, but the advantage is very clear. The overall average score of 2.5 Flash is 0.7260, reaching a Tier A rating, with a standard deviation of only 0.0194; the average score of 3.1 Flash is 0.6874; whereas 2.5 Pro is only 0.6278. This result is counterintuitive because many teams default to understanding “Pro” as a version that is stronger in all dimensions.

However, from a TTS engineering perspective, this intuition is unreliable. A larger, more complex model does not naturally mean it is stronger at the task of stably maintaining the same speaker identity. On the contrary, more complex generation paths, longer contextual dependencies, and greater expressive freedom might all cause the model to gradually deviate from the initial voice condition during long-text, multi-sentence continuous generation.

The advantage of 2.5 Flash is likely not just higher on average, but smaller fluctuations. This is critical for production environments. Because what a team truly needs is not to occasionally hit extremely high scores, but to maintain stable mid-to-high scores across different languages, content, and batch tasks. On this point, 2.5 Flash has demonstrated a very clear baseline value.

6. Finding #2: Language stability is not evenly distributed, but markedly stratified

The differences brought by the language dimension are far larger than many anticipate. Overall, Spanish ranks first at 0.7304, Brazilian Portuguese ranks second at 0.7107, and Russian follows closely at 0.7104; Hindi and Japanese are in the middle tier, while US English clearly ranks at the bottom with 0.6025.

This result means that in the current capability structure of Gemini TTS, there is no so-called unified quality surface across different languages. For the same model and the same voice, after changing the language environment, its speaker consistency performance may change significantly. Especially for English scenarios, many voices under 2.5 Pro drop into a range that is clearly unsuitable for long-text production.

From a business perspective, this finding is very important. Many systems default to English as the primary production language, even treating English results as representative of global capabilities. But this experiment shows that English is not the most stable language for Gemini TTS right now; rather, it’s a high-risk area requiring stricter governance.

7. Finding #3: What truly matters is not the highest-scoring voice, but the global stable voice

Looking at the overall leaderboard, Autonoe ranks first at 0.7131, and Vindemiatrix ranks second at 0.7075; they are also the only two Tier A voices globally. The reason they are important is not because they are extremely brilliant in a single language, but because they are relatively balanced across models, languages, and text content, with no obvious weak points.

The danger of many voices is that they might not look bad on the leaderboard, but when drilling down into language breakdowns or model slices, very obvious weaknesses are exposed. In contrast, Autonoe and Vindemiatrix are more like hexagon warriors that engineering teams can safely include in the default whitelist.

Second-tier voices like Leda, Orus, and Zephyr belong to another category of assets: they can surge to near global optimums in certain strong languages, even producing the ceiling scores of the experiment, but they also show significant pullbacks in weak languages or on certain models. Therefore, these voices are more suitable as conditional, strategic choices, rather than default voices universally open to the public.

8. Finding #4: Why look at the lower bound, and not just the average

For production environments, average scores can sometimes mislead decision-making. Even if a voice’s average score is not low, as long as its lower bound is extremely poor, it is enough to cause severe incidents in long-tail traffic. Gacrux and Achernar are typical examples. They are not bad in all scenarios, and can even get seemingly decent scores in some combinations; the problem is that once they enter a weak combination, their scores crash straight through the bottom line.

Achernar scored only 0.3651 under Gemini 2.5 Pro + English (US), the lowest score in the field; Gacrux also only scored 0.3699 under the Gemini 3.1 Flash + Japanese combination. The reason this type of voice is dangerous is not that it is unexceptional on average, but that it is very hard to predict when it will suddenly break. For a real user-facing system, this kind of sporadic collapse is absolutely unacceptable.

Therefore, if a team only looks at the overall average, it is easy to misjudge these voices as “barely usable”; but as long as the worst combination, worst language, and worst model slices are examined, it becomes clear they are fundamentally unsuitable as official production assets.

9. Finding #5: Why the best combinations are concentrated in the spanish + Flash system

Looking at the Top 15 optimal combinations, Spanish almost dominates the list. The highest score overall is Leda + Gemini 3.1 Flash + Spanish, reaching 0.8150; in addition, combinations like Puck + 3.1 Flash + Spanish, Vindemiatrix + 2.5 Flash + Spanish, and Autonoe + 2.5 Flash + Russian also enter the optimal range.

This indicates that what Gemini TTS is currently best suited for in high-consistency production is not an abstract best model, but a relatively clear combination system: strong language + Flash series models + validated high-quality voices. Only by putting these three conditions together can a team truly obtain a production baseline that can be scaled and reused.

Because of this, this article hopes readers take away not a sentence like Model X is the best, but a configuration mindset: TTS quality is not a single-point attribute, but the combined result of model, voice, language, and text conditions. A truly mature system must govern by combinations, rather than making decisions based on single-model promotional pages.

10. Why does this happen: Several explanations Closer to engineering reality

The first explanation is the attenuation of speaker conditions in autoregressive generation. As the model generates longer content, it may gradually deviate from the initial voice conditioning, reverting to a sort of average speaker state in the training data. This causes the semantics to remain correct and pronunciation to remain natural, but the timbre has drifted.

The second explanation is differences in corpus alignment quality. The excellent performance of Spanish, Portuguese, and Russian might imply that the training samples for these languages are more regular in terms of cross-text consistency for the same speaker; although English is rich in resources, its diverse sources, varied dialects, and more scattered speaking styles make it more prone to identity drift during generation.

The third explanation is bias in model optimization goals. The Flash series might employ more conservative strategies for fast, stable generation, which is therefore more conducive to speaker feature convergence; the Pro series, on the other hand, might allocate more capability budget to expressive freedom, context modeling, or complex condition response, so it is not necessarily better at the task of strictly locking the voice.

11. How to apply it in engineering

These engineering practices become especially valuable in multilingual AI dubbing workflows, where the same speaker identity must remain consistent across different languages, localized scripts, and long-form narration. Instead of relying solely on subjective listening, a benchmark-driven routing strategy helps deliver more stable dubbing quality at scale.

To truly transform these conclusions into production value, the most important thing is not to make a leaderboard, but to embed them into specific system governance rules.

First, establish Smart Routing. Do not throw all requests to the same model, but route them based on locale and use case. US English defaults to Gemini 2.5 Flash; Spanish, Portuguese, and Russian can be shunted between 2.5 Flash and 3.1 Flash based on latency and cost; when long text or multi-paragraph continuous generation is involved, prohibit automatically selecting 2.5 Pro.

Second, establish a Voice Whitelist. The default whitelist should prioritize including Autonoe and Vindemiatrix; an extended whitelist can add Leda, Orus, and Zephyr, but must be accompanied by language and model constraints; at the same time, high-risk voices like Gacrux and Achernar should be explicitly blocked at the system level, rather than leaving the caller to bear the risk.

Third, establish Prompt-Level Pinning. For complex text with proper nouns, polyphones, long paragraphs, and mixed emotion tags, a “global slow + local pause” control strategy is recommended. For example, insert [short pause] at key positions, and require the model to read at a more stable, restrained pace. This can reduce the space for free drift in long contexts.

12. Future roadmap

The significance of this experiment goes beyond writing an article. The longer-term goal is to make voice consistency evaluation a part of production infrastructure.

A direct direction is to move the TitaNet similarity evaluation forward into the generation pipeline to build a real-time quality gatekeeping mechanism. For example, in an asynchronous generation system, perform embedding validation by sampling key paragraphs; when the similarity falls below a threshold, automatically trigger a retry, switch the model, or switch to a more stable whitelist voice.

Another direction is to expand the same evaluation system to other vendors such as Azure TTS, ElevenLabs, and Minimax, forming a unified TTS Router decoupled from specific vendors. By then, the system can stop making choices based on promotional claims or subjective impressions, and instead dynamically select the most stable provider under different languages and scenarios based on real evaluation data.

13. Conclusion

The most important value of this experiment lies not in proving that a certain model is absolutely the best, but in transforming an originally highly subjective issue that is difficult to discuss at scale into a quality metric that can be automatically measured, continuously regressed, and integrated into engineering governance.

When voice consistency is quantified, many originally vague empirical judgments begin to become clear: which models are suitable as baselines, which languages need stronger constraints, which voices can be safely launched, and which combinations must be avoided in advance. For a TTS system truly aimed at production, this kind of quantifiable, automatable, and operable capability is often more important than a single stunning Demo.

In this sense, TitaNet is not just a model used for doing an experiment; it actually fills a long-missing capability layer for modern TTS infrastructure: making the issue of “whether this voice is still the same person” something that can, for the first time, be seriously incorporated into system design just like latency, cost, and error rate.

14. References

1. Koluguri, N. R., Park, T., & Ginsburg, B. TitaNet: Neural Model for speaker representation with 1D Depth-wise separable convolutions and global context.
2. NVIDIA NeMo Toolkit / TitaNet Large Model documentation.
3. Gemini TTS Voice Consistency Experiment Report (Original data and statistical results of this experiment).