AI assistance and author disclosure: AI helped draft and edit this article. I work on Velokey, an AI API platform. I verified the primary sources and reviewed the final text.
I used to think multilingual LLM evaluation was mostly a localization pipeline.
Translate the prompt. Run the same test. Back-translate the answer. Check whether the meaning survived.
Anthropic's latest research makes that workflow look incomplete.
The company analyzed 309,815 anonymized Claude.ai conversations across three models and the 20 most common languages on its platform. The result suggests that language is not only an interface choice. It can be associated with a different distribution of response tendencies.
That makes language a runtime variable.
And if it is a runtime variable, it belongs in the eval key.
TL;DR
Anthropic found structured average differences in Claude's expressed values across models and languages.
Translation quality does not establish behavior parity.
Evaluate each important
model × language × workloadcombination.Separate task quality, behavior fit, and operational cost.
Treat the research as a test generator, not a routing policy.
The Research Signal in Plain English
Anthropic did not try to prove that Claude has beliefs or a fixed personality. It studied values expressed in responses: observable tendencies such as caution, warmth, rigor, honesty, brevity, and results orientation.
The researchers started with 3,307 values from earlier work, grouped similar ones into 339 higher-level values, and compressed the largest patterns into four axes:
| Axis | What It Contrasts |
|---|---|
| Deference vs. Caution | Accommodating the user vs. guarding against risk |
| Warmth vs. Rigor | Encouragement and care vs. accuracy and precision |
| Depth vs. Brevity | Explaining nuance vs. doing only what was requested |
| Candor vs. Execution | Foregrounding uncertainty vs. producing a polished result |
The sample covered Sonnet 4.6, Opus 4.6, and Opus 4.7. It used roughly 5,000 conversations for each model-language pair and focused on subjective tasks.
The researchers also controlled for task, topic, and values expressed by the user. That matters because a code review and an emotional-support conversation naturally invite different behavior.
One number should stay visible throughout the analysis: the four axes explain 15% of measured value variation.
This is a structured signal, not a complete model of behavior.
The Model Version Already Changes the Interaction
Sonnet 4.6 leaned toward deference, warmth, and brevity. Anthropic associated it with affirming a user's work, mirroring tone, using humor, and offering comfort without judgment.
Opus 4.6 leaned toward rigor, deference, and brevity. It was more likely to stay within scope and get to the point.
Opus 4.7 showed the strongest model-level shifts toward caution and depth. It was more likely to challenge a false assumption, flag an unprompted risk, explain its reasoning, critique the user's work, or acknowledge a limitation.
That is a regression-testing problem.
If you replace one model with another behind the same endpoint, you may change how the system disagrees, how much it explains, and when it asks for evidence. An eval that checks only factual accuracy or task completion can miss the drift.
Language Adds a Second Behavior Dimension
Language moved the same four axes.
Hindi showed the strongest warmth shift. English leaned toward caution, rigor, depth, and candor. Russian leaned toward rigor and brevity. Indonesian showed the strongest execution shift. Dutch leaned toward candor. Chinese stayed closer to the average but still shifted modestly toward caution, rigor, and depth.
The wrong takeaway is that one language or culture is inherently warmer, stricter, or more honest.
The study does not prove that.
Anthropic does not yet know whether the differences come from training-data quantity, data composition, fine-tuning, conversational norms represented in the corpus, or several factors at once.
The useful takeaway is narrower:
A semantically equivalent task can be associated with a different behavior profile when the language changes.
Why a Translated Benchmark Is Not Enough
Consider a support assistant handling the same refund dispute in English, Arabic, and Chinese.
All three versions can preserve the facts. The responses may still differ in how much reassurance they provide, how strongly they challenge the user's premise, whether they request evidence, and how quickly they propose an action.
For a casual assistant, that may be acceptable.
For a regulated or policy-controlled workflow, it may be a consistency failure.
Back-translation helps answer: “Did the meaning survive?”
It does not fully answer:
Did the policy survive?
Did the intended degree of caution survive?
Did the response become too verbose or too terse?
Did refusal behavior change?
Did the assistant create another clarification turn?
Would a local reviewer trust the answer?
Those questions require native-language evaluation.
Put Language in the Eval Record
The smallest useful key is:
model × language × workload
An illustrative record could look like this:
# Example evaluation metadata, not a Claude API configuration
eval_case:
id: refund-policy-conflicting-evidence
model: claude-opus-4-7
language: ar
workload: customer_support
prompt_version: support-v12-ar
grader_version: support-rubric-v4-ar
metrics:
- task_success
- policy_compliance
- tone_fit
- clarification_turns
- output_tokens
- latency_ms
- retry_count
- escalation_required
The format is not important.
The dimensions are.
Without language and grader version in the record, a global dashboard can hide local regressions.
Separate Three Kinds of “Good”
One composite quality score is rarely enough.
I would keep three layers separate.
1. Task Quality
correct decision
required facts preserved
correct tool selected
required schema satisfied
policy applied consistently
2. Behavior Fit
appropriate warmth and rigor
useful caution rather than reflexive hesitation
enough depth without avoidable verbosity
uncertainty disclosed when necessary
refusals neither too loose nor too broad
3. Operational Performance
input and output tokens
latency
clarification turns
tool calls
retries
fallback rate
human escalation
cost per acceptable completion
A response can pass one layer and fail another.
A polite answer can apply the wrong policy. A correct answer can be too verbose for a high-volume workflow. A cautious answer can be slower while preventing a costly action.
Build Native Test Cases, Not Literal Translations
Start with the real workload distribution in each market.
The test set should include:
common user phrasing
local date, currency, and identity formats
market-specific policy edge cases
false assumptions
missing evidence
emotional context
ambiguous requests
failed tool calls
cases where a concise answer is preferable
Then use a language-specific rubric.
An English grader may reward directness, politeness, or evidence patterns that do not map cleanly to the target language. It may miss whether the answer sounds patronizing, evasive, overly confident, or unnatural.
For expensive failures, pair automated grading with bilingual human review.
Measure the Cost of the Whole Task
Anthropic did not measure API cost, latency, retry rate, or conversion in this study.
Those outcomes are production hypotheses to test.
Still, the axes point directly at behaviors that can move the bill.
More depth can mean more output tokens.
More caution can create another clarification turn.
More rigor can request evidence before acting.
More deference can reduce friction while failing to challenge a bad premise.
Some of those behaviors cost more.
Some prevent much more expensive mistakes.
The metric that matters is cost per acceptable completed task in each language, not only the provider's token price.
Route Only After You Measure
Do not convert Anthropic's charts directly into model-routing rules.
Try smaller interventions first:
Localize the system prompt instead of translating it literally.
Add target-language examples.
State the desired level of caution, depth, and evidence explicitly.
Improve the local-language grader.
Test representative edge cases again.
If a model-language pair still misses a quality, cost, or reliability threshold, routing becomes a reasonable response.
You might use a concise model for low-risk triage, a stronger model for evidence-heavy analysis, and a fallback after policy or tool failure.
The route should follow the data.
It should not replace the eval.
A unified API layer can make controlled model comparisons and fallbacks easier to run, but it cannot replace language-specific evaluation.
Monitor Language After Launch
The evaluation does not end at deployment.
Segment production telemetry by language, model version, prompt version, and workload. Watch:
accepted completion rate
average output length
clarification turns
policy violations
human edit distance
escalation rate
retry and fallback frequency
cost per completed task
Then rerun the matrix after model, prompt, retrieval, tool, or policy changes.
A stable global average can hide a language-specific regression.
The Honest Limits
This research should change test plans, not create stereotypes.
The data came from Claude.ai, not controlled API requests.
The findings are observational, not causal.
The averages do not predict every response.
The four axes explain 15% of measured variation.
The results do not automatically generalize to other providers.
Your production system adds system prompts, retrieval, tools, output constraints, sampling settings, and its own user distribution. Those variables can reduce, amplify, or reshape the patterns.
Treat the study as a source of test cases.
A Launch Checklist
[ ] Define the acceptable behavior range for every critical workload.
[ ] Build native-language examples and edge cases.
[ ] Test each important model-language-workload combination.
[ ] Grade task success and behavior fit separately.
[ ] Track tokens, latency, retries, tool calls, and escalation by language.
[ ] Use local-language graders and bilingual review where mistakes are expensive.
[ ] Set routing and fallback thresholds from measured results.
[ ] Re-run the matrix after every meaningful system change.
[ ] Monitor production outcomes by language instead of relying on one global average.
Translation can preserve the words.
Only evaluation can tell you whether it preserved the product.
If you run multilingual LLM evals, which metric drifts first in your experience: tone, refusal behavior, verbosity, task success, or cost?
Sources
About the author
- Building VeloKey with the team. Global AI API access for developers and teams. Sharing model workflows, product notes,and launch updates.
Comments
Join the discussionNo comments yet. Be the first to comment.