Would you send it?

Every few weeks a new model tops the leaderboards. For teams that send outreach, those leaderboards might not always answer the right question. They rank models on math, code, science, and general writing. An outreach team needs to know something else, something a bit more subtle. Can a model turn research about a stranger into an email a salesperson would send with their own name on it? And would the stranger want to answer?

A good cold email has to be brief without reading as mechanical. It has to use the research to show why this person, and why now, without reciting everything that was found. It has to sound like one person writing to another, and stay true to what is known about both sides. It has to know what day it is. A funding round from last spring can't open the email as if it was announced this morning.

The established benchmarks each cover a piece of this, but we didn't find one built for this task. CRMArena-Pro tests AI agents on business tasks in a CRM. IFEval and IFBench test whether a model follows instructions. EQ-Bench and LitBench look at creative writing. SDR-Bench comes closest, since it is about sales outreach. It asks a model to work out which arguments won a real deal, and scores how many of them the model recovers. That is a test of what a pitch says. A cold email has to get two things right, what it says and how it reads to the person who opens it. The second is the part we still wanted a way to measure.

twain-bench scores both. AI judges rate every email on eight criteria, from whether its facts hold up and how well it uses the research to whether it sounds like a person and whether the recipient would reply.

#The setup

• Every model gets the same brief, the research a salesperson works from. That is today's date, the sender's company, the customer segment the prospect belongs to, and everything known about the prospect.
• No writing instructions. The prompt asks for a cold email that is ready to send, and says nothing about how to write it. We measure what the models do when nobody tells them how to write.
• 1,000 emails per model, to ten prospects for each of 100 companies.
• Nothing changes between models. The dataset, the prompt and the date stay fixed, so every new model is tested on the same 1,000 briefs. As new models come out, we run them and add them to the leaderboard.

Why 1,000 emails A benchmark needs one fixed set of briefs, with the research on every prospect done on the same day, so a model tested next year gets exactly what today's models got. 1,000 emails are enough for the results to hold. With a different set of 1,000 prospects, a model's average on any criterion would very likely stay within about a point of where it is, and most gaps between models are larger than that. More emails would only sharpen the closest comparisons. Ten companies in each of ten industries also let us compare industries.

Data from 1 October 2026 · 12 models · 12,000 emails

#What we measure

AI judges score every email on eight criteria, each from 0 to 100. Each one stands on its own, and there is no overall score. Which criteria matter most depends on who is sending and to whom, so which model to use is your call. The scores are there to help you make this call.

• Personalization

  Whether the email is built on the detail about the recipient that makes the offer matter to them now.

  0100

  Near 100

  Built on the most relevant, most timely detail, used as the reason to write now.

  Near 0

  Nothing in it is about this recipient. It could go to anyone in the role.

• Creativity

  Whether the email takes a fresh angle that fits the recipient.

  0100

  Near 100

  An angle the recipient wouldn't expect, which fits their situation and the offer.

  Near 0

  The stock template, or a gimmick that doesn't fit.

• Voice

  Whether the email sounds like one particular person wrote it.

  0100

  Near 100

  A person comes through, with uneven rhythm, an aside or a hedge.

  Near 0

  Smooth and correct, and it could have been written by anyone or anything.

• Flow

  Whether the email reads as one line of thought, with every link between ideas visible.

  0100

  Near 100

  One line of thought, and every link between ideas is visible.

  Near 0

  A stack of facts in email shape. The sentences could be reordered without loss.

• Concision

  Whether the email says what it needs to say, and nothing more.

  0100

  Near 100

  Every sentence does a job the email needs.

  Near 0

  An information dump, with several facts of the same kind stacked.

• AI tells

  Whether the email is free of the telltale signs of text written by AI.

  0100

  Near 100

  Nothing a reader would flag as machine-written.

  Near 0

  Tells throughout. It reads as unedited AI output.

• Reply

  Whether the recipient would reply. A judge plays the recipient and reads the email with only what the recipient knows about themselves.

  0100

  Near 100

  The recipient would reply today, because it gives them something they want now.

  Near 0

  The recipient would delete it or feel annoyed.

• Hallucination

  Whether the email's facts about the sender and the recipient match the information in the brief. Scored as the share of its claims that are not invented or misattributed. Premises the email assumes about the recipient are not counted.

  0100

  Near 100

  Every claim checks out against the brief.

  Near 0

  Most claims are invented or attached to the wrong case.

#Each model's writing habits

Before any judging, we compute a set of statistics directly from the text of the emails, such as length, punctuation, formatting and phrase repetition. These statistics describe each model's writing and reveal its idiosyncrasies in cold outreach. They are descriptive only and do not contribute to any score.

Length, readability and formatting

Counted from the text, so no judge is involved. Averages per email, or the share of emails for the columns in percent. Click a column to sort.

Endings, placeholders and repetition

The first four columns are shares of emails. The last five measure how much a model repeats itself across its 1,000 emails together.

#Most models have a stock ask

A stock ask is the templated call to action salespeople close with out of habit, such as "open to a quick call?" or "worth 15 minutes?", in the same words whoever the email is for. In seven of the twelve models, more than three quarters of the emails contain a phrase the model uses in at least a tenth of all its emails, and almost all of those phrases are the ask. GPT-6 Sol writes "a 15-minute conversation" in 41% of its emails and Sonnet 5.5 "a 20-minute call" in 40%. Gemini 3.8 Flash is the most templated, with "open to a" in 85% of its emails and "to a brief" in 76%. Gemini 4 Argon uses the same ask, "open to a" in 78% of its emails. Grok 4.7 varies the most, with a template phrase in only a third of its emails, and GPT-6 Astra, GPT-6.1 Sol and Opus 5.5 in about half. The openings are the opposite. No model opens more than 5% of its emails the same way, and emails to different people at the same company share only 3% to 7% of their phrases. Even if the research behind each email changes, the ask mostly stays the same.

#Each model has a signature

The three GPT models write short and lean, around 100 to 110 words with the fewest claims per email. Sonnet 5.5 writes almost three times as much, a median of 274 words, and formats three quarters of its emails like documents, with bullets or bold. Fable 5 and Kimi K3 use about four em dashes per email, the punctuation readers most often spot as AI. Gemini 3.8 Flash writes the hardest sentences to read, 25 words each on average at a reading grade of 17. Gemini 4 Argon writes the longest sentences, 26 words on average at grade 16. Opus 5.5 writes the easiest, at grade 10, and almost never uses a dash.

#Following the brief is a skill

The prompt asks the model not to sign the email, since the sender adds their own signature. Almost every model complies, and only 2% of Kimi K3's emails end with a "[Your name]" placeholder. Many still end on a bare sign-off such as "Best,", which the prompt allows. That is 70% of Sonnet 5.5's emails, 46% of Kimi K3's and 36% of Fable 5's. Sonnet 5.5 also asks to be redirected to someone else in more than a third of its emails, seven times as often as GPT-6 Sol. Gemini 3.8 Flash follows the brief to the letter, with a bare sign-off in only 2 of its 1,000 emails, no placeholder and almost no giving an option of referrals after a CTA.

#The results

With the setup and the emails in view, let's look at how the twelve models compare.

No model leads on every criterion. Fable 5.1 leads on personalization, creativity, voice and reply. GPT-6 Sol leads on concision and AI tells. Gemini 3.8 Flash is the cheapest at $0.011 an email.

Which of these matters most depends on your outreach. A team that writes to a few hundred carefully chosen prospects may care most about personalization and voice. A team that sends at volume may care most about cost and AI tells. Whether you would send it is your call, and the results below are there to help you make this call.

#Best for

The strongest model on each thing a cold email has to get right.

• Finds the right hook
  Fable 5.1
  83 on personalization
• Sounds like a person
  Fable 5.1
  69 on voice
• Says it in fewer words
  GPT-6 Sol
  82 on concision
• Hardest to spot as AI
  GPT-6 Sol
  56 on ai tells
• Most likely to get a reply
  Fable 5.1
  31 on reply
• Sticks to the brief
  GPT-6 Sol
  100 on hallucination
• Cheapest
  Gemini 3.8 Flash
  $0.011 an email

#Results by criterion

Each card ranks the twelve models on one criterion. The live leaderboard has all the scores in one table with filters, and gains a row with each new model.

Scores from 0 to 100, averaged over each model's 1,000 emails and the three judges. There is no overall score. Each criterion stands on its own. Hover a bar for the model's rank.

The models score closest to each other on hallucination, flow, and reply. All twelve score 97 to 100 on hallucination, so on its own it won't mean much when choosing a model. They are also within 11 points of each other on flow and 7 on reply. Given the same research, they all get almost every fact right and build a coherent email, and the judge playing the prospect would answer few of them. Reply scores sit between 24 and 31, around the point where the rubric places a competent, researched email that the reader reads and leaves. The gaps are in the writing itself. Concision spans 56 points, voice 49, AI tells 30 and creativity 28.

Two profiles stand out. Fable 5.1 leads on personalization, creativity, voice and reply, with Opus 5.5 second on each. The GPT models lead on tight, accurate writing. GPT-6 Sol leads on concision and AI tells, GPT-6 Astra is narrowly ahead on flow, and all three make almost no unsupported claims. Grok 4.7 sits between the two profiles, third on voice and fourth on creativity. Gemini 4 Argon is third on personalization but last on AI tells and eleventh on voice and concision. Gemini 3.8 Flash and DeepSeek V4 Pro score lowest on personalization, creativity and reply, and Gemini 3.8 Flash lowest on voice, at 20 against 29 to 69 for the others.

The price spread is wide. At list price, Gemini 3.8 Flash costs $0.011 per email and Fable 5.1 $0.150, fourteen times as much. The extra money mostly buys voice and creativity. On concision the GPT models lead, and GPT-6 Sol and GPT-6.1 Sol do it at about $0.015. GPT-6 Astra costs five times as much as GPT-6.1 Sol and gains at most 3 points on any criterion. Grok 4.7 costs $0.035, about half of Opus 5.5, and comes third on voice. Opus 5.5 costs less than half as much as Fable 5.1 and stays within 8 points of it on every criterion, and scores higher on hallucination, concision and AI tells. Fable 5 costs almost as much as Fable 5.1 and scores lower on every criterion but hallucination. DeepSeek V4 Pro costs $0.019 and scores near Gemini 3.8 Flash at the bottom on most criteria. Gemini 3.8 Flash is the cheapest model and scores below every GPT model on every criterion.

Prices are the list price of the tokens each model used, without the providers' caching discounts. How much caching saves depends on the provider and the order of the calls, so it would blur the comparison.

Reasoning effort Every model runs at medium reasoning effort. We set it explicitly and the same for all of them, so no result depends on a provider's default. Models can write differently at a higher or lower setting. Those settings are not part of this release.

#Results by industry

The ten industries are within seven points of each other on every criterion. Logistics has the lowest personalization score (72), marketing agencies and professional services the highest (79 and 78). On every criterion the choice of model matters more than the industry the email is for. The two gaps come closest on reply, where the models differ by 7 points and the industries by 5.

#What drives the scores

The scores alone don't say why a model scores the way it does. Comparing them with what we can measure in the emails (length, punctuation, formatting, the facts each one states) shows where the differences come from.

#Length costs concision, and doesn't buy voice

The twelve models have different default lengths, and they stay close to them. Almost all of GPT-6 Sol's emails are between 81 and 119 words. Sonnet 5.5's shortest emails are longer than GPT-6 Sol's longest ones.

Within each model, length barely moves the voice score. Opus 5.5 and Fable 5.1 stay within 2 points at every length, and Sonnet 5.5 and Fable 5 score lower above 200 words. Voice comes from how a model writes, whatever the length. At 100 to 149 words, Fable 5.1 scores 68, Opus 5.5 60, GPT-6 Sol 42 and Gemini 3.8 Flash 21.

Concision works differently. Longer emails score lower on it in every model, as we would expect. Sonnet 5.5 loses 33 points between 100 to 149 words and 200+, and Grok 4.7 loses 28. But length is only a part of it. At the same length, 100 to 149 words, the three GPT models score 76 to 79 and the other eight 48 to 59 (Gemini 4 Argon writes too few emails that short). Two emails can be equally long and still differ in how much of what they say is needed.

#Few unsupported claims, almost all of them invented

Every model gets almost all of its claims right, and the averages hide how often something slips. The three GPT models make the fewest claims, 8 to 9 per email, and an invented or misattributed one turns up in about one email in a hundred. The others state 14 to 22 facts per email, and between about one in ten and one in three of their emails has at least one. Kimi K3, Fable 5.1, Gemini 4 Argon and Fable 5 have the most, in 31% to 34% of their emails. Nearly all of it is invented, a fact the brief doesn't contain. Misattributed facts, a real result put on the wrong customer, are rare for every model. Assumed premises are a different matter. A premise about the recipient taken as given, such as a release or an expansion the research never mentions, is a move cold emails make on purpose, and some of these are reasonable guesses about someone in that role, so they don't count against the score. They are common, though. Gemini 4 Argon has one in 32% of its emails, the GPT models in 5% to 7%, and the rest in 15% to 26%. If nobody reads an email before it goes out, one wrong fact is enough to make the prospect stop reading.

#Dashes and bullet points read as AI

In every model that uses dashes, each extra one lowers the AI tells score. Sonnet 5.5 and Opus 5.5 almost never use one. Fable 5.1 goes from 57 without a dash to 43 with three or more, and GPT-6.1 Sol from 55 to 40. Fable 5 and Kimi K3 put three or more in more than eight in ten emails, and score 30 and 29 on those. Gemini 3.8 Flash changes least, from 31 to 26, because it scores low whatever the punctuation. Formatting costs 6 to 20 points. Opus 5.5's plain text emails score 63, the highest of any model, and its formatted ones 43. This gap is quite easy to close, since much of it comes from two habits that a single line in a prompt can change.

#Prompts to fix each model's weaknesses

Each model has habits that mark its emails as written by a machine or filled into a template, and the habits are consistent enough to correct. Below is what we would add to the prompt for each one. We don't try to fix every problem of each model here. The lines are just general enough to soften the habits the measurements show and the criterion where the model trails the others most, and they leave alone what it already does well. For example, a word limit made Sonnet 5.5 shorter but also took its voice, so its lines ask it to pick what matters and leave the rest out. Every added rule can also introduce a new habit of its own, so test any change on a few dozen of your own leads before rolling it out.

#GPT-6 Sol

Short and tight, with the best scores on concision and AI tells. 41% of its emails ask for "a 15-minute conversation".

Weakest Tenth of twelve on personalization and creativity, and ninth on voice.

#Sonnet 5.5

Long and formatted. A median of 274 words, bullets or bold in three quarters of the emails, a bare sign-off in 70% and a request to be pointed to someone else in more than a third.

Weakest Last of twelve on concision and tied eleventh on flow. Its concision falls from 57 at 100 to 149 words to 24 above 200.

#Fable 5

Long sentences full of dashes. About 21 words per sentence, four em dashes per email and formatting in two thirds of the emails. Fable 5.1 does better on every criterion but hallucination at almost the same price.

Weakest Tenth of twelve on concision and ninth on AI tells. Its concision falls from 54 at 100 to 149 words to 30 above 200.

#Kimi K3

Dashes and formatting. About four em dashes per email, long and complex sentences, formatting in half the emails and a bare sign-off in almost half.

Weakest Tenth of twelve on AI tells and ninth on concision. It is also tied last on hallucination, with an invented or misattributed claim in 34% of its emails, the most of any model.

#Gemini 3.8 Flash

The hardest emails to read and the most templated ask. About 25 words per sentence at a reading grade of 17, and "open to a" in 85% of its emails.

Weakest Last or tied last of twelve on personalization, creativity and voice, and eleventh on AI tells. Its voice score of 20 is 9 points below the next model.

#Opus 5.5

Close to Fable 5.1 at less than half the price, with the easiest reading of all and almost no dashes. Formatting in nearly half its emails, which costs it 20 points on AI tells, and a request to be pointed to someone else in one email in six.

Weakest Concision is where it trails the leader most, at 46 against 82 for GPT-6 Sol. It falls from 59 at 100 to 149 words to 38 above 200, and its median email is 189 words.

#Fable 5.1

The highest scores on personalization, creativity, voice and reply, at the highest price. Dashes in four emails in ten and a request to be pointed to someone else in one in six.

Weakest Concision is where it trails the leader most, at 43 against 82 for GPT-6 Sol. It is also tied last on hallucination, with an invented or misattributed claim in 33% of its emails, nearly all of them invented.

#GPT-6.1 Sol

One of the three cheapest models. Short like GPT-6 Sol, with more creativity and the hardest reading of the short models (grade 14). An em dash in four emails in five.

Weakest Tied ninth of twelve on personalization and eighth on voice.

#GPT-6 Astra

Short and tight like the other GPT models, about 107 words, with the best flow of all and the second-best concision, at five times the price of GPT-6.1 Sol. 39% of its emails ask "would it be…", most of them "would it be useful".

Weakest Tied eighth of twelve on personalization and seventh on voice.

#DeepSeek V4 Pro

Cheap and middle length, about 155 words, with "open to a" in 38% of its emails and formatting in a quarter.

Weakest Last of twelve on flow and reply, tied last with Gemini 3.8 Flash on personalization, eleventh on creativity and tenth on voice.

#Grok 4.7

The most varied writer, with a template phrase in only a third of its emails, plain text and third place on voice, at $0.035 an email.

Weakest Ninth of twelve on flow. Its concision falls from 58 at 100 to 149 words to 30 above 200, and its median email is 173 words.

#Gemini 4 Argon

Third on personalization at $0.016 an email, with the longest sentences of any model, about 26 words each. It asks "open to a" in 78% of its emails, formats half of them and uses about two em dashes per email. At medium reasoning effort it spends almost no tokens on reasoning.

Weakest Last of twelve on AI tells, and eleventh on voice and concision. It is also tied last on hallucination, with an invented or misattributed claim in 33% of its emails and an assumed premise in 32%, the most of any model.

#A worked example

Here are two emails written from the same brief, with Twain as the sender. The recipient is a made-up prospect of Twain. Sam Sample is Head of Sales Development at Acme Finance, a growing SaaS company that just raised a Series B and is hiring SDRs in Dublin. We picked GPT-6 Sol and Sonnet 5.5 because they are the shortest and the longest writers in the benchmark, so one brief shows the widest gap in what a model does with it.

Both open on the same detail, Sam's post about research eating into selling time. From there they part ways. GPT-6 Sol makes one proof point and a short ask in 87 words. Sonnet 5.5 takes 223, with two proof points in bullets, its integrations, its pricing and an offer to send a sample sequence.

#How it works

#The prompt

Every model gets the same prompt, with no writing instructions.

Today's date is {fixed benchmark date}.

My company:
{company knowledge}

Task:
Using all you know about what makes cold outreach work, write a cold email from me to the prospect described below. This prospect is on our target list.

This prospect matches this customer profile of my company: {segment name}
{persona}

Everything we know about this prospect:
{research}

I'm going to send exactly what you write, so it has to be ready to send. You can only use the information I've given you about my company and this prospect. Don't sign it, I'll add my own signature. Reply with only the email.

• We fix the date to the day we did the research, so a launch two weeks old stays two weeks old when we test a model next year.
• The company knowledge is the sender's own research, with its customers and proof, and without pricing, links or competitors.
• The persona is the segment the person belongs to, with its problem, the cost of doing nothing, how the sender solves it, the objections to expect and the proof meant for that segment.
• The research on the person is plain information about their role, their experience, their company, and recent, sourced insights.
• Every model writes in a sandbox. It can't search the web or open links, so everything it learns about the sender and the recipient comes from the prompt. A model may still know something about a company from its training. The prompt asks it to use only the information it is given, so the hallucination judge counts any claim the brief doesn't contain as unsupported, even if it happens to be true. That matches real outreach. A company's products, people and plans change, so what a model remembers may no longer be true, and only the research is current.
• Every model runs at the same reasoning effort (medium). We set it explicitly so a provider's default can't change the result.

#The dataset

• 100 companies in ten industries. They are B2B software, marketing agencies, IT services, recruiting, financial services, healthcare, real estate, manufacturing, logistics and professional services. In each, three with 100+ employees and seven smaller ones.
• 1,000 prospects, ten per company, each one a cold sales prospect for that company. No inbound leads, no follow-ups, no nurture campaigns, no hiring outreach.
• Everything was built fresh for the benchmark. We created each company's knowledge from its website, wrote its personas from that knowledge, researched each prospect on the same day, and matched them to the customer segment they fit.

The dataset stays private. We never publish an email from the benchmark, and the example above is made up.

#Scoring

Three AI judges from three providers score every email on the eight criteria, and a model's score is their average. The reply judge plays the recipient, so it sees only the email and what the recipient knows about themselves. Scores run from 0 to 100. Hallucination is scored claim by claim, in the style of FActScore. The judge lists every specific claim the email makes about the sender or the recipient and checks each one against the information the model was given. There is no overall score, because the criteria measure different things and any weighting of them would be our choice. We measure cost and speed on every call.

The three judges agree on the order of the models. Their rankings correlate at 0.85 or more on six of the eight criteria, and at 0.57 or more on flow and reply, where the models sit within a few points of each other. They differ in strictness. The strictest judge scores 7 to 17 points lower than the other two on every criterion but hallucination. Since all three read every email, it lowers every model's scores and barely changes their order.

The criteria are ours. They describe what we have seen work in cold outreach, and a team that writes differently would weigh them differently. The benchmark measures the models against that view of a good cold email. It does not measure any product.

#Limits

• The research and the criteria are ours. We did the research every model writes from, and we decided what the criteria are. They show what we think a good cold email is, and a team that thinks differently would score the same emails differently. The results also say nothing about how a model performs inside a product.
• Only cold sales outreach for now.
• AI judges are not readers. They are consistent and fast, but a person might weigh things differently. The reply judge plays the recipient, and no real replies stand behind its scores.
• The judges have models in the benchmark. A judge might favour models from its own provider, so we tested whether ours do. On average across the criteria, no judge rates its own provider's models more than 2 points differently than the other two judges do.
• A bare prompt. Models that do better with more guidance get none here, on purpose, since we want to see what each model does on its own.

#What's next

The leaderboard is updated as new models come out. We run each one on the same 1,000 briefs and add it there. Because the dataset, the prompt, the reasoning effort and the date don't change, you can compare a score from today with a score from next year directly.

#Changelog

• 1.0 (date to come). First release, twelve models.