twain-bench leaderboard

twain-bench asks each model for the best cold email it can write for a company to a prospect, with no writing instructions, 1,000 times across 100 companies. AI judges score every email on 8 criteria. The article explains the method and the first findings.

#What each criterion measures

Scores are 0 to 100; higher is better.

#Email criteria

Each email on its own.

voice

Whether the email sounds like one particular person wrote it.

flow

Whether the email reads as one line of thought, with every link between ideas visible, from the reason for writing to the ask.

concision

Whether the email says what it needs to say, and nothing more.

ai_tells

Whether the email is free of the telltale signs of AI-written text.

personalization

Whether the email is built on the detail about the recipient that makes the offer matter to them now.

creativity

Whether the email brings a fresh angle that fits the recipient, rather than the usual cold email.

reply

Whether the recipient would reply, judged by a model playing the recipient.

hallucination

Whether the email's facts about the sender and the recipient are true to the information the writer was given.

#Writing stats

Counted, not judged. The greeting and sign-off aren't counted as sentences.

Words

Words in the body, not counting the subject line.

Sentences

Sentences of running text, not counting the greeting and sign-off.

Words / sentence

Average sentence length in words.

Length spread

Standard deviation of sentence length within an email, in words. Low means every sentence is about the same length.

Reading ease

Flesch reading ease, 0–100: higher is easier. 60–70 is plain English. Noisy on short texts, so read it as a model average.

Grade level

Flesch–Kincaid grade: the US school year needed to follow the text.

Complex sentences

Percent of sentences with a subordinate clause (because, if, which, that...), from spaCy's dependency parse.

Questions

Question marks per email.

Links

URLs or web addresses per email.

Em dashes

Em dashes (or -- used as one) per email.

Placeholders

Percent of emails with a placeholder like [Company], not counting one for the sender's own name.

Markdown

Percent of emails with markdown in the body: bold, headings, bullets, numbered lists or links.

Subject line

Percent of emails that start with a subject line.

#Repetition

Counted over all of a model's emails together, after dropping the subject line and the greeting.

Template phrase

Percent of emails containing a 3-word phrase that appears in at least a tenth of the run's emails. Phrases about one sender or lead can't reach that share, so these are phrases the model reaches for on its own.

Same opener

Percent of emails whose first four words after the greeting are the run's most common opening.

Same-company overlap

For the emails to one sender's leads, the share of 3-word phrases two emails have in common, averaged over pairs and senders. High means near-identical emails to different people at one company.

Distinct word pairs

Unique word pairs as a percent of all word pairs across the run. Higher means more varied wording. It falls as the total text grows, so a model that writes more starts lower.

Distinct word triples

The same for runs of three words. Less driven by common pairs like "of the", so it tracks repeated phrasing more closely.