Research

Three questions we tested.

How we test our own numbers: the question, the method, the result with its uncertainty, and the limits.

How much does the draft decide?

Question

Once both teams are locked in, how well do the ten champions predict who wins?

Method

We compared four models with a baseline that always picks the blue side: one number per team (the sum of each champion's patch win rate), team make-up, a rating per champion, and boosted trees. Games come from players who chose to share them, with names removed. Every score is measured on games the model hadn't seen (5-fold cross-validation, repeated 10 times), with 95% ranges.

Result

The draft matters, a little. The best model gets 60% of games right against 54% for always picking blue, and its AUC is 0.61 (95% range 0.58 to 0.65, where 0.50 is a coin flip). Shuffling the results and retraining beats it about 1 time in 300. The simplest model won: one number per team did as well as the detailed ones. Where the draft edge is clearest (a third of games) the favourite wins about 64%; in the closest third, 55%.

Limits

1,081 games is a small sample for a five-versus-five game, so the ranges are wide, and they come from a small group of players. The model ignores augments, items and skill on purpose. We rerun it as more games arrive.

Did the buff work?

Question

When the patch notes change a champion, does its win rate move by more than chance?

Method

Each change is compared with what the untouched champions did over the same patch change (a difference in differences), and judged against how far an untouched champion typically moves. Changes are also sized as Mayhem modifiers, to estimate what each kind of change is worth.

Result

Across 66 changes in 10 patches, 26 moved the champion the intended way beyond noise, 4 moved it the opposite way, and 36 stayed within noise or had no clear intent. A 5% damage-dealt modifier is worth about 1.8 win-rate points (likely range 1.3 to 2.4).

Limits

A win rate also moves for reasons the notes don't list, such as item and augment changes, new tricks and hotfixes. Many changes bundle several lines, so the value of each kind is rough, and small changes can't be told from noise.

Does the Simulator match real games?

Question

Does the Simulator rank champions' damage the way real games do?

Method

For every champion, the Simulator's damage per second on its most-played build is compared with its share of damage in real games. Agreement is the rank correlation (Spearman) with a 95% range from 2,000 resamples, and a role-adjusted model is scored leave-one-out. Every past Simulator version is replayed with today's yardstick, so progress is measured, not claimed.

Result

Across 173 champions the rank correlation is 0.59 (95% range 0.47 to 0.69). Adjusting for role lifts it to 0.62, and predicted damage shares line up with real ones with a calibration slope of 0.98 (1.00 is perfect). Over 16 Simulator versions it has climbed from 0.37 to 0.59.

Limits

Real damage share depends on how people play, not only on the champion, so a perfect score is out of reach. Within a single role the groups are small and the ranges wide. It checks damage, not whether a build is good.

See the Methodology page for how the numbers are made.