Research
Three questions we tested.
How we test our own numbers: the question, the method, the result with its uncertainty, and the limits.
How much does the draft decide?
Question
Once both teams are locked in, how well do the ten champions predict who wins?
Method
We compared four models with a baseline that always picks the blue side: one number per team (the sum of each champion's patch win rate), team make-up, a rating per champion, and boosted trees. Games come from players who chose to share them, with names removed. Every score is measured on games the model hadn't seen (5-fold cross-validation, repeated 10 times), with 95% ranges.
Result
The draft matters, a little. The best model gets 60% of games right against 54% for always picking blue, and its AUC is 0.61 (95% range 0.58 to 0.65, where 0.50 is a coin flip). Shuffling the results and retraining beats it about 1 time in 300. The simplest model won: one number per team did as well as the detailed ones. Where the draft edge is clearest (a third of games) the favourite wins about 64%; in the closest third, 55%.
Limits
1,081 games is a small sample for a five-versus-five game, so the ranges are wide, and they come from a small group of players. The model ignores augments, items and skill on purpose. We rerun it as more games arrive.
Did the buff work?
Question
When the patch notes change a champion, does its win rate move by more than chance?
Method
Each change is compared with what the untouched champions did over the same patch change (a difference in differences), and judged against how far an untouched champion typically moves. Changes are also sized as Mayhem modifiers, to estimate what each kind of change is worth.
Result
Across 66 changes in 10 patches, 26 moved the champion the intended way beyond noise, 4 moved it the opposite way, and 36 stayed within noise or had no clear intent. A 5% damage-dealt modifier is worth about 1.8 win-rate points (likely range 1.3 to 2.4).
Limits
A win rate also moves for reasons the notes don't list, such as item and augment changes, new tricks and hotfixes. Many changes bundle several lines, so the value of each kind is rough, and small changes can't be told from noise.
Does the Simulator match real games?
Question
Does the Simulator rank champions' damage the way real games do?
Method
For every champion, the Simulator's damage per second on its most-played build is compared with its share of damage in real games. Agreement is the rank correlation (Spearman) with a 95% range from 2,000 resamples, and a role-adjusted model is scored leave-one-out. Every past Simulator version is replayed with today's yardstick, so progress is measured, not claimed.
Result
Across 173 champions the rank correlation is 0.59 (95% range 0.47 to 0.69). Adjusting for role lifts it to 0.62, and predicted damage shares line up with real ones with a calibration slope of 0.98 (1.00 is perfect). Over 16 Simulator versions it has climbed from 0.37 to 0.59.
Limits
Real damage share depends on how people play, not only on the champion, so a perfect score is out of reach. Within a single role the groups are small and the ranges wide. It checks damage, not whether a build is good.
See the Methodology page for how the numbers are made.