Some recent cheminformatics papers have been using “win rate” (i.e., how often a model is the best across a set of tasks) to evaluate benchmarking performance. Although it can be a useful proxy for how good a model is, I have some gripes with this metric, particularly if it is presented without additional context.
Firstly, it can hide small differences; a model may achieve the top-ranked performance, but with only a very small margin over the next best model. Secondly, win rate is not a statistical test for ranking; a seemingly high win rate may not translate into clear separation from alternative approaches. Finally, win rate can hide weak points; a model that wins 80% of tasks but performs awfully on the rest appears more trustworthy than it perhaps should be. A more consistent model with a lower win rate may therefore be more useful in practice.<\/p>\n\n\n\n
Fortunately there are a couple of remedies for these issues:<\/p>\n\n\n\n
Here’s a worked example with simulated data:<\/p>\n\n\n\n\n\n\n\n
First, basic imports and variable definitions:<\/p>\n\n\n\n
import numpy as np, pandas as pd\nfrom itertools import combinations\nfrom scipy.stats import studentized_range, friedmanchisquare, wilcoxon\n\nrng = np.random.default_rng(42)\nmodels = [\"New Model\", \"Old Model\", \"Baseline A\", \"Baseline B\"]\nn_tasks, n_cv = 10, 25 # 5 x 5 repeated CV<\/code><\/pre>\n\n\n\nNext, a function for simulating our task data:<\/p>\n\n\n\n
def simulate(task):\n base = rng.uniform(0.55, 0.75) # a random starting point for our simulated metric\n means = base + np.array([0.015, 0.01, -0.005, -0.01]) # some base means for each model\n if task in (3, 8):\n means[0] = base - 0.10 # Set `New Model` to fail on two tasks\n fold = rng.normal(0, 0.04, (n_cv, 1)) # add a shared effect per fold\n return means + fold + rng.normal(0, 0.02, (n_cv, len(models)))<\/code><\/pre>\n\n\n\nNow, we need functions for our within task assessment:<\/p>\n\n\n\n
def paired_cohens_d(group1, group2):\n \"\"\"Paired version of Cohen's d.\"\"\"\n diff = group1 - group2\n diff_mean = np.mean(diff)\n diff_sd = np.std(diff, ddof=1)\n return diff_mean \/ diff_sd\n\ndef tukey(scores):\n \"\"\"\n Tukey's HSD using repeated measures of ANOVA\n (to account for fold pairs), plus Cohen's d.\n \"\"\"\n n, k = scores.shape\n resid = scores - scores.mean(0) - scores.mean(1, keepdims=True) + scores.mean()\n df = (n - 1) * (k - 1)\n se = np.sqrt((resid**2).sum() \/ df \/ n)\n rows = []\n for i, j in combinations(range(k), 2):\n diff = scores[:, i].mean() - scores[:, j].mean()\n p = studentized_range.sf(abs(diff) \/ se, k, df)\n cohen_d_val = paired_cohens_d(scores[:, i], scores[:, j])\n rows.append((\n models[i], models[j], diff,\n cohen_d_val, p\n ))\n return pd.DataFrame(rows, columns=[\"model_1\", \"model_2\", \"diff\", \"paired_cohens_d\", \"p_val\"])<\/code><\/pre>\n\n\n\nLet’s run it and display an example comparison table from the first task:<\/p>\n\n\n\n
wins, sig_ranks, ranks = pd.Series(0, index=models), [], []\nscore_means = []\nfor t in range(n_tasks):\n scores = simulate(t)\n score_means.append(scores.mean(0))\n comparison = tukey(scores)\n if t == 0:\n display(comparison.round(3))\n wins[models[scores.mean(0).argmax()]] += 1\n sig = comparison[comparison.p_val < 0.05]\n worse = np.where(sig[\"diff\"] > 0, sig.model_2, sig.model_1)\n sig_ranks.append(1 + pd.Series(worse).value_counts().reindex(models, fill_value=0))\n sorted_models = [models[i] for i in scores.mean(0).argsort()[::-1]]\n ranks.append(pd.Series(\n [1 + i for i in range(len(sorted_models))],\n index=sorted_models\n ))\nscore_means = np.array(score_means)\nsig_ranks = pd.DataFrame(sig_ranks, index=[f\"Task {t + 1}\" for t in range(n_tasks)])\nranks = pd.DataFrame(\n ranks,\n index=[f\"Task {t + 1}\" for t in range(n_tasks)],\n columns=models\n)<\/code><\/pre>\n\n\n\n\n