Some recent cheminformatics papers have been using “win rate” (i.e., how often a model is the best across a set of tasks) to evaluate benchmarking performance. Although it can be a useful proxy for how good a model is, I have some gripes with this metric, particularly if it is presented without additional context.

Firstly, it can hide small differences; a model may achieve the top-ranked performance, but with only a very small margin over the next best model. Secondly, win rate is not a statistical test for ranking; a seemingly high win rate may not translate into clear separation from alternative approaches. Finally, win rate can hide weak points; a model that wins 80% of tasks but performs awfully on the rest appears more trustworthy than it perhaps should be. A more consistent model with a lower win rate may therefore be more useful in practice.

Fortunately there are a couple of remedies for these issues:

1. Within each task, compare models with repeated cross-validation results and statistical tests, such as multiple comparison tests [1], like Tukey’s HSD, or pairwise tests with a p-value correction, like Wilcoxon signed-rank tests. Test results should be reported with appropriate effect sizes, such as Cohen’s d for Tukey and rank-biserial correlation for Wilcoxon. These tests enable us to observe mean rankings of the models with significance taken into consideration.
2. Subsequently, we can use a Friedman test on model metrics across tasks [2-3]. The Friedman test is for detecting whether average rankings of groups (i.e., models) are significantly different across different settings (i.e., tasks). Because it averages over every task, a model that wins often but fails badly elsewhere is penalised rather than rewarded.

Here’s a worked example with simulated data:

First, basic imports and variable definitions:

import numpy as np, pandas as pd
from itertools import combinations
from scipy.stats import studentized_range, friedmanchisquare, wilcoxon

rng = np.random.default_rng(42)
models = ["New Model", "Old Model", "Baseline A", "Baseline B"]
n_tasks, n_cv = 10, 25  # 5 x 5 repeated CV

Next, a function for simulating our task data:

def simulate(task):
    base = rng.uniform(0.55, 0.75) # a random starting point for our simulated metric
    means = base + np.array([0.015, 0.01, -0.005, -0.01]) # some base means for each model
    if task in (3, 8):
        means[0] = base - 0.10 # Set `New Model` to fail on two tasks
    fold = rng.normal(0, 0.04, (n_cv, 1)) # add a shared effect per fold
    return means + fold + rng.normal(0, 0.02, (n_cv, len(models)))

Now, we need functions for our within task assessment:

def paired_cohens_d(group1, group2):
    """Paired version of Cohen's d."""
    diff = group1 - group2
    diff_mean = np.mean(diff)
    diff_sd = np.std(diff, ddof=1)
    return diff_mean / diff_sd

def tukey(scores):
    """
    Tukey's HSD using repeated measures of ANOVA
    (to account for fold pairs), plus Cohen's d.
    """
    n, k = scores.shape
    resid = scores - scores.mean(0) - scores.mean(1, keepdims=True) + scores.mean()
    df = (n - 1) * (k - 1)
    se = np.sqrt((resid**2).sum() / df / n)
    rows = []
    for i, j in combinations(range(k), 2):
        diff = scores[:, i].mean() - scores[:, j].mean()
        p = studentized_range.sf(abs(diff) / se, k, df)
        cohen_d_val = paired_cohens_d(scores[:, i], scores[:, j])
        rows.append((
            models[i], models[j], diff,
            cohen_d_val, p
        ))
    return pd.DataFrame(rows, columns=["model_1", "model_2", "diff", "paired_cohens_d", "p_val"])

Let’s run it and display an example comparison table from the first task:

wins, sig_ranks, ranks = pd.Series(0, index=models), [], []
score_means = []
for t in range(n_tasks):
    scores = simulate(t)
    score_means.append(scores.mean(0))
    comparison = tukey(scores)
    if t == 0:
        display(comparison.round(3))
    wins[models[scores.mean(0).argmax()]] += 1
    sig = comparison[comparison.p_val < 0.05]
    worse = np.where(sig["diff"] > 0, sig.model_2, sig.model_1)
    sig_ranks.append(1 + pd.Series(worse).value_counts().reindex(models, fill_value=0))
    sorted_models = [models[i] for i in scores.mean(0).argsort()[::-1]]
    ranks.append(pd.Series(
        [1 + i for i in range(len(sorted_models))],
        index=sorted_models
    ))
score_means = np.array(score_means)
sig_ranks = pd.DataFrame(sig_ranks, index=[f"Task {t + 1}" for t in range(n_tasks)])
ranks = pd.DataFrame(
    ranks,
    index=[f"Task {t + 1}" for t in range(n_tasks)],
    columns=models
)

Next, we can investigate the mean rankings:

summary = pd.DataFrame({
    "Win Rate": wins / n_tasks,
    "Mean Rank with Tukey": sig_ranks.mean(),
    "Mean Rank": ranks.mean(),
}).sort_values("Mean Rank with Tukey")
display(summary.round(2))

So, are these difference in ranking significant? Let’s try the Friedman test:

stat, p = friedmanchisquare(*score_means.T)
print(f"Friedman: chi2 = {stat:.2f}, p = {p:.4f}")

So, the Friedman result is significant, and at least one of the models is separable from the rest. To work out which models are causing this, we can use a non-parametric post-hoc test, such as Wilcoxon or Nemenyi:

def bonferroni_correction(p_val, m):
    """Apply Bonferroni correction to p-values."""
    return np.minimum(p_val * m, 1.0)

results = pd.DataFrame(columns=[i for i in models], index=[i for i in models], dtype=float)
num_comparisons = len(models) * (len(models) - 1) / 2
for i, j in combinations(range(len(models)), 2):
    stat, p = wilcoxon(score_means[:, i], score_means[:, j])
    p = bonferroni_correction(p, num_comparisons)
    results.iloc[i, j] = p
    results.iloc[j, i] = p
results = results.round(4)
display(results)

So, the only significant differences in model ranking across tasks are between Old Model and the two baselines. Old Model is reliably better than both baselines, whereas New Model can’t be separated from any other model, despite an 80% win rate and the lowest mean rank (without considering ties by Tukey p-values).

Together, these address all three issues. Within tasks, effect sizes show whether metric differences are meaningful and statistical tests show whether they’re significant. Across tasks, average ranks reward consistency and the Friedman test shows whether differences in average rank are significant.

References

[1] Ash, J.R., et al., Practically Significant Method Comparison Protocols for Machine Learning in Small Molecule Drug Discovery. J. Chem. Inf. Model. 22 September 2025; 65 (18): 9398–9411. https://doi.org/10.1021/acs.jcim.5c01609
[2] Friedman, M. The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance. J. Am. Stat. Assoc. 1937; 32 (200): 675–701. https://doi.org/10.1080/01621459.1937.10503522
[3] Demšar, J. Statistical Comparisons of Classifiers over Multiple Data Sets. J. Mach. Learn. Res. 2006; 7: 1–30. https://jmlr.org/papers/v7/demsar06a.html