A Nature survey of ~1500 scientists reported that more than 70% had failed to reproduce another scientist’s experiments, and 50% had failed to replicate their own (Baker, 2016). This study wasn’t specific to machine learning, but the crisis has its own flavour in computational drug discovery.

One aspect of this is that available datasets have known quality issues, and results built on them can be fragile. MoleculeNet and the Therapeutic Data Commons (TDC) opened drug discovery to a wider ML community, but they’re no longer sufficient for driving further advances (Wognum et al., 2024).

Amongst a number of issues, 71% of molecules in one MoleculeNet dataset (BACE) contained at least one undefined stereocenter (a point where the same atoms can sit in two different 3D arrangements), making it unclear what chemical entity is actually being modelled (Li et al., 2026).

The stakes are high. In 2017, a research group found their cancer-target inhibitor inactive from one vendor and highly active from another. Eventually they traced this to vendors selling different mixtures of the compound’s two 3D forms where only one was active on the target and mechanism they were investigating (Baker, 2017).

Most approved pharmaceuticals are relatively small chemical molecules, typically weighing under 900 g/mol, with most of the rest being biologics. Whilst in Oxford on the UNIQ+ summer internship programme, I focused on ADME, an aspect of early stage drug discovery, where assays are measured in vitro as a stand-in to predict how a compound will behave in vivo (in the human body) before clinical trials. Absorption (does it enter the body), Distribution (where does it go), Metabolism (how quickly it’s transformed) and Excretion (how quickly it’s eliminated) are processes critical to whether a candidate succeeds. 

Using ML we can predict these in-vitro ADME endpoints before synthesis, so chemists prioritise the molecules most likely to have usable pharmacokinetics, reducing costs and wasted effort. Assays standing in for the 4 ADME processes are modelled as endpoints below (I worked with four of these: HLM, RLM, SOL and MDR1):

• Absorption – Solubility (SOL), Efflux Ratio (MDR1)
• Distribution – Efflux Ratio (MDR1), Plasma Protein Binding (PPB – Human/Rat)
• Metabolism – Human/Rat Liver Microsome CLint (HLM/RLM)
• Excretion – Not covered by these endpoints

The Biogen dataset (Fang et al., 2023) was generated by a real-world drug discovery campaign run over two years, with each endpoint measured by a single assay in one lab. The result is a high-quality dataset without the inter-assay noise that plagues other benchmarks. It isn’t immune to BACE’s problem, though: 6% of the raw dataset (210 of 3,521 compounds) still carries at least one undefined stereocentre. That’s far below BACE’s 71%, but not zero.

The paper’s authors benchmarked various molecular representations and models on their temporally split confidential dataset, and released a public version for researchers to use. The public version has the timestamps removed, so only scaffold or random splits are possible on it. 

They reported that the public results matched their confidential benchmarks, and made several further claims across model choice, feature choice, and data quantity. I focused on the three most interesting: (1) that non-RF models outperform RF; (2) that this margin depends on the endpoint; and (3) that molecular representation matters more than algorithm choice.

These are the claims anyone building on this would act on: they decide which model you reach for and how much effort goes into the representation. If they turn out to depend on choices the paper didn’t vary, that matters for anyone using this dataset as a benchmark. So I set out to reproduce them and see what happened when I changed things the paper held fixed.

I tested these by replicating their heatmaps of pairwise model performance, where each cell is the difference in mean cross-validated Pearson r between two models, and testing how molecular representation choice affects performance across 4 endpoints when using a random split.

Did it work?

I’ll go through two heatmaps. Each row and column is a model, and each cell is the difference in mean cross-validated Pearson r (the correlation between predicted and measured values) between that row’s model and that column’s model. A positive value means the row model performed better, and darker green means stronger evidence for that difference. On the HLM endpoint (random split) the replication matched up incredibly well. Compare the green shades collectively with the pink (NS) and they mirror each other closely.

One area that didn’t quite match up was that XGBoost and MPNN1 (Graph based neural network using the models learned representations) performed about the same as RF on 3 of 4 heatmaps, and significantly worse than RF on 1 of 4 heatmaps (MDR1) so not all models were better than RF on the public dataset. MPNN2 in contrast used RDKit descriptors concatenated with graph based learned representations and outperformed RF. The smaller dataset size meant MPNN1’s learned representations were insufficient to outperform RF.

Biogen Paper, 22,822 compounds	Public data, 3,087 compounds

On the SOL endpoint, replication was more challenging: the only pairwise comparison to reach significance was RF outperforming XGBoost (p < 0.01). With 2,173 compounds vs their 10,350 I had less power to detect differences. Their larger, temporally split dataset did show their top 3 models beating RF here.

I checked 2/4 endpoints for molecular representation (MDR1 and HLM). Representation had a greater effect than model choice, and the general trend of hybrid representations outperforming Morgan fingerprints and RDKit descriptors alone held up (hybrid here means FCFP4 concatenated with RDKit descriptors, and hybrid_ecfp4 the same with ECFP4), compare the 2 box plots below.

When I tested FCFP4 against ECFP4 (the industry standard), FCFP4 generally came out ahead, and when combined with RDKit descriptors it was better every time. Enough to suggest FCFP4 is worth benchmarking rather than defaulting to ECFP4. See the public box plot below.

	Biogen Paper, 17,578 compounds
	Public data, 2,642 compounds

Feature Selection

From there, I looked at other areas I could inspect and here’s the bit that really surprised me. Two vector descriptors from RDKit; PEOE_VSA (surface area binned by partial charge ranges) and SlogP_VSA (surface area binned by contribution to fat solubility) retained 82% of the 48-descriptor baseline’s predictive performance (mean cross-validated base R² = 0.42, subset R² =0.36, averaged across the four endpoints’ 5-fold CV scores on LightGBM trained on RDKit alone).

Getting there required a staged pipeline. Starting from 50 descriptors, a constant-variance filter removed the non-informative ones (50 → 48, or 314 features once the vector-valued descriptors are expanded). I then compressed each vector descriptor to its top five principal components and scored it by mutual information with the endpoints. Canonical correlation analysis dropped the lower-MI member of any near-duplicate pair (CCA ≥ 0.9): 48 → 27. Variance inflation factor pruning removed the most collinear descriptor, near-ties broken by MI, until all remaining had VIF ≤ 5: 27 → 16. Finally, LightGBM recursive feature elimination, gain summed across each descriptor’s expanded columns, maximised across the four endpoints, reduced down to an elbow at two.

Data Quantity

I also looked at how model performance changes with the amount of training data available. As scarcity increased, RF’s predictive power was equal to and in some cases surpassed the other models, contrasting with the paper’s claim that non-RF models consistently outperform RF.

This could be a result of my splitting approach: a random split leaves the test compounds in the same chemical space as the training set, so shrinking the training pool simulates having few examples of chemistry you already know. A temporal split, which they used, simulates something harder: few examples, and the compounds you’re predicting on are structurally unfamiliar. That’s closer to how these models actually get used, so my low-data result speaks to an easier problem than theirs.

Takeaways

Overall, the paper’s conclusions largely held up: non-RF models outperformed RF, the size of that margin depended on the endpoint, molecular representation was more important than algorithm choice and most interestingly, I found that retaining 82% predictive power was possible when using 2 vector descriptors. An exception was that XGBoost and MPNN1 performed about the same as RF on the smaller, public dataset. Less data on the SOL endpoint resulted in most models failing to reach a significant difference in performance. Another takeaway would be that the best models today are not necessarily the best models tomorrow, a reminder to test our assumptions as datasets grow. 

Working on this project shifted how I engage when reading papers. Rather than just understanding what the authors did, I now focus more on why they did it, and particularly on methodological choices, whether a decision reflects standard practice in the field or is specific to that piece of work. When the reasoning isn’t stated, that distinction matters, as it shapes how much weight I place on their results, how strong their approach is to build from, and how to extend their work. That applies to my own conclusions too: the model rankings I measured hold for the data I had, and a larger dataset could reorder them.

All code for this project is available here.

Acknowledgements

Thanks to Acer Blake, Prof. Fergus Imrie and Prof. Charlotte Deane for supervising this project, for their patient guidance and input, and for hosting me in OPIG. Thanks also to the group as a whole for being so welcoming, and to the UNIQ+ programme for the opportunity.

References

Baker, M. (2016). 1,500 Scientists lift the lid on reproducibility. Nature, 533(7604), 452–454. https://doi.org/10.1038/533452a

Baker, M. (2017). Check your chemistry. Nature, 548(7668), 485–488. https://doi.org/10.1038/548485a

Fang, C., Wang, Y., Grater, R., Kapadnis, S., Black, C., Trapa, P., & Sciabola, S. (2023). Prospective validation of machine learning algorithms for absorption, distribution, metabolism, and excretion prediction: An industrial perspective. Journal of Chemical Information and Modeling, 63(11), 3263–3274. https://doi.org/https://doi.org/10.1021/acs.jcim.3c00160

Li, Z., Chen, X., Wen, H., Zhang, R. Q., Li, M., Zhang, X., Yin, H., Qiang, Y., Lam, K.-Y., Pietro, L., & Yiu, S.-M. (2026). A systematic survey and benchmark of deep learning for molecular property prediction in the foundation model era. arXiv.Org. https://arxiv.org/abs/2604.16586

Wognum, C., Ash, J. R., Aldeghi, M., Rodríguez-Pérez, R., Fang, C., Cheng, A. C., Price, D. J., Clevert, D.-A., Engkvist, O., & Walters, W. P. (2024). A call for an industry-led initiative to critically assess machine learning for real-world drug discovery. Nature Machine Intelligence, 6(10), 1120–1121. https://doi.org/10.1038/s42256-024-00911-w