{"id":12852,"date":"2025-08-08T17:38:11","date_gmt":"2025-08-08T16:38:11","guid":{"rendered":"https:\/\/www.blopig.com\/blog\/?p=12852"},"modified":"2025-08-08T17:50:19","modified_gmt":"2025-08-08T16:50:19","slug":"gpt-5-achieves-state-of-the-art-chemical-intelligence","status":"publish","type":"post","link":"https:\/\/www.blopig.com\/blog\/2025\/08\/gpt-5-achieves-state-of-the-art-chemical-intelligence\/","title":{"rendered":"GPT-5 achieves state-of-the-art chemical intelligence"},"content":{"rendered":"\n

I have run ChemIQ (our chemical reasoning benchmark) on GPT-5. The model achieves state-of-the-art performance with substantial improvements in the ability to interpret SMILES strings. Read my analysis and initial findings below. Scroll to the end for some cool demos. <\/p>\n\n\n\n

\"\"<\/a><\/figure>\n\n\n\n

Figure 1: <\/strong>Success rates for each model on the ChemIQ reasoning benchmark. Horizontal brackets between adjacent bars indicate the result of a two-tailed McNemar\u2019s test comparing paired outcomes for the same questions. Significance levels are shown as: n.s.<\/code> (not significant, p \u2265 0.05), *<\/code> (p < 0.05), **<\/code> (p < 0.01), and ***<\/code> (p < 0.001). <\/em><\/p>\n\n\n\n\n\n\n\n

ChemIQ is a chemical reasoning benchmark<\/h2>\n\n\n\n

In May we released our large language model (LLM) chemical reasoning benchmark which assesses whether LLMs understand the structure of molecules. The benchmark includes a range of questions, starting with easy counting tasks, and progressing to very difficult tasks such as structure elucidation from 2D NMR data. In July we released an updated preprint where we benchmarked additional reasoning models and improved our NMR questions. You can read the paper on arxiv here: Assessing the Chemical Intelligence of Large Language Models (https:\/\/arxiv.org\/abs\/2505.07735<\/a>).<\/p>\n\n\n\n

GPT-5 reaches SOTA performance on ChemIQ<\/h1>\n\n\n\n

I ran our benchmark within 2 minutes of gaining access to GPT-5. The headline result is that GPT-5 achieves state-of-the-art (SOTA) performance on ChemIQ, scoring 70.2% and exceeding the previous best models by ~14%. It is important to highlight that our benchmark is designed to test the base LLM without tool use – in each case the LLM has to work through the question step-by-step without the aid of tools or code interpreters.<\/p>\n\n\n\n

GPT-5 is not stupid<\/h1>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n

Figure 2: <\/em><\/strong>Radar plot showing model performance by question sub category. Additional model results can be found in the ChemIQ preprint (https:\/\/arxiv.org\/pdf\/2505.07735<\/a>) <\/em><\/p>\n\n\n\n

When presenting my research, I always use the line “large language models are extremely smart, but also really stupid<\/em>“. I say this because LLMs struggle with apparently trivial tasks of counting how many letters are in a word (i.e. carbon counting), however can also do really impressive tasks like NMR elucidation. I need to update my research presentations now though: in our benchmark, GPT-5 was essentially perfect at the carbon counting, ring counting, shortest path, and Free-Wilson analysis tasks. It appears GPT-5 rarely makes stupid mistakes. <\/strong><\/p>\n\n\n\n

Of greatest surprise to me is that GPT-5 achieved 99% on the shortest path tasks. These questions are genuinely difficult, requiring interpretation of the graph structure of molecules from the SMILES string and then doing a path finding algorithm to find the shortest path between two points. Previously o3-mini struggled to answer this question when presented with a randomized SMILES representation (in this question, canonical SMILES is substantially easier to solve). In these results, GPT-5 only made a single mistake in all 108 shortest path questions.<\/p>\n\n\n\n

Gemini 2.5 pro is still best at NMR elucidation<\/h1>\n\n\n\n

My favorite subcategory in our benchmark is the 2D NMR elucidation questions. As a chemist, the concept of an LLM being able to solve the structure of a molecule from NMR data is mind blowing (and I think most chemists feel the same way). The results of our test show that GPT-5 has not had a substantial gain in its NMR elucidation capabilities, and Gemini 2.5 Pro is still in the lead. Specifically on the 2D NMR subset, both o3-mini and GPT-5 scored 6% (3\/50) whereas Gemini 2.5 Pro scored 20% (10\/50). I would have expected GPT-5 to do much better at these questions; I will try to investigate this further.<\/p>\n\n\n\n

Time for ChemIQ-2?<\/h1>\n\n\n\n

When I started working on reasoning models in January, I anticipated the performance of LLMs in chemistry to improve rapidly over the course of the year. My supervisors and I expected ChemIQ to be saturated by the end of the year. While GPT-5 only reaches 70% overall success, it has achieved essentially perfect performance on four out of eight question categories. Don’t worry though, we anticipated this would happen. The questions in ChemIQ are algorithmically generated meaning we can quickly create a new set of even harder questions when needed. I also have a few more novel benchmark questions that we didn’t include in the original paper, which I might include in ChemIQ-2.<\/p>\n\n\n\n

Demo 1: Interactive kinetics and titration dashboard<\/h1>\n\n\n\n

This is kinda crazy. I gave the model the prompt “Create an impressive chemistry demo that showcases your capabilities. I need to embed this in our wordpress blog”<\/em>. In one go, GPT-5 generated an interactive web app (embedded below). You can enter Reaction Kinetics or Acid\u2013Base Titration parameters and the tool will plot the expected curves. (I didn’t do any prompt optimization, I didn’t try multiple times, this is what came out first try.)<\/p>\n\n\n

\n
\n
\n

\ud83d\udd2c ChemLab Interactive<\/h1>\n Kinetics \u2022 Titration<\/span>\n <\/header>\n\n