Python is a shockingly slow language. A test on a raspberry pi of simply “turn this pin on and off as fast as you can” gave the results below.
| System | Library | Speed |
|---|---|---|
| Shell | /proc/mem access | 2.8 kHz |
| Shell / gpio utility | WiringPi gpio utility | 40 Hz |
| Python | RPI.GPIO | 70 kHz |
| Python | wiringPi2 bindings | 28 kHz |
| Ruby | wiringPi bindings | 21 kHz |
| C | Native library | 22 MHz |
| C | BCM 2835 | 5.4 MHz |
| C | wiringPi | 4.1 – 4.6 MHz |
| Perl | BCM 2835 | 48 kHz |
So, how do we speed up python? The simple answer is not to use python, and use a faster language. Whilst that may sound facetious, some of the most commonly used python libraries are actually written in, or have cores in much faster languages.
| Library | ~ % Python | Notes |
|---|---|---|
| PyTorch | 62 | Core in C++ |
| Numpy | 60 | Core in C |
| TensorFlow | 20 | Core in C++ |
| OpenCV | 3 | Core in C++ |
If using pre-made libraries with fast cores still doesn’t provide the speed-up you need, then what? Find the slow bit – Test it, Tidy it, Time it.
Test it – Quite often, things are slow because they’re a bit buggy. So a bit more of a concerted effort testing the code wouldn’t go amiss.
Tidy it – If the code’s spaghetti-like and has lost its elegance as you’ve had to put in workarounds for workarounds, perhaps it’s going through seldom used pathways and incurring an efficiency hit every time. Refactoring your code can help its performance.
Time it – If the above are no help, time to reach for the big guns and time each section of code and see where the slow-down is happening. This can be done with a profiler and/or a visualiser, a selection of which are mentioned below.
- Timeit – Measures the execution time of small code snippets
- Profile – Collect detailed runtime statistics (consider using cProfile or PyInstrument)
- cProfile –
- Similar to Profile, but better.
- Deterministic profiler.
- Monitors every function call, return and exception event, records timing for each.
- PyInstrument –
- Low overhead profiler and visualiser
- Statistical profiler.
- Will miss things, but mainly the things which are inherently quick anyway.
- Measures at repeating intervals which function is being called.
I would recommend a look at PyInstrument
PyInstrument can be called from the command line as a simple module, where you also specify an HTML file as interactive output on how your code is running.
python3 -m pyinstrument -o count-char.html count-char.pyPyInstrument also provides some fairly easy to read visualisations of your code.
Or even more complex code such as:
Python visualisers
- Runsnakerun (https://github.com/Shoobx/runsnake)
- Snakeviz (https://jiffyclub.github.io/snakeviz/)
- Py-spy (https://github.com/benfred/py-spy)
- PyInstrument (https://pyinstrument.readthedocs.io/en/stable/guide.html)
There are some other options which involve very little code change.
PyPy
Alternative interpreter and JIT compiler
Best for long-running code
Enable JIT
Available in v3.13+
activate using –enable-experimental-jit
Numba
Works best on code that uses NumPy and code can be dramatically sped up just by adding decorators such as
@jit decorator
@njit(parallel=True)
