Author Archives: Jack Scantlebury

The Most ReLU-iable Activation Function?

The Rectified Linear Unit (ReLU) activation function was first used in 1975, but its use exploded when it was used by Nair & Hinton in their 2010 paper on Restricted Boltzmann Machines. ReLU and its derivative are fast to compute, and it has dominated deep neural networks for years. The main problem with the activation function is the so-called dead ReLU problem, where significant negative input to a neuron can cause its gradient to always be zero. To rectify this (har har), modified versions have proposed, including leaky ReLU, GeLU and SiLU, wherein the gradient for x < 0 is not always zero.

A 2020 paper by Naizat et al., which builds upon ideas set out in a 2014 Google Brain blog post seeks to explain why ReLU and its variants seem to be better in general for classification problems than sigmoidal functions such as tanh and sigmoid.

Continue reading →

Musings on Digital Nomaddery from Seoul

The languorous, muggy heat of the Korean afternoon sun was what greeted me after 13 hour cattle-class flight from a cool, sensible Helsinki night. The goings-on in Ukraine, and associated political turmoil, meant taking the scenic route – avoiding Russia and instead passing over Turkey, Kazakstan and Mongolia – with legs contorted into unnatural positions and sleep an unattainable dream. Tired and disoriented, I relied less on Anna’s expert knowledge of the Korean language than her patience for my jet-lag-induced bad mood and brain fog. We waited an hour for a bus to take us from Incheon airport to Yongsan central station in the heart of the capital. It was 35 °C.

I’ve been here for a month. Anna has found work, starting in November; I have found the need to modify my working habits. Gone are the comfortable, temperate offices on St Giles’, replaced by an ever-changing diorama of cafés, hotel rooms and libraries. Lugging around my enormous HP Pavilion, known affectionately by some as ‘The Dominator’, proved to be unsustainable.

It’s thesis-writing time for me, so any programming I do is just tinkering and tweaking and fixing the litany of bugs that Lucy Vost has so diligently exposed. I had planned to run Ubuntu on Parallels using my MacBook Air; I discovered to my dismay that a multitude of Conda packages, including PyTorch, are not supported on Apple’s M1 chip. It has been replaced by a combination of Anna’s old Intel MacBook Pro and rewriting my codebase to install and run without a GPU – adversity is the great innovator, as the saying goes.

Continue reading →

How to prepare a molecule for RDKit

RDKit is very fussy when it comes to inputs in SDF format. Using the SDMolSupplier, we get a significant rate of failure even on curated datasets such as the PDBBind refined set. Pymol has no such scruples, and with that, I present a function which has proved invaluable to me over the course of my DPhil. For reasons I have never bothered to explore, using pymol to convert from sdf, into mol2 and back to sdf format again (adding in missing hydrogens along the way) will almost always make a molecule safe to import using RDKit:

from pathlib import Path
from pymol import cmd

def py_mollify(sdf, overwrite=False):
    """Use pymol to sanitise an SDF file for use in RDKit.

    Arguments:
        sdf: location of faulty sdf file
        overwrite: whether or not to overwrite the original sdf. If False,
            a new file will be written in the form <sdf_fname>_pymol.sdf
            
    Returns:
        Original sdf filename if overwrite == False, else the filename of the
        sanitised output.
    """
    sdf = Path(sdf).expanduser().resolve()
    mol2_fname = str(sdf).replace('.sdf', '_pymol.mol2')
    new_sdf_fname = sdf if overwrite else str(sdf).replace('.sdf', '_pymol.sdf')
    cmd.load(str(sdf))
    cmd.h_add('all')
    cmd.save(mol2_fname)
    cmd.reinitialize()
    cmd.load(mol2_fname)
    cmd.save(str(new_sdf_fname))
    return new_sdf_fname

Making pwd redundant

I’m going to keep this one brief, because I am mid-confirmation-and-paper-writing madness. I have seen too many people – both beginners and seasoned veterans – wandering around their Linux filesystem blindfolded:

Isn’t it hideous?

Whenever you want to see where you are, you have to execute pwd (present working directory), which will print your absolute location to stdout. If you have many terminals open at the same time, it is easy to lose track of where you are, and every other command becomes pwd; surely, I hear you cry, there has to be a better way!

Well, fear not! With a little tinkering with ~/.bashrc, we can display the working directory as part of the special PS1 environment variable, responsible for how your username and computer are displayed above. Putting the following at the top of ~/.bashrc

me=`id | awk -F\( '{print $2}' | awk -F\) '{print $1}'`
export PS1="`uname -n |  /bin/sed 's/\..*//'`{$me}:\$PWD$ "

… saving, and starting a new termanal window results in:

Much better!

I haven’t used pwd in 3 years.

Uniformly sampled 3D rotation matrices

It’s not as simple as you’d think.

If you want to skip the small talk, the code is at the bottom. Sampling 2D rotations uniformly is simple: rotate by an angle from the uniform distribution \theta \sim U(0, 2\pi). Extending this idea to 3D rotations, we could sample each of the three Euler angles from the same uniform distribution \phi, \theta, \psi \sim U(0, 2\pi). This, however, gives more probability density to transformations which are clustered towards the poles:

Sampling Euler angles uniformly does not give an even distribution across the sphere.

In Fast Random Rotation Matrices (James Avro, 1992), a method for uniform random 3D rotation matrices is outlined, the main steps being:

Continue reading →

C++ python bindings in 5 minutes

You don’t even need to use CMake!

Most of the time, we can use libraries like numpy (which is largely written in C) to speed up our calculations, which works when we are dealing with matrices or vectors – but sometimes loops are unavoidable. In those instances, it would be nice if we could use a compiled language such as C++ to remove the bottleneck.

This can be achieved extremely easily using pybind11, which enables us to export C++ functions and classes as importable python objects. We can do all of this very easily, without using CMake, using pybind11’s Pybind11Extension class, along with a modified setup.py. Pybind11 can be compiled from source or installed using:

pip install pybind11
Continue reading →

Uploading/downloading small files across systems

Sometimes you just want to quickly move a copy of a script, image or binary from, for example, your local (linux) machine to another (linux) machine. The usual tool would be SCP, but this can get complicated when there are several layers of ssh and sometimes it doesn’t work at all (as is the case for transfers between the Department of Statistics computers and the outside world).

Continue reading →

Green politics, the left, and Brexit

For my first non-technical blogpost, I thought I’d go in for something that we can all agree on and is entirely devoid of controversy: Brexit. Is that growning I hear from the back of the room?

One of my uncles is a professor of sociology; he returned to the UK for the first time in 10 years over Christmas 2019, and naturally we had plenty to talk about. He had left with two kids when I was a lanky, goofy teenager and had returned with four to a lanky, goofy adult. What were most interesting, though, were his views on green politics and their relationship with the traditional left-right spectrum.

Continue reading →