{"id":9935,"date":"2023-06-06T16:26:32","date_gmt":"2023-06-06T15:26:32","guid":{"rendered":"https:\/\/www.blopig.com\/blog\/?p=9935"},"modified":"2023-06-22T19:38:03","modified_gmt":"2023-06-22T18:38:03","slug":"machine-learning-strategies-to-overcome-limited-data-availability","status":"publish","type":"post","link":"https:\/\/www.blopig.com\/blog\/2023\/06\/machine-learning-strategies-to-overcome-limited-data-availability\/","title":{"rendered":"Machine learning strategies to overcome limited data availability"},"content":{"rendered":"\n

Machine learning (ML) for biological\/biomedical applications is very challenging \u2013 in large part due to limitations in publicly available data (something we recently published about [1]<\/a>). Substantial amounts of time and resources may be required to generate the types of data (eg protein structures, protein-protein binding affinity, microscopy images, gene expression values) required to train ML models, however.<\/p>\n\n\n\n

In cases where there is sufficient data available to provide signal, but not enough for the desired performance, ML strategies can be employed:<\/p>\n\n\n\n\n\n\n\n

1. Pre-training and transfer learning<\/strong><\/p>\n\n\n\n

One solution to limited data availability is to turn to a different, related problem with more data. Pre-training on a data-rich problem can be followed by transfer learning on the desired task (by initializing the weights from the pre-trained model) (Figure 1). The underlying concept is that the pre-trained model will learn the fundamentals of the system \u2013\u00a0for example, the physics governing protein-protein interactions \u2013\u00a0and the pre-trained weights will be closer to the optimum for the desired task. As such, less data would be required to calibrate the weights for the desired task.<\/p>\n\n\n\n

\"\"<\/a>
Figure 1. Schematic of transfer learning.<\/figcaption><\/figure>\n\n\n\n

There are many considerations for pre-training. The initial pre-training task can be supervised or unsupervised, although the latter is common due to the limitations in supervised (labelled) data availability. The model parameters \u2013 such as learning rate, learning rate scheduler, weight decay, optimizer, etc.\u00a0\u2013 for transfer learning can also be adjusted. Additionally, the weights for some layers can be frozen (left unchanged during further training), and layers can be added\/removed.<\/p>\n\n\n\n

A note\u00a0about terminology \u2013\u00a0for tasks that are in the same dataset domain (have the same data type\/format) as the original pre-training task, updating the model weights is typically referred to as fine-tuning (e.g. pre-training on general protein data followed by fine-tuning on antibody data).<\/p>\n\n\n\n

Examples where pre-training followed by transfer learning\/fine-tuning has been applied include:<\/p>\n\n\n\n