{"id":8964,"date":"2022-12-07T16:34:21","date_gmt":"2022-12-07T16:34:21","guid":{"rendered":"https:\/\/www.blopig.com\/blog\/?p=8964"},"modified":"2022-12-13T16:22:32","modified_gmt":"2022-12-13T16:22:32","slug":"cleaning-outliers-in-conductance-timeseries-from-molecular-dynamics","status":"publish","type":"post","link":"https:\/\/www.blopig.com\/blog\/2022\/12\/cleaning-outliers-in-conductance-timeseries-from-molecular-dynamics\/","title":{"rendered":"Cleaning outliers in conductance timeseries from molecular dynamics"},"content":{"rendered":"\n

Have you ever had an annoying dataset that looks something like this?<\/p>\n\n\n\n

\"\"\/<\/figure>\n\n\n\n

or even worse, just several of them<\/p>\n\n\n\n

\"\"\/<\/figure>\n\n\n\n

In this blog post, I will introduce basic techniques you can use and implement with Python<\/em> to identify and clean outliers. The objective will be to get something more eye-pleasing (and mostly less troublesome for further data analysis) like this<\/p>\n\n\n\n\n\n\n\n

\"\"\/<\/figure>\n\n\n\n

I will look specifically at the case of timeseries of instant conductance values of simulated ion channels (*). And at the end, I will share a code snippet you can steal and adapt to deal with similar time series.<\/p>\n\n\n\n

(*) I will probably write another blog post in the future to expand on my simulations and on how I computed instant conductance for hundreds of simulation trajectories.<\/p>\n\n\n\n

Identifying Outliers<\/strong><\/p>\n\n\n\n

\"\"
If only it were that simple \u2026<\/figcaption><\/figure>\n\n\n\n

Outliers can be defined as data points that significantly differ from other observations.<\/p>\n\n\n\n

In general, removing outliers will depend on the specific characteristics of your data and the desired outcome. I will not elaborate on that here, but this can get really complicated as the topic of timeseries spans several disciplines\u200a\u2014\u200aalso an old one<\/a>. Check this<\/a> reference if you want to get a sense of how deep the rabbit hole goes. <\/p>\n\n\n\n

Here, I will simply focus on a few approaches relevant to the case of the kind of timeseries that one can simply bound within an interval. <\/p>\n\n\n\n

In this case, to spot outliers one can simply plot the data (visual approach)<\/em> and then manually set up threshold values to filter them out. But, because this process gets impractical for hundreds of timeseries, using information drawn from the data distributions gets handy (statistical approach)<\/em>. I will cover both cases.<\/p>\n\n\n\n

Visual approach<\/strong><\/p>\n\n\n\n

Just plot it<\/strong><\/p>\n\n\n\n

Timeseries and scatter plots are useful means to visualise outliers.<\/p>\n\n\n\n

In the time series below, we can see that the bulk of the data lives between 0.5 and 2.0<\/p>\n\n\n\n

\"\"\/<\/figure>\n\n\n\n

To clean our data, we can set these as thresholds, replace outliers with NaN<\/code> values, and fill them in with interpolated data. Using pandas<\/code> it would look something like this<\/p>\n\n\n\n

import pandas as pd\nfrom numpy import NaN\ndf = pd.DataFrame({'original': timeseries})\noutliers = df[(0.5 > df.values) | (df.values > 2.0)].values\ndf_with_NaNs = df.replace(outliers, NaN)\ndf_new = df_with_NaNs.interpolate(method='linear', axis=0).ffill().bfill()<\/pre>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
Box plots<\/strong><\/p>\n\n\n\n
Using a plain plot of our timeseries does the job. However, we can use another graphic representation, a more statistical one: a box plot.<\/p>\n\n\n\n
import seaborn as sns\nsns.boxplot(df['original'],ax=ax)<\/pre>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
Box plots<\/em> graphically represent the anatomy of your data\u2019s distribution. The central box indicates the ranked quartiles<\/em> Q1<\/code> , Q2<\/code> , and Q3<\/code> which represent the 25-percentile, the median, and the 75-percentile of the distribution. While the two whiskers<\/em> indicate the minimum and the maximum of the distribution, with all data points outside these considered outliers<\/em>. For a pretty picture of the parts of a box plot, check this<\/a>.<\/p>\n\n\n\n
Again, from looking at the box plot, we can see that 0.5 and 2.0 as threshold values to filter out outliers are indeed quite sensible choices.<\/p>\n\n\n\n
Statistical Approach<\/strong><\/p>\n\n\n\n
Visualisation is great for quickly spotting outliers. However, if you have to deal with hundreds or even thousands of timeseries, cluttering will limit visualisation to pick suitable thresholds to remove outliers.<\/p>\n\n\n\n
Statistical approaches provide a methodic way to overcome this limitation. Two very well-known approaches I will look at are:<\/p>\n\n\n\n