{"id":9952,"date":"2023-07-11T16:19:36","date_gmt":"2023-07-11T15:19:36","guid":{"rendered":"https:\/\/www.blopig.com\/blog\/?p=9952"},"modified":"2023-08-08T16:22:59","modified_gmt":"2023-08-08T15:22:59","slug":"phinally-phunctionalising-my-phigures-with-phate-feat-plotly-express","status":"publish","type":"post","link":"https:\/\/www.blopig.com\/blog\/2023\/07\/phinally-phunctionalising-my-phigures-with-phate-feat-plotly-express\/","title":{"rendered":"PHinally PHunctionalising my PHigures with PHATE feat. Plotly Express."},"content":{"rendered":"\n

After being recommended by a friend, I really wanted to try plotly express but I never had the inclination to read more documentation when matplotlib gives me enough grief. While experimenting with ChatGPT I finally decided to functionalise my figure making scripts. With these scripts I manage to produce figures that made people question what I had actually been doing with my time – but I promise this will be worth your time.<\/p>\n\n\n\n

I have been using with dimensionality reducition techniques recently and I came across this paper<\/a> by Moon et al. PHATE is a technique that represents high dimensional (ie biological) data in a way that aims to preserve connections over preserving distance and I knew I wanted to try this as soon as I saw it. Why should you care? PHATE in 3D is faster that t-SNE in 2D. It would almost be rude to not try it out.<\/p>\n\n\n\n

PHATE<\/h2>\n\n\n\n

In my opinion PHATE (or potential of heat diffusion for affinity-based transition embedding) does have a lot going on but that the choices at each stage feel quite sensisble. It might not come as a surprise this was primarily designed to make visual inspection of data easier on the eyes.<\/p>\n\n\n\n\n\n\n\n

\"\"\/<\/figure>\n\n\n\n

As mentioned PHATE is way faster than t-SNE, PHATE 2D is about 40% faster than t-SNE 2D and PHATE 3D is about 10% faster than t-SNE 2D. I actually gave up on using t-SNE 3D because it was so slow. <\/p>\n\n\n\n

You can use it very similarly to t-SNE to generate a PHATE operator object:
<\/p>\n\n\n\n

from sklearn.manifold import TSNE\nimport phate\nimport time\n\n# Read in dataframe\nglobal_df = pd.read_parquet(model_name+\"_global_df.parquet\")\n\n# Features for colour mapping\ncolour_indexes = [\"Role\", \"CLASS\", \"DNN_Logits\", \"GAN_Logits\", \"Logit_Diff\" ,\"DNN_Distance\", \"GAN_Distance\"]\n\n# Create plotting dataframe\nplotting_df = pd.DataFrame()\nplotting_df[\"FP\"] = global_df[\"FP\"]\nplotting_df[\"ROLE\"] = global_df[\"ROLE\"]\nplotting_df[\"Similarity\"] = global_df[\"Similarity\"]\n\n# Add columns for colour mapping features\nfor index in colour_indexes:\n    plotting_df[index] = global_df[index]\n\n# Define the training features and dimension reducers\nfeatures = [\"FP\",  \"Similarity\", \"DNN_Features\", \"GAN_Features\"]\ndim_reducers = [\"T-SNE_2D\", \"PHATE_2D\", \"PHATE_3D\"]\n\n# Compute TSNE and PHATE for each feature and dimension reducer\nfor feature in features:\n    print(f\"Computing {feature}\")\n    data = np.stack(global_df[feature].to_numpy(), axis=0)\n    print(data.shape)\n    \n    for dim_reducer in dim_reducers[:]:\n        print(f\"Computing {dim_reducer} for {feature}\")\n        \n        if dim_reducer == \"T-SNE_2D\":\n            tsne_operator = TSNE(n_components=2, n_jobs=4, perplexity=50, verbose=1, random_state=42)\n            start_time = time.time()\n            transformed_data = tsne_operator.fit_transform(data)\n            end_time = time.time()\n\n        elif dim_reducer == \"PHATE_2D\":\n            phate_operator = phate.PHATE(n_components=2, n_jobs=4, knn=5, decay=40, random_state=42)\n            start_time = time.time()\n            transformed_data = phate_operator.fit_transform(data)\n            end_time = time.time()\n\n        elif dim_reducer == \"PHATE_3D\":\n            phate_operator = phate.PHATE(n_components=3, n_jobs=4, knn=5, decay=40, random_state=42)\n            start_time = time.time()\n            transformed_data = phate_operator.fit_transform(data)\n            end_time = time.time()\n        \n        # Create a column name combining the method and features used\n        column_name = f\"{dim_reducer}_{feature}\"\n        \n        plotting_df[column_name] = [row for row in transformed_data]\n        print(f\"Execution time for {column_name}: {end_time - start_time} seconds\")\n\nplotting_df.to_parquet(model_name + \"_plotting_df.parquet\")<\/pre>\n\n\n\n
Example of what this might look like:<\/p>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n
With the help of ChatGPT I FINALLY decided to functionalise my plots instead of just having one long script:<\/p>\n\n\n\n
def plot_scatter(ax, dataframe, cname, cmap, method, feature):\n    \"\"\"\n    Plot a scatter plot on the given axes.\n\n    Args:\n        ax (matplotlib.axes.Axes): The axes to plot on.\n        data (np.ndarray): The data to be plotted.\n        c: The color data for the scatter plot.\n        cmap: The colormap for the scatter plot.\n\n    Returns:\n        None\n    \"\"\"\n    c = dataframe[cname]\n\n    column_name = f\"{method}_{feature}\"\n\n    data = np.stack(dataframe[column_name].to_numpy(), axis=0)\n\n    scatter = ax.scatter(data[:, 0], data[:, 1], c=c, cmap=cmap, s=3, alpha=0.5)\n    ax.set_xlabel(f\"{method} 1\")\n    ax.set_ylabel(f\"{method} 2\")\n    ax.set_title(f\"{method} plot on {feature} by {cname}\")\n    ax.legend()\n\n    # Add colorbar\n    cbar = plt.colorbar(scatter, ax=ax)\n    cbar.set_label(cname)\n\n\ndef plot_scatter_grid(dataframe, cnames, method, feature):\n    \"\"\"\n    Plot a grid of scatter plots.\n\n    Args:\n        dataframe (pd.DataFrame): The data to be plotted.\n        cnames (list): List of column names for the scatter plots.\n        method (str): The dimension reduction method.\n        feature (str): The feature to be plotted.\n\n    Returns:\n        None\n    \"\"\"\n    colours = ['green', 'orange', 'purple', 'pink']\n\n    fig, axs = plt.subplots(len(cnames), len([method]), figsize=(6*len([method]), 4*len(cnames)))\n\n    for i, cname in enumerate(cnames):\n        ax = axs[i]\n        if cname == \"Role\":\n            cmap = matplotlib.colors.ListedColormap(colours)\n        else:\n            cmap = 'viridis'\n        plot_scatter(ax, dataframe, cname, cmap, method, feature)\n\n    plt.tight_layout()\n    plt.show()\n\ufeff<\/pre>\n\n\n\n
Usage (to create the above plot):<\/p>\n\n\n\n
colour_indexes = [\"Role\", \"CLASS\", \"DNN_Logits\", \"GAN_Logits\", \"Logit_Diff\" ,\"DNN_Distance\", \"GAN_Distance\" ]\nfeatures = [\"FP\", \"Similarity\", \"DNN_Features\", \"GAN_Features\"]\ndim_reducers = [\"T-SNE_2D\", \"PHATE_2D\", \"PHATE_3D\"]\n\n# Define the parameters\ncnames = colour_indexes[:]\ncnames = [\"Role\", \"CLASS\", \"DNN_Logits\", \"DNN_Distance\"]\nmethod = dim_reducers[1]\nfeature = features[2]\n\n# Call the function to plot the scatter grid\nplot_scatter_grid(plotting_df, cnames[2:4], method, feature)\n\ufeff<\/pre>\n\n\n\n
PHATE is way faster to compute even in 3 dimensions. But matplotlb’s 3D plotting is a little fiddly – having to set the viewing angles can make visualisation slow, especially when plotting large datasets:<\/p>\n\n\n\n
def plot_scatter_3D(ax, dataframe, cname, cmap, method, feature, elev, azim):\n    \"\"\"\n    Plot a scatter plot in 3D on the given axes.\n\n    Args:\n        ax (mpl_toolkits.mplot3d.axes3d.Axes3D): The axes to plot on.\n        dataframe (pd.DataFrame): The data to be plotted.\n        cname (str): The column name for color data.\n        cmap: The colormap for the scatter plot.\n        method (str): The method used for the plot.\n        feature (str): The feature used for the plot.\n        elev (float): The elevation angle for the viewing perspective.\n        azim (float): The azimuth angle for the viewing perspective.\n\n    Returns:\n        None\n    \"\"\"\n    c = dataframe[cname]\n    column_name = f\"{method}_{feature}\"\n    data = np.stack(dataframe[column_name].to_numpy(), axis=0)\n\n    scatter = ax.scatter(data[:, 0], data[:, 1], data[:, 2], c=c, cmap=cmap, s=3, alpha=0.5)\n    ax.set_xlabel(f\"{method} 1\")\n    ax.set_ylabel(f\"{method} 2\")\n    ax.set_zlabel(f\"{method} 3\")\n    ax.set_title(f\"{method} plot on {feature} by {cname}\")\n    ax.legend()\n\n    # Add colorbar\n    cbar = plt.colorbar(scatter, ax=ax)\n    cbar.set_label(cname)\n\n    # Set the view angle\n    ax.view_init(elev=elev, azim=azim)\n\ufeff<\/pre>\n\n\n\n
It also looks pretty ugly without some tinkering:<\/p>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n
Plotly Express is a library that is designed for figures that are meant to be viewed on a computer, very interactive and incredibly fast to boot. Here is a simple example of a 3D plot that you can alter the perspective in vscode:<\/p>\n\n\n\n
from plotly.subplots import make_subplots\nimport plotly.graph_objects as go\n\ndef plot_scatter_3d(dataframe, cname, cmap, method, feature):\n    c = dataframe[cname]\n\n    column_name = f\"{method}_{feature}\"\n\n    data = np.stack(dataframe[column_name].to_numpy(), axis=0)\n\n    scatter = go.Scatter3d(\n        x=data[:, 0],\n        y=data[:, 1],\n        z=data[:, 2],\n        mode='markers',\n        marker=dict(\n            size=1,\n            color=c,\n            colorscale=cmap,\n            opacity=0.5\n        ),\n        name=cname\n    )\n\n    fig = go.Figure(data=[scatter])\n    fig.update_layout(\n        title=f\"{method} plot on {feature} by {cname}\",\n        scene=dict(\n            xaxis_title=f\"{method} 1\",\n            yaxis_title=f\"{method} 2\",\n            zaxis_title=f\"{method} 3\"\n        )\n    )\n    fig.show()\n\n\ufeff<\/pre>\n\n\n\n
On the surface, Plotly express can be very unfamiliar for those used to matplotlib. The problem is that often you want to use matplotlib for publication quality figures so why bother writing something in another library if it wont end up in your final work? <\/p>\n\n\n\n
I used ChatGPT to rewrite my plots. Since it was already functionalised I never had to worry about the syntax, this is especially important when experimenting with different ways to display your data. Much faster, way cleaner. <\/p>\n\n\n\n
Here’s how I prompted it:<\/p>\n\n\n\n
\n
\"\"<\/a><\/figure>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n
\"\"<\/a><\/figure>\n<\/figure>\n\n\n\n
I have found ChatGPT to be really effective for code where the syntax and usage is unclear to provide some examples so that you can better understand whats going on.<\/p>\n","protected":false},"excerpt":{"rendered":"
figure making scripts. With these scripts I manage to produce figures that made people question what I had actually been doing with my time – but I promise this will be worth your time.<\/p>\n
I have been using with dimensionality reducition techniques recently and I came across this paper by Moon et al. PHATE is a technique that represents high dimensional (ie biological) data in a way that aims to preserve connections over preserving distance and I knew I wanted to try this as soon as I saw it. Why should you care? PHATE in 3D is faster that t-SNE in 2D. It would almost be rude to not try it out.<\/p>\n","protected":false},"author":115,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"nf_dc_page":"","wikipediapreview_detectlinks":true,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"ngg_post_thumbnail":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[361,621,189,278],"tags":[],"ppma_author":[725],"class_list":["post-9952","post","type-post","status-publish","format-standard","hentry","category-data-science","category-data-visualization","category-machine-learning","category-statistics"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"authors":[{"term_id":725,"user_id":115,"is_guest":0,"slug":"alexi","display_name":"Alexi Hussain","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/aa505acb767aa3aa80da3b68d4684bf5d5064ac6aed2a01206a61383a967535a?s=96&d=mm&r=g","0":null,"1":"","2":"","3":"","4":"","5":"","6":"","7":"","8":""}],"_links":{"self":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/9952","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/users\/115"}],"replies":[{"embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/comments?post=9952"}],"version-history":[{"count":4,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/9952\/revisions"}],"predecessor-version":[{"id":10217,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/9952\/revisions\/10217"}],"wp:attachment":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/media?parent=9952"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/categories?post=9952"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/tags?post=9952"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=9952"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}