\n\n\n\n
As with many surprising properties of high-dimensional spaces, this behaviour has its roots in the curse of dimensionality<\/a>—i.e. the fact that higher-dimensional spaces have exponentially more volume than lower-dimensional ones. For instance, a cube of width 2 will be 23<\/sup>=8 times as large as a unit cube in three dimensions, but 210<\/sup>=1024 times as large in ten dimensions.<\/p>\n\n\n This exponential increase in volume means that, while the density of a standard Normal distribution is still maximal at the origin, most of it is not actually concentrated around its mean—as our low-dimensional bell-curve-intuition would suggest. Instead, as the dimensionality N increases, it becomes increasingly indistinguishable from a uniform distribution on a spherical shell<\/a> of constant width and radius \u221aN.<\/p>\n\n\n\n We can easily verify this empirically by converting i.i.d. Normal samples of increasing dimensionality into spherical coordinates and comparing their radii, yielding the following histograms.<\/p>\n\n\n\n Conversely, this means that the cumulative probability density in \u03bc\u00b12\u03c3—which, as a useful rule of thumb, amounts to around 95% in lower dimensions—decays rapidly as well.<\/p>\n\n\n\n While such surprising and counterintuitive properties can be a lot of fun to think about from a theoretical standpoint, I also found them extremely helpful when implementing, debugging and testing models that make heavy use of high-dimensional multivariate Normals—of which there are, as mentioned, quite a few.<\/p>\n\n\n\n <\/p>\n","protected":false},"excerpt":{"rendered":" Multivariate Normal distributions are an essential component of virtually any modern deep learning method—be it to initialise the weights and biases of a neural network, perform variational inference in a probabilistic model, or provide a tractable noise distribution for generative modelling. What most of us (including—until very recently—me) aren’t aware of, however, is that these […]<\/p>\n","protected":false},"author":86,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"nf_dc_page":"","wikipediapreview_detectlinks":true,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"ngg_post_thumbnail":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[633,189,278],"tags":[],"ppma_author":[616],"class_list":["post-8600","post","type-post","status-publish","format-standard","hentry","category-ai","category-machine-learning","category-statistics"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"authors":[{"term_id":616,"user_id":86,"is_guest":0,"slug":"leo","display_name":"Leo Klarner","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/8a288902cdb15c98aa887d33d06a4061fa3ebe87388f89f76734cf2be40ec362?s=96&d=mm&r=g","0":null,"1":"","2":"","3":"","4":"","5":"","6":"","7":"","8":""}],"_links":{"self":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/8600","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/users\/86"}],"replies":[{"embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/comments?post=8600"}],"version-history":[{"count":5,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/8600\/revisions"}],"predecessor-version":[{"id":10394,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/8600\/revisions\/10394"}],"wp:attachment":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/media?parent=8600"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/categories?post=8600"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/tags?post=8600"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=8600"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![\"\"](\"https:\/\/i0.wp.com\/www.blopig.com\/blog\/wp-content\/uploads\/2023\/09\/image.png?resize=625%2C269&ssl=1\")
<\/a><\/figure>\n<\/div>\n\n\n![\"\"](\"https:\/\/i0.wp.com\/www.blopig.com\/blog\/wp-content\/uploads\/2023\/09\/image-1.png?resize=625%2C348&ssl=1\")
<\/a><\/figure>\n\n\n\n
\n\n\n\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\n\nnum_samples = 100_000\ndims = [1, 2, 10, 20]\n\n# generate samples\nrng = np.random.default_rng()\nsamples = {d: rng.standard_normal((num_samples, d)) for d in dims}\n\n# get radius\nsamples = {d: np.linalg.norm(samples[d], axis=-1) for d in dims}\n\n# convert to pd.DataFrame and plot\nsamples_df = pd.DataFrame(samples).melt(var_name=\"dim\", value_name=\"r\")\ng = sns.FacetGrid(data=samples_df, col=\"dim\")\ng.map_dataframe(sns.histplot, x=\"r\", element=\"step\")\n\nfor i, d in enumerate(dims):\n ax = g.axes[0][i]\n ax.axvline(np.sqrt(d), c=\"r\")<\/pre>\n\n\n\n
\n\n\n\n![\"\"](\"https:\/\/i0.wp.com\/www.blopig.com\/blog\/wp-content\/uploads\/2023\/09\/Untitled.png?resize=625%2C153&ssl=1\")
<\/a><\/figure>\n\n\n\nimport numpy as np\nfrom scipy.stats import multivariate_normal\n\ndims = [1, 2, 10, 20]\n\nfor d in dims:\n mvn = multivariate_normal(mean=np.zeros(d), cov=np.eye(d))\n p = mvn.cdf(2 * np.ones(d)) - mvn.cdf(-2 * np.ones(d))\n print(f\"Dimension: {d}\\tdensity in \u03bc\u00b1\u03c3: {p:.3f}\")\n\nOutput:\nDimension: 1 density in \u03bc\u00b1\u03c3: 0.954\nDimension: 2 density in \u03bc\u00b1\u03c3: 0.954\nDimension: 10 density in \u03bc\u00b1\u03c3: 0.794\nDimension: 20 density in \u03bc\u00b1\u03c3: 0.631<\/pre>\n\n\n\n
\n\n\n\n