{"id":9459,"date":"2023-03-28T11:25:09","date_gmt":"2023-03-28T10:25:09","guid":{"rendered":"https:\/\/www.blopig.com\/blog\/?p=9459"},"modified":"2025-10-03T23:07:02","modified_gmt":"2025-10-03T22:07:02","slug":"the-ultimate-modulefile-for-conda","status":"publish","type":"post","link":"https:\/\/www.blopig.com\/blog\/2023\/03\/the-ultimate-modulefile-for-conda\/","title":{"rendered":"The ultimate modulefile for conda"},"content":{"rendered":"\n<p><a href=\"https:\/\/modules.readthedocs.io\/en\/latest\/\" data-type=\"URL\" data-id=\"https:\/\/modules.readthedocs.io\/en\/latest\/\">Environment modules<\/a> is a great tool for high-performance computing as it is a modular system to quickly and painlessly enable preset configurations of environment variables, for example a user may be provided with modulefile for an antiquated version of a tool and a bleeding-edge alpha version of that  same tool and they can easily load whichever they wish. In many clusters the modules are created with a tool called <a href=\"https:\/\/easybuild.io\/\">EasyBuild<\/a>, which delivered an out-of-the-box installation. This works for things like a single binary, but for conda this severely falls short as there are many many configuration changes needed.<\/p>\n\n\n\n<!--more-->\n\n\n\n<h2 class=\"wp-block-heading\">Activating conda<\/h2>\n\n\n\n<p>Conda is a curious fish to start with. It is not distributed in Linux package managers. It does have a licence that needs accepting (automatically accepted with the -b flag), but so do many linux packages. Out of the box it needs activating. If installed without the -b flag (batch) or conda init is run, a bunch of messy bash commands get appended to .bashrc, primarily a bunch of failsafes to the command.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>eval $(\"$CONDA_PREFIX\/bin\/conda shell.bash hook 2&gt; \/dev\/null\")<\/code><\/pre>\n\n\n\n<p>Namely, conda binary is run (<code>$(...)<\/code>) with the arguments bash and hook, wherein its error messages get sent to the null bucket, while the output is a shell script, which gets evaluated.<br>In bash running the command <code>source<\/code> (or <code>.\/<\/code> or <code>eval<\/code>) and <code>bash<\/code> (or <code>exec<\/code> kind of) can have different effects: source runs the shell script using the same shell while bash run it in a different one. The environment initialisation needs to be sourced therefore \u2014an obvious but important detail.<\/p>\n\n\n\n<p>There are three ways to initialise conda.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Allowing the messy snippet to be added to <code>$HOME\/.bashrc<\/code> and sourcing it, which happens on logging in. I.e. one is sourcing\/evaluating the output of <code>conda bash hook<\/code><\/li>\n\n\n\n<li>sourcing the <code>$CONDA_PREFIX\/etc\/profile.d\/conda.sh<\/code> script, which is basically the same evaluation. Personally, I prefer this option.<\/li>\n\n\n\n<li>Adding the variables manually<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Conda variables<\/h2>\n\n\n\n<p>Once this is done one can activate the base environment, via <code>conda activate<\/code> or <code>conda activate base<\/code>,  or a virtual environment, via <code>conda activate ENVNAME<\/code>. Double misleadingly, <code>conda activate<\/code> without <code>base<\/code> when run on an already activated base environment will fail telling you that the environment variables are missing, which is a lie.<\/p>\n\n\n\n<p>Various environment variables get set in doing so. <code>printenv<\/code> prints your environment variables, which makes it really handy.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The key one is the adding to PATH the bin folder of the conda folder. This can be tampered with outside of conda. Files prepended at the front get priority, appended to the back are the last resort.<\/li>\n\n\n\n<li>CONDA_ROOT \u2014the folder where conda lives. say <code>$HOME\/.conda<\/code><\/li>\n\n\n\n<li>CONDA_PREFIX \u2014the folder of the current environment, for base <code>$CONDA_ROOT<\/code> = <code>$CONDA_PREFIX<\/code><\/li>\n\n\n\n<li>CONDA_EXE \u2014 <code>$CONDA_PREFIX\/bin\/conda<\/code><\/li>\n\n\n\n<li>CONDA_PYTHON_EXE \u2014 <code>$CONDA_PREFIX\/bin\/python<\/code> <\/li>\n\n\n\n<li>PKG_CONFIG_PATH \u2014this is a system package alternative path, nothing do with python packages. But you might have a <code>$CONDA_PREFIX\/lib\/pkgconfig<\/code> folder<\/li>\n\n\n\n<li>CONDA_SHLVL \u2014conda shell level. 1 is base.<\/li>\n\n\n\n<li>CONDA_PROMPT_MODIFIER \u2014 the text that gets prepended to <code>$PS1<\/code>, which is the text that appears before your cursor. In my .bashrc I have <code>export PS1=\"[\\u@\\h \\W]\\$\"<\/code>, which makes my prompt remind me of my username (<code>$USER<\/code>) and the hostname (<code>$HOST<\/code>) and my working directory (<code>$PWD<\/code>).<\/li>\n\n\n\n<li>CONDA_DEFAULT_ENV \u2014your environment name\n<ul class=\"wp-block-list\">\n<li>There are many possible conda environment variables as any (on paper) config in a <code>.condarc<\/code> file can be used as an environment variable by going uppercase and underscored. For examle <code>$CONDA_SOLVER<\/code>, <code>$CONDA_CHANNELS<\/code>, <code>$CONDA_YES<\/code>, <code>$CONDARC<\/code> and <code>$CONDA_ENVS_PATH<\/code> etc.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p>A big caveat needs raising regarding the last one. Always check. For example, <code>$CONDA_CHANNELS<\/code> does not work in all versions.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Modulefile<\/h2>\n\n\n\n<p>Now that we have gone over how to get conda activated, we need to configure the various environment variables for the module command to use.<\/p>\n\n\n\n<p>A module file is a file that tells the module command how to load it. It is written in TCL. There is generally a panel of <a href=\"https:\/\/modules.readthedocs.io\/en\/stable\/modulefile.html\" data-type=\"URL\" data-id=\"https:\/\/modules.readthedocs.io\/en\/stable\/modulefile.html\">modulefile<\/a> written by the sys-admin of your cluster, but you can add your own by appending the folder of your modulefiles to <code>$MODULEPATH<\/code>, e.g. <code>export MODULEPATH=\"$MODULEPATH:my-path-with-modulefiles\"<\/code>. The main commands to remember are <code>setenv<\/code>, <code>set-alias<\/code> and <code>prepend-path<\/code>, <code>system<\/code> and <code>puts stderr\/stdout<\/code>.<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">#%Module\nproc ModulesHelp { } {}\nmodule-whatis {}\n\nputs stderr \"This is shown to the user on module load and unload\"\n\n# set variables within this code: env variables from shell are called via &lt;code&gt;$env(...)&lt;\/code&gt;\nset root path-where-conda-lives\nset userhome $env(HOME)\nset userconda $env(HOME)\/.conda\n\nconflict conda\n\n# conda deactivate on unload\nif {[module-info command unload]} {system AUTO_ACTIVATE_BASE=false $root\/bin\/conda deactivate}\n\n# add envs (on unload they will be unset or replaced)\nprepend-path\tMANPATH\t\t$root\/man\nprepend-path\tMANPATH\t\t$root\/share\/man\n# See footnote?\n# prepend-path\tPATH\t\t$root\/bin\n# prepend-path\tPATH\t\t$root\/sbin\nprepend-path\tPKG_CONFIG_PATH\t\t$root\/lib\/pkgconfig\n\n# `python install -u` by default\nsetenv PYTHONUSERBASE   $userhome\/.local\n\n# user created envs go here:\nsetenv CONDA_ENVS_PATH\t$userconda\/envs\nsetenv JUPYTER_CONFIG_PATH $userhome\/.jupyter\n\n# base config for jupyter.\nsetenv JUPYTER_CONFIG_DIR $root\/.jupyter\n# CONDA_ENVS_DIR is not the base config\n\n# if not using a $CONDA_ENVS_PATH environment variables can do the job\nsetenv CONDARC\t$userhome\/.conda\nsetenv\tCONDA_SOLVER\tlibmamba\nsetenv CONDA_YES\ttrue\nsetenv CONDA_CHANNELS \"conda-forge nvidia bioconda\"\n# etc.\n\nif {[module-info command load]} {\n    # give hits to the user\n    system touch $env(HOME)\/.condarc\n    system mkdir -p $userconda\/envs\n    system mkdir -p $userhome\/.local\n    system mkdir -p $userhome\/.jupyter\n    # enable\n    puts stdout \"source $root\/etc\/profile.d\/conda.sh ;\"\n}<\/pre>\n\n\n\n<p>The footnote is that one could add to one environment that way. Conda has its own system which allows one even to &#8220;subclass&#8221; one virtual environment into another.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>conda env config vars set PATH=$PATH:\/Users\/matteo\/.conda\/bin:some-other-env-bin<\/code><\/pre>\n\n\n\n<p>In the above the virtual environment will search its own folders, then the \/usr\/local\/bin etc. and lastly the other environment.<br>The big catch to this variable is stored in the conda-meta\/state file in the environment and not an environment specific <code>.condarc<\/code>, which is not a thing as instead <code>conda&nbsp;env&nbsp;export<\/code> output is generated on the fly. This means that the user would need to be made aware of the alteration as there is no way they are going checking, so the modulefile way (along with a <code>puts stderr<\/code> call maybe) is way more clear.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Footnote: Can I borrow this?<\/h2>\n\n\n\n<p>As mentioned, adding to <code>$MODULEPATH<\/code> results in the modulefiles therein to be visible when running <code>module avail<\/code>. This means one can have one&#8217;s personal modulefile collection. One could copy these from other clusters: the headers on the output of module avail tells you the path. In the file there will be written where the binary files are.<br>Then one can share in turn. Except permissions get in the way. To make a folder visible to someone with some with a common group one needs to first change the group ownership to that group <code>chgrp -R COMMONGROUPNAME FILEPATH<\/code> and change group permissions <code>chmod -R g+rX FILEPATH<\/code>. Uppercase X means give x if user has it (equivalent to g=u,g-w basically). To make a folder visible to all <code>chmod -R o+rX FILEPATH<\/code>.<\/p>\n\n\n\n<p> <\/p>\n","protected":false},"excerpt":{"rendered":"<p>Environment modules is a great tool for high-performance computing as it is a modular system to quickly and painlessly enable preset configurations of environment variables, for example a user may be provided with modulefile for an antiquated version of a tool and a bleeding-edge alpha version of that same tool and they can easily load [&hellip;]<\/p>\n","protected":false},"author":102,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"nf_dc_page":"","wikipediapreview_detectlinks":true,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"ngg_post_thumbnail":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[14,227],"tags":[194,152],"ppma_author":[701],"class_list":["post-9459","post","type-post","status-publish","format-standard","hentry","category-howto","category-python-code","tag-conda","tag-python"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"authors":[{"term_id":701,"user_id":102,"is_guest":0,"slug":"nuben","display_name":"Matteo Ferla","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/2185fc527a4ace0863c903d27646996994413c31d80e8e4c175477b836e7715c?s=96&d=mm&r=g","0":null,"1":"","2":"","3":"","4":"","5":"","6":"","7":"","8":""}],"_links":{"self":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/9459","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/users\/102"}],"replies":[{"embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/comments?post=9459"}],"version-history":[{"count":5,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/9459\/revisions"}],"predecessor-version":[{"id":9982,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/9459\/revisions\/9982"}],"wp:attachment":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/media?parent=9459"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/categories?post=9459"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/tags?post=9459"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=9459"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}