Your research group slurmified their servers? You basically have two options now.<\/p>\n\n\n\n
Either you install all your necessary things on one of the slurm nodes within an interactive session, e.g.:<\/p>\n\n\n\n
srun -p funkyserver-debug --pty --nodes=1 --ntasks-per-node=1 -t 00:10:00 --wait=0 \/bin\/bash<\/code><\/pre>\n\n\n\nand always specify this node by adding the \u2018#SBATCH –nodelist=funkyserver.cpu.do.work\u2019 line to your sbatch scripts or you set up some template scripts that will help you to install all your requirements on multiple nodes so you can enjoy the benefits of the slurm system. <\/p>\n\n\n\n
Here is how I did it; comments and suggestions welcome!<\/p>\n\n\n\n
Step 1<\/strong>: Create an sbatch template file (e.g. sbatch_job_on_server.template_sh) on the submission node that does what you want. In the ‘#SBATCH –partition’ or ‘–nodelist’ lines use a placeholder, e.g. \u2018<server>\u2019, instead of funkyserver.\u00a0<\/p>\n\n\n\nFor example, for installing the same conda environment on all nodes that you want to work on:<\/p>\n\n\n\n\n\n\n\n
#!\/bin\/bash \n#SBATCH -J <server>_installer # Job name\n#SBATCH -A opig # Project Account \n#SBATCH --time=00:10:00 # Walltime \n#SBATCH --mem=1G\n#SBATCH --ntasks=1 # 1 tasks \n#SBATCH --cpus-per-task=1 # number of cores per task \n#SBATCH --nodes=1 # number of nodes \n#SBATCH --chdir=\/your\/fav\/localhost\/directory # From where you want the job to be run\n#SBATCH --mail-user=dominik@do.work # set email address \n#SBATCH --verbose\n#SBATCH --partition=<server>-debug # Select a debug partition of server\n#S BATCH --nodelist=<server>.cpu.do.work # commented out, use partition or nodelist \n#SBATCH --output=\/your\/fav\/localhost\/directory\/slurm_outs\/slurm_%j_%N_%x_%A_%a.out # Writes standard output to this file. \n#SBATCH --error=\/your\/fav\/localhost\/directory\/slurm_outs\/slurm_%j_%N_%x_%A_%a.out # Writes error messages to this file. \n\n## install conda env from spec file\n# copy spec file from (nonslurm) submission node\nscp \"nonslurmnode:~\/Documents\/my_env_specs.txt\" .\n\n# first source bashrc (with conda.sh), then conda can be used\nsource ~\/.bashrc\n\n# make sure conda base is activated\nconda activate\n\n# if already present the old env is removed\nconda remove --name my_env --all\n# and the updated one created from scratch\n# (this is necessary if you want to ensure all environments are identical because conda update and conda create --force have a bug)\nconda create --name my_name --file my_env_specs.txt\n<\/code><\/pre>\n\n\n\nStep 2<\/strong>: Create a create_sbatch_from_template.sh that goes through a list of servers and runs a sed command for each, e.g.:<\/p>\n\n\n\n#!\/bin\/bash\n### This script creates the individual sbatch install file for each server\n# 18\/02\/2020 by Dominik\n\n# define server list\nserver_list=(\"funkyserv01\" \"funkyserv02\" \"technoserv01\")\n\n# go through list\nfor server in \"${server_list[@]}\"; do\n # replace placeholders in script and script name\n sed \"s\/<server>\/${server}\/g\" \u201c~\/scripts\/templates\/sbatch_job_on_server.template_sh\" > \"~\/scripts\/sbatch_job_on_${server}.sh\"\ndone\n<\/code><\/pre>\n\n\n\n<\/p>\n\n\n\n
Run it.<\/p>\n\n\n\n
Step 3<\/strong>: Create a script that goes through the server list and submits the sbatch files created by Step 2. (This can also be done within the script of Step 2, directly after creating the sbatch file from the template.)<\/p>\n\n\n\n#!\/bin\/bash\n### This script submits sbatch scripts, one for each slurm server that you want to work on\n# 18\/02\/2020 by Dominik\n\n# define server list\nserver_list=(\"funkyserv01\" \"funkyserv02\" \"technoserv01\")\n\n# submit sbatch for each slurm server\nfor server in \"${server_list[@]}\"; do\n sbatch \"~\/scripts\/sbatch_job_on_${server}.sh\"\ndone<\/code><\/pre>\n\n\n\nRun it.<\/p>\n\n\n\n
Additionally to the programs you need for your research, I find some things very useful to have on multiple servers: e.g. your bashrc, your environments, your ssh token, \u2026<\/p>\n\n\n\n
Miniconda is probably one of the first things you want to have on the nodes. It could be worth installing them on all nodes at the same time as different conda versions might behave differently. I had some nodes on 4.5 and others on 4.7 which caused me some confusion as 4.5 had a bug which prevented .conda files in the spec file to be parsed.<\/p>\n\n\n\n
Also note that the ‘conda remove –name my_env –all’ followed by a creation from scratch is necessary if you want environments that are identical. Conda 4.4+ has a bug so that libraries not specified in the file but present earlier (e.g. through testing something on that node) will not be uninstalled when conda update (work around conda create –force doesn\u2019t work yet, link to github issue missing, apologies).<\/p>\n","protected":false},"excerpt":{"rendered":"
Your research group slurmified their servers? You basically have two options now. Either you install all your necessary things on one of the slurm nodes within an interactive session, e.g.: and always specify this node by adding the \u2018#SBATCH –nodelist=funkyserver.cpu.do.work\u2019 line to your sbatch scripts or you set up some template scripts that will help […]<\/p>\n","protected":false},"author":55,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"nf_dc_page":"","wikipediapreview_detectlinks":true,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"ngg_post_thumbnail":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[29,14],"tags":[292],"ppma_author":[539],"class_list":["post-5520","post","type-post","status-publish","format-standard","hentry","category-code","category-howto","tag-slurm"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"authors":[{"term_id":539,"user_id":55,"is_guest":0,"slug":"dominik","display_name":"Dominik","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/56bd12e029ec48ca4480b0927066663d35abe9d040c80e2ed0ace3f60b41419f?s=96&d=mm&r=g","0":null,"1":"","2":"","3":"","4":"","5":"","6":"","7":"","8":""}],"_links":{"self":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/5520","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/users\/55"}],"replies":[{"embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/comments?post=5520"}],"version-history":[{"count":5,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/5520\/revisions"}],"predecessor-version":[{"id":5526,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/5520\/revisions\/5526"}],"wp:attachment":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/media?parent=5520"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/categories?post=5520"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/tags?post=5520"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=5520"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}