{"id":8563,"date":"2022-09-22T11:03:54","date_gmt":"2022-09-22T10:03:54","guid":{"rendered":"https:\/\/www.blopig.com\/blog\/?p=8563"},"modified":"2022-10-04T16:26:41","modified_gmt":"2022-10-04T15:26:41","slug":"how-to-make-your-own-singularity-container-zero-fuss","status":"publish","type":"post","link":"https:\/\/www.blopig.com\/blog\/2022\/09\/how-to-make-your-own-singularity-container-zero-fuss\/","title":{"rendered":"How to make your own singularity container zero fuss!"},"content":{"rendered":"\n

In this blog post, I’ll show you guys how to make your own shiny container for your tool! Zero fuss(*) and in FOUR simple steps.<\/p>\n\n\n\n

As an example, I will show how to make a singularity container for one of our public tools, ANARCI, the antibody numbering tool everyone in OPIG and external users are familiar with – If not, check the web app and the GitHub repo here<\/a> and here<\/a>.<\/p>\n\n\n\n

(*) Provided you have your own Linux machine with sudo<\/code> permissions, otherwise, you can’t do it – sorry. Same if you have a Mac or Windows – sorry again.
BUT, there are workarounds for these cases such as using the remote singularity builder here<\/a>, for which you only need to sign up and create an account, and the use of Virtual Machines (VMs), as described here<\/a>. <\/p>\n\n\n\n\n\n\n\n

A brief preamble<\/h1>\n\n\n\n

If you already know what singularity and containerisation are, you can skip this section and move on to the actual fun!<\/p>\n\n\n\n

In previous blog posts, other OPIGlets already introduced singularity<\/em> and illustrated the security hazards posed by Docker<\/em> (another popular containerisation technology) in shared systems, like HPC, which singularity prevents [here<\/a>]. Also, explained further the idea of containerisation and highlighted its advantages for scientific reproducibility, to avoid the irritating “It works on my machine<\/em>” scenario. As well as a basic introduction to building your own singularity container [here<\/a>] and how to run singularity in Windows with WSL2 [here<\/a>]. <\/p>\n\n\n\n

In this blog post, I’ll go further on how to build containers using definition files and provide a recipe to equip our container with conda<\/code>, to manage environment variables and dependencies, and illustrate the process by cloning and installing one of our tools available on GitHub. <\/p>\n\n\n\n

STEP 1. Install singularity. <\/h1>\n\n\n\n

Instructions are here<\/a>.<\/p>\n\n\n\n

A word of caution<\/em>: Aim to install any Singularity-CE version > 3.3; CE = Community Edition. Previous versions had bugs resulting in a mount error<\/a> post-installation. If you find any other bugs, have a look at the issue list of the singularity repo here<\/a>. On my laptop, I have SingularityCE 3.9.5v and Go 1.17.7v<\/p>\n\n\n\n

STEP 2. Steal these lines.<\/h1>\n\n\n\n

Copy and paste the block below into a text file, for example, myfile.def<\/code> <\/p>\n\n\n\n

BootStrap: library\nFrom: ubuntu:18.04\n\n%environment\n    # set up all essential environment variables\n    export LC_ALL=C\n    export PATH=\/miniconda3\/bin:$PATH\n    export PYTHONPATH=\/miniconda3\/lib\/python3.9\/:$PYTHONPATH\n    \n    # activate conda environment\n    source activate base;\n    conda activate;\n    \n%post\n    # update and install essential dependencies\n    apt-get -y update\n    apt-get update && apt-get install -y automake build-essential bzip2 wget git default-jre unzip\n    apt-get install muscle # ANARCI dependency\n    \n    # download, install, and update miniconda3\n    wget https:\/\/repo.continuum.io\/miniconda\/Miniconda3-latest-Linux-x86_64.sh\n    bash Miniconda3-latest-Linux-x86_64.sh -b -f -p \/miniconda3\/\n    rm Miniconda3-latest-Linux-x86_64.sh\n    \n    # install dependencies via conda\n    export PATH=\"\/miniconda3\/bin:$PATH\"\n    conda install -y -c conda-forge pip numpy # general dependencies\n    conda install -y -c bioconda biopython hmmer # ANARCI dependencies\n    conda update --all\n    \n    # clone and install your github code\n    mkdir mycode\n    git clone https:\/\/github.com\/oxpig\/ANARCI.git mycode\/ANARCI\n    cd mycode\/ANARCI\n    pip install .\n\n%labels\n    Author Me\n    Version v0.0\n    MyLabel Hello World<\/code><\/pre>\n\n\n\n
What do these lines mean?<\/em>
In singularity jargon, this is called a definition file<\/em>, which is a recipe for singularity to build your singularity image file, aka .sif<\/code>, which is a highly compressed file that will allow you to run your tool in the command line. <\/p>\n\n\n\n
As you can see from the definition file, it tells singularity to build a system with an Ubuntu 18.04<\/code>  kernel pulled from its public library<\/code> or file repository, and to define essential system environment<\/code> variables. Also, a set of operations post<\/code> or after pulling the kernel that will be necessary to configure and update the container’s inner system, as well as the tools you want your system to have along with their dependencies; conda<\/code> and ANARCI<\/code> in this example. <\/p>\n\n\n\n
The definition file can be further customised, here I just show some basic sections of it for the purposes of this example. Have a look here<\/a> for what else you can add to your definition file. <\/p>\n\n\n\n
STEP 3. Build your own sh*t<\/h1>\n\n\n\n
Run the following command line in terminal<\/p>\n\n\n\n
sudo singularity build mytool.sif myfile.def<\/code><\/p>\n\n\n\n
Again, you need sudo<\/code> rights to be able to build your output container file. But, as I mentioned above, you can try out the remote builder<\/a> to go around it. <\/p>\n\n\n\n
An important point<\/em>: probably building directly the .sif<\/code> file is not the best practice. The problem is that once built, you can’t modify anything inside unless you modify the definition file. This is irritating if you want to modify or debug a tool built into your container. As a workaround, you can create instead a sandbox<\/code><\/p>\n\n\n\n
sudo singularity build --sandbox mytool myfile.def<\/code><\/p>\n\n\n\n
This will output a folder, mytool<\/code>,  containing all the system files that go into your container content which you can actually see and modify at will. For instance, in shell mode (sudo singularity --writable shell mytool\/<\/code>), you can manually install and modify files or update dependencies with conda too. And once you’re happy with your modifications, then you can create a .sif<\/code> file, which actually is much lighter than a sandbox folder for sharing. <\/p>\n\n\n\n
sudo singularity build mytool.sif mytool<\/code>\/<\/p>\n\n\n\n
For more on this, look here<\/a> and this<\/a> previous blog post.<\/p>\n\n\n\n
STEP 4. Have fun!<\/h1>\n\n\n\n
If you made it to this last step without stumbling across any errors, then, you’re nearly there, cause now you get to play with your tool!<\/p>\n\n\n\n
For ANARCI, we’ll execute its command line tool to number an input sequence by using the exec<\/code> command of singularity upon calling your container<\/p>\n\n\n\n
singularity exec mytool.sif ANARCI -i EVQLQQSGAEVVRSGASVKLSCTASGFNIKDYYIHWVKQRPEKGLEWIGWIDPEIGDTEYVPKFQGKATMTADTSSNTAYLQLSSLTSEDTAVYYCNAGHDYDRGRFPYWGQGTLVTVSA<\/code><\/p>\n\n\n\n
If everything goes well you should see an output like in the screenshot <\/p>\n\n\n\n
\"\"<\/a><\/figure><\/div>\n\n\n\n
If you got an output like this, then pat yourself on the back and congratulate yourself. Well done! \ud83c\udf89<\/p>\n\n\n\n
Now, you can just share mytool.sif<\/code> with your friends and they should be able to run your tool with singularity too!<\/p>\n\n\n\n
The bottom line.<\/h1>\n\n\n\n
Containerisation is an amazing technology that lowers the reproducibility barrier in scientific computing by providing a self-contained environment to pack your tools and dependencies to make them ready to use.<\/p>\n\n\n\n
Here we showed how to use singularity to containerise ANARCI<\/code> in addition to the use  miniconda<\/code> to equip our container with an environment manager to handle dependencies. Now, you can just modify the definition file to pull and install your favourite tool.<\/p>\n\n\n\n
Certainly, this is just the tip of the tip the iceberg. I will probably post more about this topic in the future. In the meantime, have fun and if you have any questions I’d love to hear from you. <\/p>\n","protected":false},"excerpt":{"rendered":"
In this blog post, I’ll show you guys how to make your own shiny container for your tool! Zero fuss(*) and in FOUR simple steps. As an example, I will show how to make a singularity container for one of our public tools, ANARCI, the antibody numbering tool everyone in OPIG and external users are […]<\/p>\n","protected":false},"author":100,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"nf_dc_page":"","wikipediapreview_detectlinks":true,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"ngg_post_thumbnail":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[29,14],"tags":[24,388,198,386,649,648],"ppma_author":[631],"class_list":["post-8563","post","type-post","status-publish","format-standard","hentry","category-code","category-howto","tag-bioinformatics","tag-docker","tag-reproducibility","tag-singularity","tag-software-services","tag-technical"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"authors":[{"term_id":631,"user_id":100,"is_guest":0,"slug":"broncio","display_name":"Broncio Aguilar-Sanjuan","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/98ed21f7b05613d3b21c650ec13af1cafba2e7d8c40a23dba86b9a8b623bd3ef?s=96&d=mm&r=g","0":null,"1":"","2":"","3":"","4":"","5":"","6":"","7":"","8":""}],"_links":{"self":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/8563","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/users\/100"}],"replies":[{"embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/comments?post=8563"}],"version-history":[{"count":5,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/8563\/revisions"}],"predecessor-version":[{"id":8608,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/8563\/revisions\/8608"}],"wp:attachment":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/media?parent=8563"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/categories?post=8563"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/tags?post=8563"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=8563"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}