{"id":10272,"date":"2023-10-09T23:07:18","date_gmt":"2023-10-09T22:07:18","guid":{"rendered":"https:\/\/www.blopig.com\/blog\/?p=10272"},"modified":"2023-10-11T15:35:45","modified_gmt":"2023-10-11T14:35:45","slug":"ssh-the-boss-fight-level-jupyter-notebooks-from-compute-nodes","status":"publish","type":"post","link":"https:\/\/www.blopig.com\/blog\/2023\/10\/ssh-the-boss-fight-level-jupyter-notebooks-from-compute-nodes\/","title":{"rendered":"SSH, the boss-fight level: Jupyter notebooks from compute nodes"},"content":{"rendered":"\n<p>Secure shell (SSH) is an essential tool for remote operations. However, not everything with it is smooth-sailing. Especially, when you want to do things like reverse\u2013port-forwarding via a proxy-hump or two a Jupyter notebook to your local machine from a compute node on a no-home container . Even if it sounds less plausible than the exploits on <em>Mr Robot<\/em>, it actually can work and requires zero social-engineering or sneaking in server rooms to install Raspberry Pis while using a baseball cap as a disguise.<\/p>\n\n\n\n<!--more-->\n\n\n\n<h2 class=\"wp-block-heading\">Disclaimer: Advanced only<\/h2>\n\n\n\n<p>This post addresses particular advance cases and is not intended to be a beginners tutorial. For that please search:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Windows machine? Search: <em>Activate Windows Subsystem Linux (WSL)<\/em>. Don&#8217;t bother with Putty as <em>older<\/em> posts will recommend.<\/li>\n\n\n\n<li>SSH not working? Use argument <em>-v<\/em> on ssh and copy-paste the last command.<\/li>\n\n\n\n<li>Connect without password. Search &#8220;SSH key&#8221; and also with your OS SSH agent.<\/li>\n\n\n\n<li>SSH keys not working? Check you can get it by typing passwords then search: <em>chmod .ssh folder<\/em> as your permissions might be wrong<\/li>\n\n\n\n<li>SSH seems to hang? See below<\/li>\n\n\n\n<li>Unix root folders? The meanings of some Unix root folders seems trial but a lot of confusion stems from these, so see footnote root folders if at all unsure.<\/li>\n\n\n\n<li>What&#8217;s a config file? It&#8217;s <code>$HOME\/.ssh\/config<\/code>.  Search SSH config file<\/li>\n\n\n\n<li>ssh runs on port 22. Different ports are used either for routing or for obfuscation.<\/li>\n\n\n\n<li>Environment variables below are not working: remember double quotes not single to extrapolate environment variable, and env and export have different scopes. Source not bash scripts with variables to make them available (eg. <code>source \/etc\/os-release; echo $PRETTY_NAME;<\/code>).<\/li>\n\n\n\n<li>Most programs are okay with <code>path\/\/path<\/code> but not ssh. So don&#8217;t have environment variables ending in a slash!<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Privacy<\/h3>\n\n\n\n<p>A basic concept worth reiteration is privacy. Always think security:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>If you have SSH keys someone remotely who can see them?<\/li>\n\n\n\n<li>If only the root users, is there anything they cannot already get to?<\/li>\n\n\n\n<li>If on a mounted drive that can it be mounted anywhere and what type of drive is it?<\/li>\n\n\n\n<li>If the sys-admin&#8217;s family were held hostage by &lt;insert villain of current news cycle&gt;, what of yours would they coerce they sys-admin to give?<\/li>\n\n\n\n<li>What steps can you take to encrypt or obfuscate that data? <\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Fail2ban and local ssh settings<\/h3>\n\n\n\n<p>SSH will appear to hang if your are port forwarding or doing an scp\/rsync: this is normal it&#8217;s doing its jump. But there is a second case: you have been jailed.<\/p>\n\n\n\n<p>A common safeguard against attackers is Fail2ban. This Perl script monitors your logs and jails any IP that violates a given rule. When logged into a remote machine as a non-root user you are unable to see the settings, but the <code>maxretry<\/code> is generally 3 with a <code>timeout<\/code> of 600 seconds. If you triggered a jailing for your IP, the <code>ssh -v<\/code> will hang on &#8220;connecting&#8221;.<\/p>\n\n\n\n<p>Parenthetically, Fail2ban is useful beyond SSH: on my webservers, e.g. <a href=\"https:\/\/michelanglo.sgc.ox.ac.uk\/\" data-type=\"link\" data-id=\"https:\/\/michelanglo.sgc.ox.ac.uk\/\">Michelanglo<\/a>, I not only have an SSH jail, but also Apache-log jails for <a href=\"https:\/\/github.com\/projectdiscovery\/nuclei\" data-type=\"link\" data-id=\"https:\/\/github.com\/projectdiscovery\/nuclei\">Nuclei project<\/a> user-agents \u2014yup, happens\u2014 and for excessive 404 brute crawlers \u2014a wee caveat is that as per <em>misread<\/em> EU rules, one should not keep IP Apache logs, but that is utter lunacy par with leaving your ignition key in your car on the dashboard, so don&#8217;t play it legally safe, play it safe against attackers. <\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Proxy Jumping<\/h2>\n\n\n\n<p>Now the fun stuff.<\/p>\n\n\n\n<p>To ssh twice, you proxy jump. The inline command to go to inner-host from a gateway host is <code>ssh -J $GATEWAY_ADDRESS $INNERNODE_ADDRESS<\/code>. In your config file, it&#8217;s the <code>ProxyJump<\/code> option, as shown in the following example, where the <code>emerald-cpu<\/code> host is reachable solely via <code>emerald<\/code> gateway host.  The Emerald city is at the centre of the Land of Oz, it is a fictional place, so don&#8217;t bother hacking it \u2014unless you are related to Judy Garland. For the sake of sanity the option <code>CanonicalizeHostname<\/code> is used on all hosts (first block), which means that it is order-invariant, otherwise the gateway host must come before the internal host.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Host *\n     ControlMaster auto\n     ControlPersist 300\n     ControlPath ~\/.ssh\/control-%h-%p-%r\n     GSSAPIAuthentication yes\n     ServerAliveInterval 30\n     # Mac stuff:\n     AddKeysToAgent yes\n     UseKeychain yes\n     IgnoreUnknown AddKeysToAgent,UseKeychain\n     # Repeat settings:\n     CanonicalizeHostname yes\n\nHost emerald\n     Hostname ssh.emerald.ac.uk\n     User abc12345\n     IdentityFile ~\/.ssh\/emerald\n\nHost emerald-cluster\n     Hostname submitter.emerald.ac.uk\n\nHost *.emerald.ac.uk !ssh.emerald.ac.uk\n     ProxyJump emerald\n     User abc12345\n     IdentityFile ~\/.ssh\/emerald_intra<\/code><\/pre>\n\n\n\n<p>A variant of this to use the <code>ProxyCommand<\/code> option, this is a command that gets executed before to get to the host, unlike RemoteCommand, which gets run once in the host.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ssh \\\n-o ProxyCommand=\"ssh -W %h:%p $GATEWAY_ADDRESS \\\n<code>$INNERNODE_ADDRES<span style=\"background-color: initial;font-size: 0.857143rem\">S;<\/span><\/code><\/code><\/pre>\n\n\n\n<p>Specifically, the <code>-W<\/code> option in <code>ssh<\/code> enables a built-in way to forward a TCP connection over the established SSH connection. SSH does not use bash syntax and has a few tokens that get expanded when the command gets executed: %h is the target host (%L is the local one) and %p is the port. In the above example the environment variables will be expanded by the shell when the command is run, while the tokens will be expanded by SSH.<\/p>\n\n\n\n<p>Whereas <code>ProxyJump<\/code> makes a clean entry in an SSH config file, the <code>ProxyCommand<\/code> allows a single argument-laden command, which is needed in cases where there is no home folder or using the home folder is not viable.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Port Forwarding<\/h2>\n\n\n\n<p>Port forwarding or tunnelling allows you to make a port run via SSH. Say make port 6789 on your local machine a forwarded port 6789 on your remote host. This can go two directions: local port forwarding or remote\/reverse port forwarding.<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"bash\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\"># Local to remote\nssh -N -L localhost:$LOCAL_PORT:localhost:$REMOTE_PORT $HOST_ADDRESS\n# Remote to local\nssh -N -R localhost:$LOCAL_PORT:localhost:$REMOTE_PORT $HOST_ADDRESS\n# With fluff\nssh -N -L localhost:$LOCAL_PORT:localhost:$REMOTE_PORT -p $SSH_PORT $HOST_ADDRESS -o ServerAliveInterval=180 -o ExitOnForwardFailure=yes<\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The -N argument tells SSH that you don&#8217;t want a bash prompt, although it will execute the <code>~\/.bashrc<\/code>. This has a different effect that -t\/-T arguments which are used to run a command.<\/li>\n\n\n\n<li>The -L argument is for local forwarding (i.e. <em>to<\/em> the local host), -R for remote (i.e. <em>to<\/em> the remote host).<\/li>\n\n\n\n<li>The local port (in the variable $LOCAL_PORT above) that gets bound cannot be a privileged port (i.e. below 1024) without sudo \u2014no port 666, sorry.<\/li>\n\n\n\n<li>The localhost (or <code>127.0.0.1<\/code>) part can be changed to <code>0.0.0.0<\/code> if one requires the port to be visible on the local network without going through Apache. This applies to HTTP&nbsp;servers such as the <code>.run()<\/code> method of a Flask app (don&#8217;t), waitress, runserver in django or tornado used by Jupyter.<\/li>\n\n\n\n<li>The L\/R argument can be repeated for different ports.<\/li>\n\n\n\n<li>A variant on the above is that you can forward a port that is actually on a different host on the network.<\/li>\n\n\n\n<li>SSH by default is via port 22, but a sys-admin may configure a different port, which is provided via -p argument as above (ignore this is if 22). This is unrelated therefore to the port forward.<\/li>\n\n\n\n<li><code>ServerAliveInterval=180<\/code> is a must as it makes the local machine nudge the remote periodically so the connection does not fail due to inactivity. The -o argument is to specify an option as you&#8217;d use them in the ~\/.ssh\/config file (or \/etc\/ssh file or one specified via Include in either). This is different from <code>ConnectTimeout<\/code> which is for the connection-establishment step.<\/li>\n\n\n\n<li><code>ExitOnForwardFailure=yes<\/code> quits as opposed to hang pointlessly. This means one could use a loop to make sure it is always working:<\/li>\n<\/ul>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"bash\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">#!\/bin\/bash\n\n: &lt;&lt; USAGE\nThis snippet requires\n* $SSH_ADDRESS the address\n* $SSH_FORWARD_PORT the port you want to forward\n* $SSH_USER your username there\n* $SSH_FOLDER\/$SSH_KEY the key to use\nUSAGE\n\nwhile true;\ndo\nssh -N -R 0.0.0.0:$SSH_FORWARD_PORT:localhost:$SSH_FORWARD_PORT \\\n    -l $SSH_USER \\\n    -p $SSH_PORT \\\n    -o ServerAliveInterval=180 \\\n    -o ExitOnForwardFailure=yes \\\n    -i $SSH_FOLDER\/$SSH_KEY \\\n    -o UserKnownHostsFile=$SSH_FOLDER\/known_hosts \\\n$SSH_ADDRESS;\n\necho \"Connection to $SSH_ADDRESS lost\" 1&gt;&amp;2;\nsleep 600;\ndone;<\/pre>\n\n\n\n<p>Combining this with the proxy jumping we start to get this behemoth:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"bash\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">#!\/bin\/bash\n\n: &lt;&lt; USAGE\nThis snippet requires\n* $SSH_ADDRESS is now $SSH_GATEWAY_HOST and $SSH_INNER_HOST\n* $SSH_FORWARD_PORT the port you want to forward\n* $SSH_USER your username there\n* $SSH_FOLDER\/$SSH_KEY the key to use\nUSAGE\n\nwhile true;\ndo\nssh -N -R 0.0.0.0:$SSH_FORWARD_PORT:0.0.0.0:$SSH_FORWARD_PORT \\\n-o ProxyCommand=\"ssh -v -W %h:%p \\\n                     -l $SSH_USER \\\n                     -i $SSH_FOLDER\/$SSH_KEY \\\n                     $SSH_INNER_HOST\" \\\n-i $SSH_FOLDER\/$SSH_KEY \\\n-o ServerAliveInterval=180 \\\n-o UserKnownHostsFile=$SSH_FOLDER\/known_hosts \\\n-l $SSH_USER \\\n-o ExitOnForwardFailure=yes \\\n$SSH_GATEWAY_HOST;\n\necho \"Connection to $SSH_INNER_HOST lost\" 1&gt;&amp;2;\nsleep 600;\ndone;<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Home network<\/h2>\n\n\n\n<p>If one wants to connect to a machine behind a router, for example a home network, they need to configure the router to allow it. Most users are likely to use their home network as opposed to do sys-admin work at their work place, so I will focus on that case in this section.<\/p>\n\n\n\n<p>First, have a machine permanently connected to act as a homeserver, such as Raspberry Pi or desktop \u2014don&#8217;t use a laptop as that may be a fire risk.<br>Install fail2ban as mentioned and enable the ssh jail.<br>Log into your modem\/router \u2014192.168.1.1 or somesuch as likely directed by a sticker on its underside. In the DHCP settings make the IP of the homerserver reserved to its mac address. In the security or advanced settings there will be a port forwarding option, where you can make the local port 22 of your homerserver the external port you want (security by obfuscation as I mentioned). Some fancier\/new modems will have port triggering\/knocking options, which is worth exploring especially you are exposing sensitive content.<\/p>\n\n\n\n<p>Do note that home network IPs are not fixed, but can change periodically. With fibre, generally the IP changes only when the switchboard resets (the dreaded email stating &#8220;we are performing essential maintenance in your area between 8 am to 8 pm, we apologise for the inconvenience&#8221;).<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Known host fingerprints<\/h2>\n\n\n\n<p>When one first connects to a remote host one gets asked if it&#8217;s trusted. Then a hash is made for future connections and when the host changes the hash will not match and an error will occur making a safeguard in case it something nefarious happened. Problems arise if one is running a script and the script is prompted if it is trustworthy and it hangs as there is no TTY.<\/p>\n\n\n\n<p>This behaviour is controlled by <code>StrictHostKeyChecking<\/code> option. Setting this to no will therefore skip it. This is considered unsafe, so in  OpenSSH 7.6 and above the value <code>accept-new<\/code> can be used to accept new keys. The catch is that the ancient OS that refuses to die as it underpins a lot of infrastructure, CentOS 7, is older.<\/p>\n\n\n\n<p>In which case, ssh-keygen, curiously, comes to the rescue as it allows one to exchange fingerprints among other secret things. <\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"bash\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">ssh-keygen -R gate.stats.ox.ac.uk -f \"$SSH_FOLDER\/known_hosts\"\nssh \ud83d\udc7e\ud83d\udc7e\ud83d\udc7e -o \"UserKnownHostsFile=$SSH_FOLDER\/known_hosts\"<\/pre>\n\n\n\n<p>I should mention that if you are going from one host to another within an intranet the <code>StrictHostKeyChecking=no<\/code> is most likely fine, unless it is a very lawless intranet (i.e. college dorms).<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">No home and not interactive<\/h2>\n\n\n\n<p>In a container (e.g. Singularity), you might be a no-home user, ie. you (<code>$USER<\/code>) lack a folder in <code>\/home<\/code>, but have write permission somewhere (i.e. $HOME is some gibberish \/dev path). This has three consequences for SSH. No config file to make life easier, fingerprints are new and if no identity key file is specified (the <code>-i<\/code> argument) it does not check all identity key files in <code>$HOME\/.ssh<\/code>.<\/p>\n\n\n\n<p>Exporting HOME as a different path will do nothing (bar change <code>~<\/code> in that bash) whereas to change a user&#8217;s home path a root user either has to run <code>usermod -d \/not-home\/\ud83d\udc7e\ud83d\udc7e\ud83d\udc7e \ud83d\udc7e\ud83d\udc7e\ud83d\udc7e<\/code> or they manually edit <code>\/etc\/passwd<\/code> file.<\/p>\n\n\n\n<p>A few hacks (say specifying keys via <code>-i<\/code> while remembering about permissions and accepting fingerprints to a file) can be done, although as far as I know there is no way to provide a SSH config file to ssh. <\/p>\n\n\n\n<p>Using Singularity is an option, but that is overkill for that alone.<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"bash\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">#!\/bin\/bash\n\n# ========================\n: &lt;&lt; USAGE\nAs above...\nUSAGE\n# ========================\n\n# your standard boiler plate....\n# $SSH_USER\nif [ -z \"$SSH_USER\" ]; then\n     # raise error is not bash, but will raise an error...\n    raise error \"Your remote username SSH_USER ($SSH_USER) is not specified\"\nfi\n\nif [ -z \"$SSH_GATEWAY_HOST\" ]; then\n     raise error \"No SSH_GATEWAY_HOST\"\nfi\n\nif [ -z \"$SSH_INNER_HOST\" ]; then\n     raise error \"No SSH_INNER_HOST\"\nfi\n\nif [ -n \"$SSH_FORWARD_PORT\" ]; then\n    echo '$SSH_FORWARD_PORT provided directly.'\nelif [ -n \"$JOB_PORT\" ]; then\n    export SSH_FORWARD_PORT=$JOB_PORT\nelif [ -n \"$JUPYTER_PORT\" ]; then\n    export $SSH_FORWARD_PORT=$JUPYTER_PORT\nelif [ -n \"$APPTAINERENV_JUPYTER_PORT\" ]; then\n    export $SSH_FORWARD_PORT=$APPTAINERENV_SSH_FORWARD_PORT\nelse\n    raise error 'No $SSH_FORWARD_PORT provided'\nfi\n\n# ========================\n\nexport DATA=\/\ud83d\udc7e\ud83d\udc7e\ud83d\udc7e;\nexport SSH_KEY=${SSH_KEY:-*}\nexport SSH_FOLDER=${SSH_FOLDER:-$HOME\/.ssh}\n#export SSH_PORT=${SSH_PORT:-22}\n\n# most applications are okay with path\/\/path but not ssh\nexport SSH_FOLDER=$(echo \"$SSH_FOLDER\" | sed \"s\/\\\/\\\/\/\\\/\/g\" | sed \"s\/\\\/$\/\/\")\n\ntouch $SSH_FOLDER\/test.txt\nif [ ! -f $SSH_FOLDER\/test.txt ]\nthen\n\techo \"The folder $SSH_FOLDER is inaccessible\"\n        mkdir -p \/tmp\/ssh\n        export SSH_FOLDER=\/tmp\/ssh\nfi\n\necho 'prep connections by moving keys from $SSH_FOLDER to $HOME'\nmkdir -p $SSH_FOLDER\ntouch $SSH_FOLDER\/known_hosts\nchmod 700 $SSH_FOLDER\nchmod 600 $SSH_FOLDER\/*\n# ========================\n\necho 'accepting fingerprints'\nssh-keygen -R $SSH_GATEWAY_HOST -f \"$SSH_FOLDER\/known_hosts\"\nwhile true;\ndo\nssh -N -R 0.0.0.0:$SSH_FORWARD_PORT:0.0.0.0:$SSH_FORWARD_PORT \\\n-o ProxyCommand=\"ssh -v -W %h:%p -l $SSH_USER -i $SSH_FOLDER\/$SSH_KEY \\\n-o StrictHostKeyChecking=no \\\n$SSH_GATEWAY_HOST\" \\\n-i $SSH_FOLDER\/$SSH_KEY \\\n-o ServerAliveInterval=180 \\\n-o UserKnownHostsFile=$SSH_FOLDER\/known_hosts \\\n-l $SSH_USER \\\n-o ExitOnForwardFailure=yes \\\n-o StrictHostKeyChecking=no \\\n$SSH_INNER_HOST \\\n-v;\n\necho 'Connection to stats lost' 1&gt;&amp;2;\nsleep 600;\ndone;\n\n<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Boss fight<\/h2>\n\n\n\n<p>Now, having established this is possible, let&#8217;s see what this would mean in practive.<\/p>\n\n\n\n<p>Say in the fictional Emerald city in the land of Oz, HTCondor is used in the cluster, then a job could be launched that runs a Jupyter notebook in a singularity container reverse port forwarding it. It just requires a lot of varibles to pass around:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>#!\/bin\/bash\n\n# JOB_ is a custom convention used here to mark the job variables\nexport COMMON_DIR=\ud83d\udc7e\ud83d\udc7e\ud83d\udc7e\nexport JOB_SCRIPT=$COMMON_DIR\/singularity.sh\nexport JOB_INIT_SCRIPT=$COMMON_DIR\/connection.sh\nexport JOB_INNER_SCRIPT=$COMMON_DIR\/notebook.sh\nexport JOB_PORT=\ud83e\udd16\ud83e\udd16\n\n# SSH_ is a somewhat custom convention for the ssh script \nexport SSH_FORWARD_PORT=$JOB_PORT\nexport SSH_KEY=\ud83d\udc7d\ud83d\udc7d\ud83d\udc7d\nexport SSH_USER=\ud83d\udc7b\ud83d\udc7b\ud83d\udc7b\nexport SSH_FOLDER=$COMMON_DIR\/tmp\n\n# CONDA and APPTAINER (Singularity) variables\n# APPTAINERENV are environment variables in Singularity container\nexport CONDA_PREFIX=$COMMON_DIR\/waconda\nexport CONDA_ENVS_PATH=$CONDA_PREFIX\/envs:\ud83e\udd20\ud83e\udd20\ud83e\udd20\/envs:\ud83e\udd12\ud83e\udd12\ud83e\udd12\/envs\nexport JUPYTER_CONFIG_DIR=$COMMON_DIR\/jupyter \nexport APPTAINER_CONTAINER=$COMMON_DIR\/rockycuda.sif\nexport APPTAINERENV_CONDA_PREFIX=$CONDA_PREFIX\nexport APPTAINERENV_JUPYTER_notebook_dir=$COMMON_DIR\nexport APPTAINERENV_JUPYTER_CONFIG_DIR=$JUPYTER_CONFIG_DIR\nexport APPTAINERENV_CONDA_ENVS_PATH=$CONDA_ENVS_PATH\nexport APPTAINERENV_BASHRC_PATH=$COMMON_DIR\/bashrc.sh\nexport APPTAINER_HOSTNAME='\ud83d\udca9\ud83d\udca9\ud83d\udca9'\n\n# Submit the job\ncondor_submit $COMMON_DIR\/target_script.condor -a 'Requirements=(machine == \"\ud83d\udc80\ud83d\udc80\ud83d\udc80.\ud83c\udf83\ud83c\udf83\ud83c\udf83.novalocal\")'\n\n# remind me what to do:\necho \"To debug condor_ssh_to_job and run:\";\necho \"curl localhost:$JOB_PORT\/api\";<\/code><\/pre>\n\n\n\n<p>Where <code>target_script.condor<\/code> is something like<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\"># ============================================================\n# This script runs $JOB_NODE_SCRIPT within initial dir $HOME2\n# to specify a particular machine \n# add `-a 'Requirements=(machine == \"\ud83d\udc7e\ud83d\udc7e\ud83d\udc7e\ud83e\udd16\ud83e\udd16.novalocal\")'` as a cmd arg\n# Envs used:\n# * $HOME2 the fake home, e.g. in a mounted CephFS volume.\n# * $JOB_SCRIPT the job to run\n# ============================================================\n\nExecutable      = \/bin\/bash\narguments       = $ENV(JOB_SCRIPT)\nUniverse        = vanilla\ngetenv          = JOB_*,SINGULARITY_*,JUPYTER_*,CONDA_*,APPTAINER_*,APPTAINERENV_*,PYTHON*,HOME2,SSH_*\ninitialdir      = $ENV(HOME2)\nOutput          = $ENV(HOME2)\/logs\/condor-log.$(Cluster).$(Process).out\nError           = $ENV(HOME2)\/logs\/condor-log.$(Cluster).$(Process).err\nLog             = $ENV(HOME2)\/logs\/condor-log.$(Cluster).$(Process).log\nrequest_cpus = Target.TotalSlotCpus\nrequest_gpus = Target.TotalSlotGPUs\nrequest_memory = Target.TotalSlotMemory\n+RequiresWholeMachine = True\nQueue\n<\/pre>\n\n\n\n<p>Which runs the file <code>$JOB_SCRIPT<\/code>, which is <code>singularity.sh<\/code>, which accepts <\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"bash\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">#!\/bin\/bash\n\n# ========================\n# Run $JOB_INNER_SCRIPT on a singularity container\n# if JOB_INIT_SCRIPT is specified it will run that first.\n# for example a ssh connection script!\n# ========================\n\nexport HOST=${HOST:-$(hostname)}\nexport USER=${USER:-$(users)}\nexport HOME=${HOME:-$_CONDOR_SCRATCH_DIR}\nsource \/etc\/os-release;\necho \"Running script ${0} as $USER in $HOST which runs $PRETTY_NAME.\"\n# ---------------------------------------------------------------\n\nexport DATA=\ud83e\udd16\ud83e\udd16\ud83e\udd16\ud83e\udd16\nexport APPTAINER_CONTAINER=${APPTAINER_CONTAINER:-$DATA\/cuda113ubuntu20cudnn8_latest.sif}\nexport APPTAINER_BIND=\"$DATA:$DATA,$HOME:\/data\/outerhome\"\nexport APPTAINER_HOSTNAME=${APPTAINER_HOSTNAME:-singularity}\necho \"Singularity $APPTAINER_CONTAINER with target script: $JOB_INNER_SCRIPT\"\nexport APPTAINERENV_HOME2=$HOME2\nexport APPTAINERENV_DATA=$DATA\nexport APPTAINERENV_CONDA_ENVS_PATH=${APPTAINERENV_CONDA_ENVS_PATH:-$CONDA_ENVS_PATH}\nexport APPTAINERENV_JUPYTER_CONFIG_DIR=${APPTAINERENV_JUPYTER_CONFIG_DIR:-$JUPYTER_CONFIG_DIR}\nexport APPTAINER_WORKDIR=${APPTAINER_WORKDIR:-\/tmp}\n#export APPTAINER_WRITABLE_TMPFS=${APPTAINER_WRITABLE_TMPFS:-true}\nexport SSH_FORWARD_PORT=$JOB_PORT;\nexport SINGULARITY_CACHEDIR=\/tmp\/singularity;\nexport SINGULARITY_LOCALCACHEDIR=\/tmp\/singularity;\nexport APPTAINER_TMPDIR=\/tmp\/singularity;\nexport APPTAINERENV_JUPYTER_PORT=${APPTAINERENV_JUPYTER_PORT:-$JOB_PORT}\n# overlay does not work.\n# ---------------------------------------------------------------\n\nif [ -n \"$JOB_INIT_SCRIPT\" ]; then\n    bash $JOB_INIT_SCRIPT &amp;\nfi\n\n# ---------------------------------------------------------------\n\necho 'Running singularity ...'\nif ls \/dev | grep -q '^nvidia'; then\n    echo \"NVIDIA GPU is present.\"\n    \/usr\/bin\/singularity exec --nv --writable-tmpfs $APPTAINER_CONTAINER \/bin\/bash $JOB_INNER_SCRIPT;\nelse\n    echo \"No NVIDIA GPU found.\"\n    \/usr\/bin\/singularity exec --writable-tmpfs $APPTAINER_CONTAINER \/bin\/bash $JOB_INNER_SCRIPT;\nfi\n\n\necho 'DIED!' 1&gt;&amp;2;\n<\/pre>\n\n\n\n<p>This script does nothing more than run $JOB_INNER_SCRIPT on a singularity container after that it runs a $JOB_INIT_SCRIPT if present. The latter would be the ssh connection, for example as mentioned above. While the former would be a notebook:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">#!\/bin\/bash\n\nexport HOST=${HOST:-$(hostname)}\nexport USER=${USER:-$(users)}\nexport HOME=${HOME:-$_CONDOR_SCRATCH_DIR}\nsource \/etc\/os-release;\n\nif [ -n \"$JUPYTER_PORT\" ]; then\n    echo \"$JUPYTER_PORT set\"\nelif [ -n \"$JOB_PORT\" ]; then\n    export JUPYTER_PORT=$JOB_PORT\nelif [ -n \"$SSH_FORWARD_PORT\" ]; then\n    export JUPYTER_PORT=$SSH_FORWARD_PORT\nelse\n    raise error \"Your JUPYTER_PORT is not specified\"\nfi\n\nif [ -z \"$JUPYTER_CONFIG_DIR\" ]; then\n    raise error \"Your JUPYTER_CONFIG_DIR is not specified either\"\nfi\n\nexport LD_LIBRARY_PATH=\/usr\/local\/cuda\/compat:$CONDA_PREFIX\/lib:$LD_LIBRARY_PATH;\n\necho \"************************\"\necho \"HELLO JUPYTER!\"\necho \"************************\"\necho \"Greet from Jupyter lab script ${0} as $USER in $HOST which runs $PRETTY_NAME on $JUPYTER_PORT with settings from $JUPYTER_CONFIG_DIR\"\n\nif [ -n \"$BASHRC_PATH\"]; then\n    source $BASHRC_PATH;\nelse\n    source $JUPYTER_CONFIG_DIR\/.bashrc;\nfi\n\nconda activate\n\n#export JUPYTER_CONFIG_PATH=$HEADQUARTERS\/.jupyter\n\n# First time? Remember to set:\n# jupyter notebook --generate-config\n# yes foo | jupyter server password\n\n# port is JUPYTER_PORT\nwhile true\ndo\njupyter lab --ip=\"0.0.0.0\" --no-browser --NotebookNotary.db_file=':memory:'\ndone<\/pre>\n\n\n\n<p>See I said it was easy! but wait. There is still one more level to conquer. However, the boss flight in the Land of Pipes and Sockets is for another time!<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Miscellaneous Footnotes<\/h2>\n\n\n\n<p><strong>System deamon<\/strong>. If you need to ssh via a system deamon make sure to put a sleep before doing anything and set the service to wait until the network is &#8220;online&#8221; (not <code data-enlighter-language=\"generic\" class=\"EnlighterJSRAW\">network.target<\/code>, which just means your network card is working) .<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>After=network-online.target\nWants=network-online.target<\/code><\/pre>\n\n\n\n<p><strong>Missing binary.<\/strong> The more barebones the OS is the less stuff is present. As soon as you apt-get or dnf upate it will install ssh (but with the addition of a user called ssh). It is not installed with apt-utils.<\/p>\n\n\n\n<p><strong>Give grief to cluster abusers with wall<\/strong>. On a cluster there are some pieces of etiquette that get abused, like running heavy jobs on log-in nodes. Namely, don&#8217;t run notebooks in log-in nodes, run them in compute nodes. If you are a victim of someone doing that and if you don&#8217;t know who they are, the command <code>wall \"\ud83d\udc7e\ud83d\udc7e\ud83d\udc7e\"<\/code> (with a polite message) will allow you to  display your message to all logged in users.<\/p>\n\n\n\n<p><strong>Mac TextEditor<\/strong>. If you are editing <code>.ssh\/config<\/code> locally why not use your GUI editor? On a mac adding to your <code>.zshrc<\/code> the line <code>alias open_text='open -a \/System\/Applications\/TextEdit.app'<\/code> will mean that <code>open_text .ssh\/config<\/code> will do it.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Root Folders<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>\/usr<\/code> Unix system resources like the PATH folder <code>\/usr\/bin<\/code> \u2014nothing to do with user.<br>Often environment module files get put in <code>\/usr\/share\/Modules<\/code> (<code>module avail<\/code> will tell the path on the dashed header lines)<\/li>\n\n\n\n<li><code>\/home\/$USER<\/code> is the default location of your <code>$HOME<\/code>. On a Mac it&#8217;s <code>\/Users<\/code>, but that&#8217;s a rabbithole of differences for another time.<\/li>\n\n\n\n<li><code>\/tmp<\/code> A scratch folder. If you cannot write anyway, can you write there?<\/li>\n\n\n\n<li><code>\/dev<\/code> The devices folder. You will have see it <em>ad nauseaum<\/em> with mount command, as the null bucket (<code>\/dev\/null<\/code>) or as one form of input or other (<code>\/dev\/tty<\/code>). But in Singularity &amp; co. <code>df -h<\/code> will reveal that it does more \u2014if technobabble, please search.<\/li>\n\n\n\n<li><code>\/etc<\/code> folder has configs<\/li>\n\n\n\n<li><code>\/var<\/code> folder has logs<\/li>\n\n\n\n<li><code>\/opt<\/code> is where stuff gets put when the root user is not sure.<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Secure shell (SSH) is an essential tool for remote operations. However, not everything with it is smooth-sailing. Especially, when you want to do things like reverse\u2013port-forwarding via a proxy-hump or two a Jupyter notebook to your local machine from a compute node on a no-home container . Even if it sounds less plausible than the [&hellip;]<\/p>\n","protected":false},"author":102,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"nf_dc_page":"","wikipediapreview_detectlinks":true,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"ngg_post_thumbnail":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[348,296,14],"tags":[],"ppma_author":[701],"class_list":["post-10272","post","type-post","status-publish","format-standard","hentry","category-bash","category-hints-and-tips","category-howto"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"authors":[{"term_id":701,"user_id":102,"is_guest":0,"slug":"nuben","display_name":"Matteo Ferla","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/2185fc527a4ace0863c903d27646996994413c31d80e8e4c175477b836e7715c?s=96&d=mm&r=g","0":null,"1":"","2":"","3":"","4":"","5":"","6":"","7":"","8":""}],"_links":{"self":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/10272","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/users\/102"}],"replies":[{"embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/comments?post=10272"}],"version-history":[{"count":5,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/10272\/revisions"}],"predecessor-version":[{"id":10461,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/10272\/revisions\/10461"}],"wp:attachment":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/media?parent=10272"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/categories?post=10272"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/tags?post=10272"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=10272"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}