Last week, a sizable fraction of OPIG headed to “The Plough” near Bradford on Avon, Wiltshire, for our OPIG Retreat (a.k.a. “OPIGtreat”). Some of us travelled on Monday by train, encountering a biblical deluge, a darkness resembling a train tunnel but filled with pelting rain and bath-tubs of water washing down the train’s windows — and lightning strikes every 30s while changing in Bath Spa…
The red monster storm with black eye is what greeted us as we were arriving in Bath Spa. (Credit: UK Thunderstorm Updates: “This is what the satellite imagery is currently showing from the storm affecting Somerset/ Wiltshire. Cloud tops on this monster have exceeded 40,000ft….”)
Undeterred, and either shuttled by the fabulous Anita from Bradford on Avon station, or walking on foot, we arrived at our lovely destination and home for the week.
When you wrangle data for a living, you start to wonder why everything takes so darn long. Through five years of introspection, I have come to conclude that two simple factors limit every computational project. One is, of course, your personal productivity. Your time of focused work, minus distractions (and yes, meetings figure here), times your energy and mental acuity. All those things you have little control over, unfortunately. But the second is the productivity of your code and tools. And this, in principle, is a variable that you have full control over.
Even quick calculations, when applied to tens of millions of sequences, can take quite some time!
This is a post about how to increase your productivity, by helping you navigate all those instances when the progress bar does not seem to go fast enough. I want to discuss actionable tools to make your code run faster, and generate more results, with less effort, in less time. Instructions to tinker less and think more, so you can do the science that you truly want to be doing. And, above all, I want to give out advice that is so counter-intuitive that you should absolutely consider following it.
In preparing for battle I have always found that plans are useless, but planning is indispensable.
Dwight D. Eisenhower
Following the previous post about OPIG retreat 2022, and having received numerous requests for recipes, I thought I’d document the process of ensuring that 24 people are kept fed and happy. Recipes at the foot of the post.
Disclaimer – these recipes are entirely my own interpretations, adapted where necessary to suit a range of dietary requirements. They are in no way authentic to any national cuisines and are not intended to be.
Disclaimer II: The Disclaiming – all measurements are approximate. I rarely write down recipes or use precise measurements. Taste as you go, and don’t be afraid to add more salt.
Websites store cookies to enhance functionality and personalise your experience. You can manage your preferences, but blocking some cookies may impact site performance and services.
Essential cookies enable basic functions and are necessary for the proper function of the website.
Name
Description
Duration
Cookie Preferences
This cookie is used to store the user's cookie consent preferences.
30 days
These cookies are used for managing login functionality on this website.
Name
Description
Duration
wordpress_logged_in
Used to store logged-in users.
Persistent
wordpress_sec
Used to track the user across multiple sessions.
15 days
wordpress_test_cookie
Used to determine if cookies are enabled.
Session
Statistics cookies collect information anonymously. This information helps us understand how visitors use our website.
Google Analytics is a powerful tool that tracks and analyzes website traffic for informed marketing decisions.
Contains information related to marketing campaigns of the user. These are shared with Google AdWords / Google Ads when the Google Ads and Google Analytics accounts are linked together.
90 days
__utma
ID used to identify users and sessions
2 years after last activity
__utmt
Used to monitor number of Google Analytics server requests
10 minutes
__utmb
Used to distinguish new sessions and visits. This cookie is set when the GA.js javascript library is loaded and there is no existing __utmb cookie. The cookie is updated every time data is sent to the Google Analytics server.
30 minutes after last activity
__utmc
Used only with old Urchin versions of Google Analytics and not with GA.js. Was used to distinguish between new sessions and visits at the end of a session.
End of session (browser)
__utmz
Contains information about the traffic source or campaign that directed user to the website. The cookie is set when the GA.js javascript is loaded and updated when data is sent to the Google Anaytics server
6 months after last activity
__utmv
Contains custom information set by the web developer via the _setCustomVar method in Google Analytics. This cookie is updated every time new data is sent to the Google Analytics server.
2 years after last activity
__utmx
Used to determine whether a user is included in an A / B or Multivariate test.
18 months
_ga
ID used to identify users
2 years
_gali
Used by Google Analytics to determine which links on a page are being clicked
30 seconds
_ga_
ID used to identify users
2 years
_gid
ID used to identify users for 24 hours after last activity
24 hours
_gat
Used to monitor number of Google Analytics server requests when using Google Tag Manager
1 minute
Google reCAPTCHA helps protect websites from spam and abuse by verifying user interactions through challenges.
Name
Description
Duration
_GRECAPTCHA
Google reCAPTCHA sets a necessary cookie (_GRECAPTCHA) when executed for the purpose of providing its risk analysis.