{"id":6663,"date":"2021-02-18T16:03:51","date_gmt":"2021-02-18T16:03:51","guid":{"rendered":"https:\/\/www.blopig.com\/blog\/?p=6663"},"modified":"2021-02-18T16:16:48","modified_gmt":"2021-02-18T16:16:48","slug":"to-pickle-or-not-to-pickle-quickle-%f0%9f%a5%92","status":"publish","type":"post","link":"https:\/\/www.blopig.com\/blog\/2021\/02\/to-pickle-or-not-to-pickle-quickle-%f0%9f%a5%92\/","title":{"rendered":"To Pickle, Or Not To Pickle? \u2014 Quickle!"},"content":{"rendered":"\n<p>Pickling in Python can be <a href=\"https:\/\/intoli.com\/blog\/dangerous-pickles\/\">dangerous<\/a>.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large is-style-rounded\"><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/thumb\/b\/bb\/Pickle.jpg\/320px-Pickle.jpg\" alt=\"A pickled gherkin, also known as a deli pickle. Public domain.\"\/><figcaption>A pickled gherkin, a.k.a. a deli pickle. Public <a href=\"https:\/\/commons.wikimedia.org\/wiki\/File:Pickle.jpg\">domain<\/a>.<\/figcaption><\/figure>\n\n\n\n<p>That&#8217;s where <a href=\"https:\/\/jcristharif.com\/quickle\/\"><code>Quickle<\/code><\/a> comes in \u2014 as long as you&#8217;re using Python 3.8 or later&#8230;<\/p>\n\n\n\n<!--more-->\n\n\n\n<p>The Python standard library has a package for translating between Python objects and files called <a href=\"https:\/\/docs.python.org\/3\/library\/pickle.html#module-pickle\"><code>pickle<\/code><\/a>. Technically, this process is called <a href=\"https:\/\/en.wikipedia.org\/wiki\/Serialization\">serializing<\/a> or deserializing, depending on the direction. This can be a really handy way to save work in Python.<\/p>\n\n\n\n<p>But, as the documentation for <code>pickle<\/code> says,<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\"><p><strong>Warning<\/strong>: The <code>pickle<\/code> module is <strong>not secure<\/strong>. Only unpickle data you trust.<br><br>It is possible to construct malicious <code>pickle<\/code> data which will <strong>execute arbitrary code during unpickling<\/strong>. Never unpickle data that could have come from an untrusted source, or that could have been tampered with.<br><br>Consider signing data with <a href=\"https:\/\/docs.python.org\/3\/library\/hmac.html#module-hmac\"><code>hmac<\/code><\/a> if you need to ensure that it has not been tampered with.<br><br>Safer serialization formats such as <a href=\"https:\/\/docs.python.org\/3\/library\/json.html#module-json\"><code>json<\/code><\/a> may be more appropriate if you are processing untrusted data. See <a href=\"https:\/\/docs.python.org\/3\/library\/pickle.html#comparison-with-json\">Comparison with <code>json<\/code><\/a>.<\/p><cite>Pink warning box from the Python documentation for pickle.<\/cite><\/blockquote>\n\n\n\n<p><code>Quickle<\/code> prevents the possibility of <a href=\"https:\/\/stackoverflow.com\/questions\/47705202\/pickle-exploiting\">executing arbitrary code upon deserializing<\/a>, and natively supports a wide range of builtin Python types (unlike <code>msgpack<\/code> or <code>json<\/code>). <code>Quickle<\/code> is also <a href=\"https:\/\/jcristharif.com\/quickle\/benchmarks.html#\">faster<\/a> than <code>pickle<\/code>, according to the developer&#8217;s website.<\/p>\n\n\n\n<p>It&#8217;s easy to install, using either <code>conda<\/code> or <code>pip<\/code>:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"shell\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\"> # How to install quickle using conda:\nconda install -c conda-forge quickle\n\n# How to install quickle using pip:\npip install quickle<\/pre>\n\n\n\n<p>Q<code>uickle<\/code> uses <code><a href=\"https:\/\/jcristharif.com\/quickle\/api.html#quickle.dumps\">dumps<\/a><\/code> to serialize, and <code><a href=\"https:\/\/jcristharif.com\/quickle\/api.html#quickle.loads\">loads<\/a><\/code> to deserialize, Python objects, respectively \u2014 just like <code>pickle<\/code> \u2014 but it is also possible to create an <code><a href=\"https:\/\/jcristharif.com\/quickle\/api.html#quickle.Encoder\">Encoder<\/a><\/code> (or <code><a href=\"https:\/\/jcristharif.com\/quickle\/api.html#quickle.Decoder\">Decoder<\/a><\/code>) for a more efficient implementation.<\/p>\n\n\n\n<p>The <a href=\"https:\/\/github.com\/jcrist\/quickle\">source code for <code>quickle<\/code><\/a> is available from Jim Crist-Harif&#8217;s GitHub repo.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Pickling in Python can be dangerous. That&#8217;s where Quickle comes in \u2014 as long as you&#8217;re using Python 3.8 or later&#8230;<\/p>\n","protected":false},"author":35,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"nf_dc_page":"","wikipediapreview_detectlinks":true,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"ngg_post_thumbnail":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[29,296,14,227],"tags":[363,362,364],"ppma_author":[488],"class_list":["post-6663","post","type-post","status-publish","format-standard","hentry","category-code","category-hints-and-tips","category-howto","category-python-code","tag-pickling","tag-security","tag-unpickling"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"authors":[{"term_id":488,"user_id":35,"is_guest":0,"slug":"garrett","display_name":"Garrett","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/df625261419c37dd5c5937e37f17a732626acd6eea1e6fabd03d935c25b453bf?s=96&d=mm&r=g","0":null,"1":"","2":"","3":"","4":"","5":"","6":"","7":"","8":""}],"_links":{"self":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/6663","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/users\/35"}],"replies":[{"embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/comments?post=6663"}],"version-history":[{"count":2,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/6663\/revisions"}],"predecessor-version":[{"id":6666,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/6663\/revisions\/6666"}],"wp:attachment":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/media?parent=6663"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/categories?post=6663"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/tags?post=6663"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=6663"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}