Anyways, the hypothesis<\/a> says the following: “Dense, randomly-initialized, feed-forward networks contain subnetworks (winning tickets) that\u2014when trained in isolation\u2014reach test accuracy comparable to the original network in a similar number of iterations.” In their analogy, the random initialization of a models weights is treated like a lottery, where some combination of a subset of these weight is already pretty close to the network you want to train (winning ticket). For a better description and a summary of advances in this field I would recommend this blog post<\/a>.<\/p>\n\n\n\n The key idea here has to do with finding the best optimizer<\/a> to train a model capable of generalization. According to this paper<\/a>, a model that has converged to a sharp minima will be less likely to generalize than one that has converged to a flatter minima. They show the following plot to provide an intuition of why this may be the case. <\/p>\n\n\n\n In the SAM paper<\/a> (and ASAM for adaptive<\/a>) the authors implement an optimizer that is more likely to converge to a flat minima. I found this blog post by the authors of ASAM <\/a>gives a very good description of the field.<\/p>\n\n\n\n\n\n\n\n There are some amazing memes being generated by DALL-E mini,<\/a> but its older brother DALL-E 2 has been shown to be capable of generating some beautiful images<\/a>. I would break the magic of DALL-E 2 into two main ideas: CLIP<\/a> and denoising diffusion<\/a>. <\/p>\n\n\n\n CLIP (Contrastive Language\u2013Image Pre-training<\/em>) is a training method in which you task the model with finding the best fitting caption from a limited set of captions for a given image. In theory, to achieve this, the model has to learn to recognize visual concepts and associate them with their names. For a great description of CLIP check out this blog by OpenAI<\/a>.<\/p>\n\n\n\n Denoising diffusion is a unsupervised way of training generative models that basically consists on adding increasing amounts of random noise to your data and then asking your model to denoise it. Once the added noise is large enough, you have a model capable of generating new data points by sampling from noise. For a nice intro into denoising diffusion, I would recommend this blog post<\/a>. And finally, for a proper description of how DALL-E 2 works I would point you to this nicely written blog post<\/a> or their paper<\/a>.<\/p>\n\n\n\n So there you go!! A bunch of links to deep learning blogs written by people that can write way better than me!! If you have read this far I hope this was not a complete waste of your time.<\/p>\n","protected":false},"excerpt":{"rendered":" I was planning on doing a blog post about some cool random deep learning paper that I have read in the last year or so. However, I keep finding that someone else has already written a way better blog post than what I could write. Instead I have decided to write a very brief summary […]<\/p>\n","protected":false},"author":71,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"nf_dc_page":"","wikipediapreview_detectlinks":true,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"ngg_post_thumbnail":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[632,189],"tags":[],"ppma_author":[552],"class_list":["post-8417","post","type-post","status-publish","format-standard","hentry","category-deep-learning","category-machine-learning"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"authors":[{"term_id":552,"user_id":71,"is_guest":0,"slug":"brennan","display_name":"Brennan Abanades Kenyon","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/5c85dcbb5b1499e82ecfc264ec387c8302ac238c786e68cc5c92e9c21904d260?s=96&d=mm&r=g","0":null,"1":"","2":"","3":"","4":"","5":"","6":"","7":"","8":""}],"_links":{"self":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/8417","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/users\/71"}],"replies":[{"embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/comments?post=8417"}],"version-history":[{"count":2,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/8417\/revisions"}],"predecessor-version":[{"id":8431,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/8417\/revisions\/8431"}],"wp:attachment":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/media?parent=8417"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/categories?post=8417"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/tags?post=8417"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=8417"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}SAM: Sharpness aware minimization<\/h2>\n\n\n\n
![\"\"](\"https:\/\/i0.wp.com\/cdn.codeground.org\/nsr\/images\/img\/researchareas\/ai-article10_04.png?resize=625%2C263&ssl=1\")
<\/figure>\n\n\n\nDALL-E 2<\/h2>\n\n\n\n