{"id":323,"date":"2013-02-26T16:24:18","date_gmt":"2013-02-26T16:24:18","guid":{"rendered":"http:\/\/blopig.com\/blog\/?p=323"},"modified":"2013-02-26T19:12:13","modified_gmt":"2013-02-26T19:12:13","slug":"recognizing-pitfalls-in-virtual-screening-a-critical-review","status":"publish","type":"post","link":"https:\/\/www.blopig.com\/blog\/2013\/02\/recognizing-pitfalls-in-virtual-screening-a-critical-review\/","title":{"rendered":"Recognizing pitfalls in Virtual Screening: A critical review"},"content":{"rendered":"

So, my turn to present at Group Meeting Wednesdays<\/a> – and I decided to go for the work by Scior et al.\u00a0<\/em>about pitfalls in virtual screening<\/a>. \u00a0As a general comment, I think this paper is well written and tackles most of the practical problems when running a virtual screening (VS) exercise. \u00a0Anyone, who intends to either develop a method in this field or else is planning to run a virtual screening \u00a0exercise should read it. \u00a0I’ve often heard the phrase “virtual screening doesn’t work”, and that comes almost exclusively from people who run computational experiments as a black box, without understanding what is going on and by accepting all defaults for a specific protocol. \u00a0This paper highlights what to watch out for. \u00a0Of the author list, I’ve only met with Andreas Bender once at an MGMS meeting a few years back – his main PhD work was on molecular similarity<\/a>.<\/p>\n

The article describes pitfalls associated with four areas; expectations and assumptions; data design and content; choice of software; and conformational sampling as well as ligand and target flexibility. \u00a0The authors start off by arguing that the expectations are too high; people just run a VS experiment and expect to find a potent drug. \u00a0But this a rare occurrence indeed. \u00a0Below is a set of notes for their main points.<\/p>\n

Erroneous assumptions and expectations<\/strong><\/p>\n

    \n
  1. High expectations: main goal is to identify novel bioactive chemical matter for the particular target of interest. \u00a0Highly potent compounds desirable but not required. \u00a0Expectations too high. \u00a0Lead; single digit \u00b5M and Hit; < 25\u00b5M
    \n<\/span><\/li>\n
  2. Stringency of queries: strict vs loose search criteria. \u00a0Strict; no diversity, few good results returned. \u00a0Loose; returns many false positives. \u00a0 Removal of one feature at a time \u2013 a kind of pharmacophoric feature bootstrapping which highlights which features are important.<\/li>\n
  3. Difficulty in binding pose prediction. \u00a0Taken from their reference [46], “For big (> 350 Daltons) ligands, however, there is currently very little evidence. We hope investigators will come forward with crystallographic confirmation of docking predictions for higher molecular weight compounds to shed more light on this important problem.<\/em>” \u00a0This point is really interesting and tools, such as Gold, even have an FAQ entry<\/a> which addresses this question.<\/li>\n
  4. Water: hydrogen bonds are mediated by water which are often visible in the crystal structure. \u00a0Hard to predict exact number, position and orientation. \u00a0Realism of model at a cost of computational resources.<\/li>\n
  5. Single vs. multiple\/allosteric binding pockets: Sometimes binding site is not known yet we always assume that ligand binds to one specific place.<\/li>\n
  6. Subjectivity of Post VS compound: \u00a0The result of a VS experiment is a ranking list of the whole screening database. \u00a0Taking top N<\/em> results in very similar compounds \u2013 so some sort of post processing is usually carried out (e.g. clustering, but even a subjective manual filtering). \u00a0Difficult to reproduce across studies.<\/li>\n
  7. Prospective validation: the benchmarking of VS algorithms is done retrospectively. \u00a0Test on an active\/decoy set for a particular target. \u00a0Only putative inactives. \u00a0Rarely validated in an external prospective context.<\/li>\n
  8. Drug-likeness: most VS experiments based on Lipinski Ro5 – not more than 5 hydrogen bond donors (nitrogen or oxygen atoms with one or more hydrogen atoms), not more than 10 hydrogen bond acceptors (nitrogen or oxygen atoms), a molecular mass less than 500 daltons, An octanol-water partition coefficient log P<\/em> not greater than 5. \u00a0But these apply to oral bioavailability. \u00a0Lots of drugs fall out of this scope; intravenous drugs, antibiotic, peptidic drugs. \u00a0VS validated on Lipinski space molecules.<\/li>\n
  9. Diversity of benchmark library vs diversity of future , prospective vs runs: Library must fit the purpose of the experiment. \u00a0Most VS validation experiments are on commercially available libraries ~ small fraction of chemical space. \u00a0Type of screening library must be closely related to objective of VS campaign. \u00a0 Results have to be transferable between runs. Validation on specific target family? If the goal is lead optimization combinatorial libraries are attractive. \u00a0Natural versus synthesized compounds; different chemical space.<\/li>\n<\/ol>\n

    Data design and content<\/strong><\/p>\n

      \n
    1. Incomparability of benchmark sets: some datasets for docking studies, others for ligand based VS \u2013 incomparable methods. \u00a0In general 2D methods better than 3D (surprising!). \u00a0In 2D methods fingerprints outperform 3D methods. \u00a0Same datasets for validation of different methods \u2013 hard to reproduce any study otherwise.<\/span>\u00a0<\/span><\/li>\n
    2. Limited comparability of performance metrics: Tag along on previous point; performance measurement used for different measurements should be the same. Mean EF risky because of ratio between actives to inactive molecules. ROC curves a problem because of early and late performance \u2013 use of BedROC (different importance to early and late stages of retrieved list of compounds). EF = (number of actives \/ number of expected) for a given % of the database<\/span><\/li>\n
    3. Hit rate in benchmark data sets; small libraries not good enough. Typical VS hit rates ~0.01% \u2013 0.14%. Analogue bias; actives all look very similar to each other. Artificial enrichment; easy to tell between actives and decoys. Recent study found that for ligand based VS using no. of atoms gives half the VS performance.<\/span><\/li>\n
    4. Assay Comparability and Technology: Properly designed datasets such as MUV<\/a>, use of similarity distributions to remove anything very similar to each other. \u00a0Remove problematic molecules like autofluoroscence . \u00a0MUV uses data from pubchem; different bioassays from different groups hence different quality. \u00a0Choices of targets; cutoffs; parameters; etc. \u00a0“Ideal VS benchmark deck will never happen.”<\/span><\/li>\n
    5. Bad molecules as actives: No real activity but either reactive or aggregating molecules in the assay which gives up a false positive; PAINs Pan assay interfering substances or frequent hitters. Small number of actives compared to inactives, false positives worse than false negatives.<\/span><\/li>\n
    6. Putative inactive compounds as decoys. The decoys are actually actives.\u00a0<\/span><\/li>\n
    7. Feature weights: LBVS based on a single query fails to identify important parts of the the molecule, e.g. benzamidine warhead in factor Xa inhibitors<\/span><\/li>\n<\/ol>\n

      Choice of Software<\/strong><\/p>\n

        \n
      1. Interconverting chemical formats; errors or format incompatibilities. \u00a0Information lost or altered; or when using same format across different software (e.g. chirality, hybridization, and protonation states).<\/span><\/span><\/li>\n
      2. Molecule preparation; query molecules must be preprocessed exactly the same way as the structures in the database being screened to ensure consistency (e.g. partial charge calculation)<\/span><\/li>\n
      3. Feature definition: Specific rules which are sometimes left out of pharmacophoric definition. e.g. O, N in oxazole do not both behave as a HBA. Watch out for tautomers, protonation state, and chirality<\/span><\/li>\n
      4. Fingerprint selection and algorithmic implementation: different implementations of same fingerprint MACCS result in different fingerprints. Choice of descriptors; which ones to pick? Neighbourhood? Substructure?<\/span><\/li>\n
      5. Partial charges: Mesomeric effects; formal +1 charge spread over guanidine structure.<\/span><\/li>\n
      6. Single predictors versus ensembles: no single method works best in all cases. Consensus study; apply multiple methods and combine results.<\/span><\/li>\n<\/ol>\n

        Conformational sampling as well as ligand and target flexibility<\/strong><\/p>\n

          \n
        1. Conformational coverage: four main parameters: (i) sampling algorithms and their specific parameters; (ii) strain energy cutoffs (iii) maximum number of conformations per molecule (iv) clustering to remove duplicates<\/span><\/li>\n
        2. Defining bioactive conformations: most ligands have never been co-crystallized with their primary targets and even fewer have been cocrystallized with counter targets. Same ligand might bind to different proteins in vastly different conformations. How easy is it to reproduce the cognate conformation? Also ligand changes shape upon binding. Minimum energy conformations are a common surrogate.<\/span><\/li>\n
        3. Comparing conformations: definitions of identity thresholds. 0.1 < rmsd < 0.5 excellent; 0.5 < rmsd < 1.0 good fit; 1.0 < rmsd < 1.5 acceptable; 1.5 < rmsd < 2.0 less acceptable; >2.0 not a fit in terms of biological terms. All atoms vs fragments RMSD makes direct comparison hard.<\/span><\/li>\n
        4. Size of conformational ensemble; trade off between computational cost and sampling breadth. Conformer generator may not generate bioactive conformation. How many conformations required to have bioactive one. Many bioactive conformations might exist.<\/span><\/li>\n
        5. Ligand flexibility \u2013 hard upper limit for no. of conformations. Conformer sizes depend mostly on number of rotatable bonds. Conformer generation tools don’t work well on some classes of molecules e.g. macrocycles<\/span><\/li>\n
        6. High energy conformations \u2013 high energy conformers (or physically unrealistic molecules; e.g. a cis secondary amide) detrimental to VS experiments. 3D pharmacophore searches sometimes result in matching strained structure; but 70% of ligands bind at strain energies below 3kcal\/mol (stringent).\u00a0<\/span><\/li>\n
        7. Target flexibility \u2013 target flexibility \u2013 can do simple things like sidechain rotation, but nothing major like backbone flexibility. Sometimes docking to multiple structures snapshots resulting from molecular dynamics<\/span><\/li>\n
        8. Assumption of ligand overlap \u2013 lots of 3D shape based VS attempt to maximize the overlap between ligands \u2013 but based on X-ray structures this is not always the case (different ligands may occupy slight different regions of the binding pocket).<\/span><\/li>\n
        9. Missing positive controls \u2013 Strict cutoff stops you from retrieving positive controls in your Virtual Screening experiment. Selectivity (lower number of false postives)\/ sensitivity\u00a0(larger percentage of true positives) cutoff needs to be determined appropriately.<\/span><\/li>\n<\/ol>\n

          In conclusion, VS can be run by a monkey \u2013 but if that is the case expect bad results. Careful database preparation, judicious parameter choices, use of positive controls, and sensible compromises between the different goals one attempts to obtain are required. VS probabilistic game \u2013 careful planning and attention to detail increases probability of success.<\/p>\n","protected":false},"excerpt":{"rendered":"

          So, my turn to present at Group Meeting Wednesdays – and I decided to go for the work by Scior et al.\u00a0about pitfalls in virtual screening. \u00a0As a general comment, I think this paper is well written and tackles most of the practical problems when running a virtual screening (VS) exercise. \u00a0Anyone, who intends to […]<\/p>\n","protected":false},"author":6,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"nf_dc_page":"","wikipediapreview_detectlinks":true,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"ngg_post_thumbnail":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[10],"tags":[],"ppma_author":[483],"class_list":["post-323","post","type-post","status-publish","format-standard","hentry","category-groupmeetings"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"authors":[{"term_id":483,"user_id":6,"is_guest":0,"slug":"jp","display_name":"JP Ebejer","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/0c2ee7f7a071e0ff6aebc09ddb6c7bdd90146efeb8dfac34f7724f0b04ceafe8?s=96&d=mm&r=g","0":null,"1":"","2":"","3":"","4":"","5":"","6":"","7":"","8":""}],"_links":{"self":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/323","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/comments?post=323"}],"version-history":[{"count":9,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/323\/revisions"}],"predecessor-version":[{"id":333,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/323\/revisions\/333"}],"wp:attachment":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/media?parent=323"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/categories?post=323"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/tags?post=323"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=323"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}