Comments on: PDB Structure Quality Tools

By: Morten G

Morten G — Mon, 08 Nov 2010 14:04:15 +0000

About 1adv. From the electron density server:

“There is no map available for this entry (1adv),
because our automatic script failed to produce an electron density map
with an R-value (0.272) within 5 percentage points of the published one (0.206).”

So maybe they mistyped when they submitted? Though the B-factor is very bizarre too though… Oh – the bond lengths are completely overfitted and the angles somewhat.

By: Sean

Sean — Fri, 05 Jun 2009 01:16:41 +0000

Eric: When I was getting together the ‘Do we need an X-ray Diffraction Image Data Bank?’ post ready I came across the paper entitled Case-controlled structure validation in Acta D65, 2009, 140-147. I was amazed at how many times your work was mentioned.

Also @Oliver and Eric – interesting discussion on what is the proper way to treat the R-factor/R-free gap.

By: Eric Brown

Eric Brown — Thu, 04 Jun 2009 19:22:09 +0000

Sean: I’m working on a follow-up paper discussing the improvements that have been made in the past year. But I don’t have any money to work on this so my day-job is taking priority right now.

Oliver: I agree that a single metric cannot express everything we want to know about a structure’s quality. I hope to expand the site to show how the different “orthogonal” features that individually make up the q-value vary — thus letting a user see that their structure might be decent on a global scale (R-free/R-factor) but still have some local error (ex. Ramachandran problems).

The process that I’m using now makes it easy to add additional “quality” measures. So I’ll try adding a R-factor / R-free gap and see how those relate. I’m not sure how the gap will correlate with the R-free or R-factor but one benefit of the method is that it will look for correlations and work with them.

By: Oliver Clarke

Oliver Clarke — Tue, 02 Jun 2009 23:10:15 +0000

While I think this is a great tool for assessing overall quality trends in groups of structures, I think it can be a little misleading for individual cases if one only looks at the Q-factor. Partly I think this is because as a single metric, individually very worrying statistics can be missed if all the other stats are better than average.

Additionally, the Q-factor does not (as far as I can see) incorporate any assessment of the gap between R and Rfree, which can give an in indication of either overfitting (if R and Rfree are too far apart) or a biased validation set (if they are too close)

Eg, as an example have a look at 1adv. This is a 3.2A structure, with an R/Rfree of 21/34. A gap of 13 points between the R and Rfree indicates clear overfitting, but because R and Rfree apparently are assessed independently wrt the Q-factor (the lower the better), the structure isn’t penalised for this.

Further, this structure has an average B-factor of 17 (2-10 in the core of the protein), which is clearly a bit odd for a 3.2A structure, but there is no penalty for having a lower B-factor than expected, so the overall Q-factor is excellent – -0.79!

So overall, I’d say it’s an excellent tool to get a quick summary of the stats, but I’d still want to have a look at the model and map to get a real idea of the quality of any particular structure.

By: Sean

Sean — Mon, 01 Jun 2009 21:44:38 +0000

Personally, when I think of high impact Nature, Science and Cell come to mind, but note taken that higher impact does not necessarily mean worse structures.

Also thanks for all the work in putting the site together! I really appreciate tools that have such broad appeal to the crystallographic community. I will be looking forward to the addition of the latest structures.

Do you have any plans to publish a paper giving further details on the expanded independent and dependent variables?

By: Eric Brown

Eric Brown — Mon, 01 Jun 2009 19:30:17 +0000

Thanks for the mention. I’d like to point out, though, that we do only show that the journals Nature, Science, Cell, and Molecular Cell publish structures that are worse that you’d expect, not that all high impact journals publish poor structures. In fact, there are some high impact journals such as JBC, JMB, Proteins, and Biochemistry that publish better than expected structures.

BTW, we are working on updating the latest statistics for the latest structures.