Skip to Main Content

Plots of grades vs. who gets selected

They say that you're in the selectable category, but ever wondered what, really, are your chances of being selected?  Historically, the Science Mission Directorate has not reported on selection rates as a function of peer reviewed scores.  We are very pleased to share this data with you!

The first plot, below, shows the distribution of grades (along the x axis) vs. how many were selected, or not. The upward blueish bars (and positive values on the y axis) represent selected proposals, and the downward redish bars (and the negative numbers on the y axis) represent the proposals that were not selected.

 

The format of the figures is described in the text on the web page. In addition, the numbers from which the prior charts were derived has been attached in tabular format. In the table all of the numbers (both # of proposals selected and not selected) are positive, which makes logical sense. The numbers of not selected proposals in the first graphic are associated with negative values on the y axis only because they are represented by downward bars. Its just an artifact of the dumb program I used to make the figure.

The second plot, below, shows the same data but recast as percentages, so that each vertical bar now shows the percentage chance, overall, that a proposal with the grade along the x axis will be selected.  For example, a proposal with a grade of 'excellent' was (at least partially) funded 94% of the time, with an 'excellent/very good' 80%, 'very good' 45%, and so on.

The format of the figures is described in the text on the web page. In addition, the numbers from which the prior charts were derived has been attached in tabular format. In the table all of the numbers (both # of proposals selected and not selected) are positive, which makes logical sense. The numbers of not selected proposals in the first graphic are associated with negative values on the y axis only because they are represented by downward bars. Its just an artifact of the dumb program I used to make the figure.

The data from which these plots were derived were based on a sample of 1,597 proposals (20 ROSES program elements) reviewed in 2007 and 2008 for which data was readily available. They are believed to be representative of the approximately 4000 proposals reviewed in that time frame.  There are considerable sources of error including the fact that the data across divisions, reviews and in some cases even panels were not normalized in any way.  When only one task within a proposal was funded, some program officers reported the overall (lower) grade for the proposal, whereas others reported the (higher) grade for just the funded task.  The author makes no apologies, he is just happy to be able to share what data he has.

Subscribe to the SARA mailing list: