Fixing ARC Discovery Projects

This is a contentious subject, and I’m probably doing this at some personal political risk, but I think it’s a discussion that must be had, and it only happens if someone is brave enougn to kick it off, so here goes. Before I begin, a disclaimer — I’m happy to be corrected about anything written below, particularly if it might improve the transparency of the system and/or promote mature discussion.

The Problem

The problem, as I see it, is a demographic skew in funding that likely comes from many factors, but is one that, in the current super-tight funding environment, threatens to leave us lots of retiring professors, lots of people at DECRA and Future Fellowship (FT) stage, and a wide gulf in between.

The skew is oft talked about amongst more junior researchers, and often claimed to be bogus by those at the top of the system (or ‘anti-meritocracy negativity’), so let me back it with real stats… As my raw data set, I will take the ARC’s own outcome statistics from today (see below):

2015-10-30_2115

The statistics state that there were 10769 participants, combined CIs and PIs, of which 2587 (24%) are female and 8162 (76%) are male.The graph is in terms of CIs, not CIs + PIs, so if you add up the percentages, they should add to less than 100% (and do, see below). This enables you to tease out how many CIs there are if you extract the data from their plot accurately enough. Since I like being precise about these things, I’ve chosen to do this using Datathief III… The results appear below (happy to share actual spreadsheet by direct request):

Results_002

If you add up the percentages measured from the bar graph (Column C), they only add to 93.3% (Cell C24). The missing 6.7% of 10,769 must be PIs, this turns out to 723. Running the percentages extracted on 10,769 participants (Column D) adds to 10,046, with 10,046 + 723 = 10,769.* Column E is Column D recalculated as a percentage of total CIs (10,046) rather than total participants (10,769) — this is vital to getting meaningful data in Column I (see below). Anyway, now that we know the exact number of CIs, we can just pull out all the numbers of funded CIs using the success in band values extracted from the ARC’s own graph (Column F).

Doing so, Column G is the raw number of successful/funded CIs in each gender/age cohort — note this is CI of any position, lead or otherwise, a factor which we will return to further below. In total there are 1788 funded CIs, of which 1310 (73%) are male and 478 (27%) are female. The overall success rate at CI level (not CI + PI) is 17.8%.

Finally, in Column H I calculate the percentage funded in each cohort relative to the total funded and in Column I then look at how this % funded varies from the % of CIs. If this number is negative, then the success rate in your cohort is lower than the overall success rate; and vice versa if it is positive. Note that the value in Column I adds to net zero, as it should: It represents a measure of ‘success’ somehow displaced from one cohort into another relative to what was submitted.

Numbers are nice, but lets look at this in terms of graphs.

Graphs_001

I don’t want to gloss over this, so let’s look at the graphs one by one. The pie charts are percentage of CIs (top) and percentage of CIs funded for each cohort (bottom), starting with young men at North and running around clockwise with increasing age. I’ve highlighted male researcher slices with blue borders and female researcher slices with pink borders (this is naff, I know, please forgive me this one). The big result in these pie plots should be obvious to anyone who works in academia — the male:female ratio is massively skewed. To the ARC’s credit (since I know they do put effort against this), the ratio doesn’t get appreciably worse in the carriage from application to funded grant.

From here, the stats are better viewed as bar graphs, both of which are the same data as the pie charts. Comparing the two bar-plots, the most apparent feature is that the percentage of males with 25+ years PhD is the greatest and it is the most appreciably higher relative to the percentage of CIs. The latter is even more obvious when you plot the difference between the percentage a cohort contributes to the funded CIs and the percentage the same cohort contributes to the CIs applying. As mentioned earlier, a positive value here means your success rate as a cohort is higher than it is for all CIs put together, a negative value means it is worse.

If you look at this graph it conclusively proves what many are complaining about — Younger researchers, both male and female, are actually suffering a lower success rate, in real terms, than older researchers, and the real winners out of this are late career males. Now there’s two important things to bear in mind here that make the story my graphs tell look better than what is the true reality:

1. These stats are for CI in any position only and not for lead CI or sole CI (data unavailable — but see Gaetano Burgio’s excellent article on data-mined lead CI stats for DP16 round for more). I’d love to see a deeper demographic analysis on either of these, but my prediction the truth is that lead CI and sole CI grants will be are overwhelmingly dominated by late career males (see plot from Gaetano’s blog below) — this means they have more cash as they are less likely to share it, and if they do share it, they have more control over it. As such, they will gain accumulated advantage that helps them in the heavily track-record dominated (40%) assessment for this scheme. The ability to be lead CI on two DP projects whilst others have none exacerbates this effect.

Gaetano

2. We are not considering the ‘multiplying’ effect of other funding schemes, such as CoE, LIEF, Laureate, etc. Assuming these have a similar demographic skew, it is highly likely that those with a big advantage via Point 1 above also have more cash in general, further accumulating their competitive advantage in this system. There were several late-career male CIs in today’s results who already hold CoE funding, and now have also got DP money as lead CI to add on top.

But let’s consider the converse for a second now. The younger researchers will have less success as a cohort, probably aren’t sole or lead CI, and so have to get what falls off the table from above. If you average this success rate over time — they are more likely to have stretches without winning ARC funding and, at mid-career level, are less likely to get internal funding as they are too senior to get ECR grants and not senior enough to be politically connected or attract the attention of the upper academic hierarchy and get funds ‘off the top’ or outside announced competitive rounds.

The net result of this is a big problem for Australian science. It is what I like to call the ‘no-Future Fellowship’. It’s what you get after your Future Fellowship when you start your tenured middle- career stage, can’t apply for fellowships any more, and suffer a disproportionately low cohort success rate in the ‘open pool’ contest for Discovery grants (for more, see my other post on grant outcome demographics — and the figure below that comes from it). The net result is, that with much of the spoils preferentially going to the late career males, a gap will form behind them, and when they all retire, that gap is going to mean scientific output in Australia goes backwards. In a sense, we’re engineering our own scientific recession that we will eventually have to have….

Model 2

The problem is now pretty clear I think… so let’s look at:

The solutions

I’d like to now speculate on some ways that we can potentially fix these problems in the Discovery Projects system.

  1. Change the assessment fractions — Currently it’s 40% investigators, 25% project quality and innovation, 20% feasibility and benefit, 15% research environment. In other words, 55% of the assessment comes from criteria where accumulated advantage plays a massive role. I would realign the fractions considerably, making them 65% project quality and innovation, 15% feasibility and benefit, 15% investigators and 5% research environment. I would possibly even toss research environment in the bin, because anything more than a tick-box for whether the project is feasible at the institute proposed is just aiming to skew the assessment in favour of higher-ranked universities (i.e., institutional elitism).In the end, ideas are what really matter in innovation, and the best ones should be supported equally, whether you’re a young researcher with a few papers or a senior professor with a h-index of 1000.
  2. Split the Discovery Projects Scheme into two bands: Discovery Senior and Discovery Junior — There is clearly a need to manage the success rates at cohort level in the data above. One way to do this would be to make the proposal go into a separate scheme if any of the CIs on the proposal are 20+ years post-PhD. Another option would be to do this by number of DPs held within the past 10 years, as soon as this exceeds 3, your proposal goes into a separate pool. Alternatively, one could ‘handicap’ the track-record score for all late career CIs — some would argue that ‘track record relative to opportunity’ should do this, but it’s clear in the data above that this is not working.
  3. Go back to the old system of oz/intreaders and rankings over scores — I’m happy to be corrected, but my understanding of the systems, based on many research office info sessions and corroborated heresay is this. In the old system, the rankings that went to the panel meetings were a complex combination of rankings by different levels of readers, with rankings weighted by how many grants a given reader saw. The benefit of this system is that it removes the bias between one reader and another to a decent extent, and is a little less easy to manipulate by readers who read a small handful of grants.The new system of scores has obvious biases in it. Take two grants, one obviously better than the other. One reader might give them an A and a B as they’re a generous marker. Another might give them a C and D because they’re a hard marker. In a system where scores really count, and aren’t weighted heavily by how many grants you read, those two grants above will suffer very different fates (likely only one of them will be funded). One might ask in a ranking system how you tell an A and B from an A and D if you can only say one is better than the other — well that’s why you have some readers reading a lot of proposals and their rankings having a high weight.I think a lot of researchers who have lived through that shift from the old Ozreader/Intreader days to now will know that the system feels much more random, with your outcomes heavily dependent on your ‘luck’ in getting the right or wrong referee. You can have great comments, and still get nothing in the outcomes. An added advantage of the ranking system above over scores is that it is harder for malicious referees to make soft-kills (i.e., pegging the score down slightly, just enough to spike a competitor’s grant without it being obviously anomalous).Hell, I’d almost say that readers shouldn’t score or rank at all. Leave that to the panel who see enough proposals that they can reliably and meaningfully judge the quality of one relative to another. The readers can make their points via their comments, which should be almost entirely focussed on the project and advice on technical aspects beyond the knowledge of a panel member (and probably would be if we implemented Point 1).
  4. Ban anyone who is a CI in a Centre of Excellence from holding any Discovery grant for the duration of funding to the centre — This one is pretty obvious really. You put a bunch of sharks in a pond full of goldfish, and before long you have lots of hungry sharks and no goldfish.
  5. Make CIs only eligible for holding one DP and not two — This will be a controversial one, but let’s think about it for a second. Each year less than 20% of proposals get funded (this year it was 17.8%). This is not because 80% of them aren’t worth funding, quite the opposite, for the 20% that are funded, there’s probably another 30% that are equally good and only further down due to biases in the scoring system, luck with referees, etc — as everyone knows the distribution of quality in grants has a tall narrow peak and that peak sits under the level where the cash runs out so that only the high-side tail of the peak gets any cash before the budget runs out.If we cut the number of DPs held from two to one per eligible CI would it hurt us that much? Probably not, really. More people with great projects would get funded, and they would be more competitive than they would be in a system where they can’t get money (or get it inconsistently) and others continuously have two grants running, year after year, mostly to do closely allied ideas.On that idea of closely-allied ideas, by funding more people to do only their #1 most innovative project, we actually diversify our funding system into more areas, more viewpoints and more mindsets than we have with some doing their most innovative project and another one they can come up with along side it to bring in money and advance their career. Most of these researchers also teach, and with only one DP, they would have more time available to teach better, improving the strength of the students coming into Australian science and leaving to other countries. Some of these researchers also do outreach, which is under-rewarded given it is essential in convincing the public that they should invest some of their taxes in us doing our technical stuff they can’t understand — with only one DP there’s more time for that too. And finally, there’s more time for researchers to have healthy work-life balance if they aren’t permanently chasing or managing two DP grants. As we all know, healthy balance means more creative thinking, which means more innovation. It would also be significantly more family friendly, which matters a lot to the cohorts that have lower proportional success rate in the graphs above!If, at some point, the ARC budget came back to a level where there was more cash available than worthy projects demanding it (unlikely), then one could always revert to holding more than one grant.
  6. Reduce the amount of paperwork involved in applying for grants — My colleagues overseas can’t believe how long our proposals are. My last one was 100 pages for myself plus 2 PIs. Only 10 pages of it were actual science. This is insanity — it means we waste lots of time writing them, especially when the success rate is 17.8%, and it means many international readers won’t assess them as they take forever to wade through. Bear in mind that this disproportionately affects those who have a lower grant success rate. Those who get grant after grant get money for every time they invest in the forms, whilst those who have to fish for years, do more work — this produces an accumulated productivity advantage that skews the system in favour of those cohorts with a disproportionately high cohort success rate (late career males, inevitably).The ARC needs to have a look at best practice overseas. Rarely have I reviewed a grant that’s more than 20 or 30 pages, even with a half dozen investigators on it. The problem in Australia, in the end, stems from track record being such a massive part of the assessment. It inevitably means a CV arms race, with ever growing detail in the forms as people try to engineer the system in their favour via application policy. In the systems I’ve seen with the shortest grants, it’s more about the idea, and a 2 page CV suffices — in those systems the readers don’t even score the track record, they’re just asked to comment on whether the researchers have the ability to do the research or not. It really should be all that matters in a system valuing innovation: sufficient competence not a giant CV.
  7. Once you reach 20+ years post-PhD your track record is entirely about legacy — A slightly more innovative approach might be to make it such that you have 20 years post-PhD where your track-record is entirely measured by the traditional means — what you produce as published output. After 20 years, that gets completely ignored and it’s all about the quality of the people you produce. This would put the onus on the late career folks to repay their success in past funding with enabling the next generation to do science exchange for some slice of the action. This could be combined with Idea 8 below.
  8. Enable the budget to be weighted by CI even between institutions — A major impediment to collaboration in the DP scheme is that there is a budget that all goes to the lead CI’s host institution. As a collaborating CI, the credit you get at your own institution for a grant with another host institute is near zero — mostly because they don’t see any block funding by you doing so. This provides a disincentive to collaborate. However, if you could split the funding up front, say have a UNSW-ANU collaboration where from scratch 50% goes to UNSW and 50% to ANU (or 40/60 or 80/20 decided by the CIs) then everyone’s happy, and if you need to adjust later, you can transfer funds like happens now.The same could happen with senior CIs under Idea 8. They can come on a grant lead by more junior CIs, with a stipulated percentage specified for them to spend. This would ensure legacy building in the next generation whilst keeping senior researchers alive in the system. It would also prevent bullying by ‘silverback’ lead CIs carrying junior CIs to strengthen their proposal in the track-record arms-race whilst giving them little real control in the research once it gets funded.
  9. Properly qualify ‘opportunity’ in the context of track record in the proposal — If we are going to insist on track record being such a large part of the assessment, then I think we need paperwork sections that enable real opportunity to be properly defined. The key thing that should be declared here is exactly how many tenured staff, postdoctoral staff and Ph.D. students you have working under you. It’s easy to have a massively stellar publication output when you are a senior professor with 4 junior academic hires under your control, a half dozen postdocs, 3 technical staff members provided by your university and a small army of Ph.D. students. If you have 3 Ph.D. students and that’s it, getting even close to the same input out is just completely impossible. Internal funds awarded to your projects should, in principle, be declared also.I’ve heard lots of valiant talk about how track record is always ‘carefully considered relative to opportunity’, and find this mystifying because often the precise information that you need to judge that as a reader is never made available. I’d still argue this problem is best fixed via Item 1 (making track record count much less), but failing that, we need to start doing this properly.

It is now nearly 3am, and I can’t think of any more ideas to round out the 10, but perhaps that’s ok. If you’ve read this far, thanks for paying attention to all this. Improving the depth, breadth and diversity of the scientific community is central to innovation. Having a grant system that is skewed to one cohort and/or largely decided by accumulated advantage destroys this. The data I’ve presented, in my opinion, shows this is clearly a problem in the current ARC Discovery Projects Scheme even before you add on exacerbating influences like certain advantaged cohorts being more likely to be sole or lead CI, hold more than one DP, or concurrently benefit significantly from multi-million dollar Center of Excellence funding.

Fixing this problem is vital to maximising the national innovation potential against available finite resources, and the current government should consider it an urgent problem if they are serious about science and innovation in Australia.

For more reading — see also:

  1. “A Note on the Australian Research Council (ARC) Discovery Program” by Gaetano Burgio.
  2. “Demographics of Destruction — A Bonus Analysis” by myself.

—–

* For full honesty, since I believe in it, the spreadsheet actually gives a total of 10,770 in Cell C28, which is off by 1. This comes about because of rounding issues in Cells D10, D21, D24 and ultimately D28, since I need to deal with x% of 10,769 being a real number, and humans coming as integer units :).

Advertisements

7 thoughts on “Fixing ARC Discovery Projects

  1. Pingback: Demographics of Destruction — A bonus analysis | Fear and Loathing in Academia

  2. I’m 21 years post PhD. 50 years old and not planning on retiring any time soon. Most of the grant funding I got yesterday will be spent on the salary for an ECR. Effectively, she is using my track record to get funding. it is true that the Australian funding system is very track record weighted. This allows the researchers to propose the topics to be researched. The US and I think European systems are much more based on calls for proposals on particular topics, which are driven by bureaucrats and senior researchers on advisory panels driving the agenda. It is a pity that the ARC dropped the DORA fellowships.

  3. Thanks for doing all this work. Thanks also for the suggested fixes. I grow increasingly concerned that academia pushes us toward the production of conservative knowledge. Your analysis above does nothing to dissuade me about this. I also noted, although haven’t looked at it statistically as you have above, that the “economic performance and productivity” category seemed to outweigh all the others this year. I haven’t checked to see if this is a similar pattern as previous years, but it’s another dire sign to my mind.

  4. Adam Great analysis of the ARC black box. If you controlled for discipline would engineering / hard science scoop a disproportion of Discoveries ? These disciplines are heavily over-represented by males.

  5. Thanks for this article. It is not only discouraging to see so many proposals by career-young researchers not get funded year after year, but also the weight that universities put on obtaining these grants for tenure/promotion purposes. It can be very demotivating.

    For your suggestion 2) Of course, this setup used to be the case around 8 years ago, where there was an explicit pool of DP funds for ECR-only teams. This setup was probably the only reason I got funding in those days, and it proved to be a huge boost to my research and career. I am not sure why it was dropped, and would be more than happy to see it be reinstated.

    For your suggestion 4) Unless you also combined this with reducing the number of DP grants allowed to be held concurrently, this would just result in the more established researchers walking away from CoE’s and focusing on 2 DPs instead. Obviously there is some needed interplay between your suggestions here.

  6. Pingback: Research grant inequity a ‘worrying trend’, experts say | Cameron Scott

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s