Challenging choices: How we make funding decisions
On 1 October, the Simons Foundation Autism Research Initiative (SFARI) announced a new request for applications (RFA). Letters of intent (LOIs) are due no later than 30 November.
It seems timely, therefore, to dust off and update a column that I wrote well over a year ago describing how we, the members of SFARI’s science team, make decisions on research proposals. Although the fundamentals remain unchanged, there have been a number of notable changes as a result of advances in the field and our evolving view of SFARI’s mission and priorities.
The LOI stage:
All applicants are required to send in a LOI. These are short (three-page) statements that are intended to save time — for applicants and SFARI science team members — compared with the effort required for full applications. In the past few years, we have received about 400 LOIs in each RFA cycle. We invite roughly one-fourth of those applicants to submit full proposals.
At the LOI stage, we read applications primarily with an eye toward whether they are in line with SFARI's mission and whether and how they fit in with our existing portfolio of awards.
Each LOI is read by at least two members of the SFARI science team, which includes Marta Benedetti, Marian Carlson, Alice Luo Clayton, Gerry Fischbach, Alan Packer, Julia Sommer and myself. We take advice from outside experts if we need to, but by making this first-level decision in-house, we are able to streamline the process significantly.
Tough decisions, priorities:
Judging from previous years, the vast majority of LOIs will describe interesting work. It would be great if SFARI could fund all of them, but that's simply not possible. For example, a few years ago the LOI requests totaled $310 million and our budget was (and remains) in the range of $18 million for new grants each year. (Investigator-initiated grants make up roughly one-third of SFARI’s yearly budget of $60 million.)
Clearly, we have to make tough choices about which applicants to invite to submit full proposals. We generally like to give principal investigators the benefit of the doubt because we know it can be challenging to describe work in three pages. Then again, if we feel an application is not likely to be competitive for one reason or another, we prefer to tell the applicants early, rather than after they have invested more effort.
Although our decisions at this point are mostly binary (either yes or no to a full application), we do advise many applicants to consider submitting more streamlined full proposals, such as reducing a request for a SFARI Research Award to a Pilot Award. Proposals are judged to some degree on whether the budgets are reasonable given the potential risks and rewards of the research (larger budgets raise the bar for funding), and whether there is funding from other sources. We also work with successful applicants to adjust budgets after the review process.
How we interpret which applications are likely to best support SFARI's mission is of course nuanced and evolves as new results emerge. But there are a few general guidelines that have emerged from previous RFA review panels, and from multiple internal and external discussions, that influence our decisions. I outline those below.
We have supported (and continue to support) a number of large-scale efforts aimed at defining the landscape of autism genetics. These have primarily centered on gene discovery in the Simons Simplex Collection (SSC), a repository of samples and data from families that have one child with autism, and unaffected parents and siblings. Within the next year, the entire collection of approximately 2,800 families will have been analyzed genome-wide at the level of copy number variations and exome sequences, and the data will be made available to the research community. (Whole genomes of a small number of participants with autism will also be completed.)
Prospective applicants in this area of research are encouraged to familiarize themselves with these results, and to think carefully about how their research would complement the efforts. Given our extensive investment in the SSC and the informatics platform to support it, we tend to rank gene discovery efforts in other autism groups less favorably unless there is a compelling reason.
With hundreds of genes implicated, the research community will need to replicate results from the SSC and other collections in larger populations, prioritize genes using innovative methods (bioinformatics and others) and, importantly, integrate all of this information in novel ways to gain a complete picture of genetics’ contribution to autism spectrum disorders.
We receive a large number of applications that propose to use rodent models to explore molecular mechanisms underlying autism spectrum disorders. In September, my colleagues wrote a thorough piece on SFARI’s plans for mouse models. I strongly encourage all applicants who propose to use rodent models to read it, as it summarizes our efforts to prioritize, behaviorally characterize and distribute mouse models. We believe that making well-characterized mouse models readily available is key for attracting top talent in systems and circuits neuroscience to work on autism.
Where's the cohort?
We often receive applications proposing clever new ways of measuring various genetic, biochemical, psychophysical or neuroimaging parameters in people with autism spectrum disorders. Given that autism is a behavioral diagnosis, we strongly encourage efforts to forge links between the behavior and the biology.
The trouble is, the first step for many of these investigators is to assemble a study group. We've learned from previous grants that recruitment and retention is far more difficult and costly than many researchers anticipate. So, unless there is a compelling reason for a new cohort, we recommend that researchers collaborate with existing teams that already have a study group, and pay careful attention to ascertainment biases that can influence the results.
We encourage investigators to use structured cohorts such as the Autism Genetic Resource Exchange, the SSC and the Simons VIP, in which multiple types of data (medical history, psychometric testing, genetic, brain imaging, etc.) can be layered to create a coherent picture, and, importantly, can be shared with the community to accelerate research.
One more thought on cohorts: Genetic discovery is moving fast, and the potential for connecting genotype with phenotype seems especially promising. So our view is that it is no longer sensible to study individuals who have autism without knowing something about their genetic background, or at a minimum having their blood in the freezer for future analysis.
Relevance and reach:
The issues in autism research are complex, and deciding which projects are truly relevant to our mission is perhaps one of our greatest challenges. One might argue that autism is a result of altered brain function, so any project that aims to understand the brain — especially a brain system whose function has been linked to autism — is relevant.
But SFARI's funding is finite, and to fulfill our mission we need to maintain a tight focus. The time seems ripe to bridge different levels of understanding in a push for a more coherent picture of autism from genes to circuits to behavior. For example, if a particular cellular or circuit deficit is apparent in one mouse model of autism, is it generalizable to other backgrounds or genetic models?
Autism is highly heterogeneous, so documenting points of convergence for cells, circuits and systems will be critical. For work in the clinical realm, we favor research that is grounded in biology. Over the long term, we hope this approach will lead to transformative and effective interventions. We recognize the value of other types of work, such as research into improvements of behavioral therapies or assistive technologies, but have generally prioritized studies that elucidate biological mechanisms.
Proposals that focus on reproducing previous results are often at a disadvantage in review panels because, almost by definition, they don’t bring new ideas. But we are aware of the critical importance of replication, especially for potential clinical interventions. As an example, a few teams have shown that correcting the genetic lesion even in adulthood can reverse the characteristic symptoms in various genetic mouse models of autism.
These are potentially paradigm-shifting experiments, as they suggest that early neurodevelopmental defects may be repaired or compensated for later in life. But there is an urgent need for them to be replicated and extended. And if they cannot be replicated, the ‘negative’ findings need to be swiftly communicated to the research community. In short, SFARI will consider requests that involve replicating and expanding key findings.
We expect SFARI investigators to share renewable reagents and data developed using SFARI funds with other qualified investigators. In the case of several projects for which early release of data is critical, we made funding contingent on pre-publication sharing of results with SFARI investigators. In these cases, we added important safeguards to ensure that the researchers are protected in various ways and can’t be scooped on their own data. In a Viewpoint published in October, Randy Buckner articulates the critical need to share data, and how best to balance this need with protecting the intellectual contributions of the investigators.
We try to balance the needs of new investigators with continuation applications from current investigators. We have always been enthusiastic about bringing in investigators who have a strong track record in another research area and can offer a fresh approach to autism research.
Our referees are a diverse group of scientists who dig into details of the proposals (the study plan, investigator’s track record, quality of preliminary data, data-sharing plan, budget). Some referees have been with us for many years, but each year we add and subtract a few members. In the past few years, we’ve included some SFARI investigators.
After an initial electronic review, in which referees submit short comments (proposals that do poorly in the electronic pre-review are rejected without full discussion), we hold a lively two-day meeting to discuss the grants in person. At that meeting, referees present grant proposals, and we together debate strengths and weaknesses and, eventually, rank each proposal. We discuss conceptually related proposals in groups so that we can make direct comparisons.
We don't divide the committee into subgroups by expertise. All members are present for discussion of all proposals — except, of course, when there are potential conflicts of interest. This approach may be less efficient than that of a specialized subcommittee, but the upside is that it offers a check against overspecialization unduly influencing decisions.
The referees and SFARI staff both have substantial flexibility in suggesting changes to the scope or budget of a grant application. The feedback that we get from referees about this flexibility has been very positive compared with other types of review committees that are less interactive.
About a month after the meeting, we present our recommendations to the Scientific Advisory Board. The board's role is to help judge consistency and rigor of the review process, and to ensure a balance of the overall portfolio in line with SFARI's mission.
Once the Scientific Advisory Board and the Simons Foundation Board of Trustees have approved the decisions, we communicate the decisions to all applicants. We try to give investigators we do not fund a summary of the reasons for the decision. However, we ask applicants to keep in mind that referees are not asked to prepare formal remarks for the applicants, and it can be challenging to summarize the verbal discussion.
In addition to our yearly RFA cycle described above, last year we launched SFARI Explorer Awards. Unlike Pilot Awards granted through the yearly RFA, applications for Explorer Awards are accepted throughout the year on a rolling basis, are for one year only and are capped at $50,000 in direct costs.
Explorer Awards are intended to provide resources to support exploratory experiments that strengthen hypotheses and lead to competitive applications for larger-scale funding from SFARI or other organizations. We review the applications internally with a response time as short as 30 days. What distinguishes both Explorer and Pilot Awards from the Research Awards is that they place less emphasis on preliminary data. We receive five to ten Explorer applications per month, and have funded about 20 percent of them.
We always welcome feedback about our decision-making process. The more specific the feedback, the better. "I can't believe you didn't fund my grant; you guys make lousy decisions," is less helpful than a reasoned critique of some of the criteria I have outlined above.
Also, please keep in mind that a negative decision this round is not a broad judgment about the quality of work emerging from a lab, or a blanket statement about the potential for future SFARI funding. It's simply that the abundance of high-quality grants and a finite budget force us to make tough decisions.