Much has been made recently of a supposed lack of reproducibility of biomedical research studies¹. One impediment to reproducible research is a lack of attention to methodological details and rigorous experimental design. SFARI is working to address these problems by encouraging our investigators to carefully consider the design, analysis and interpretation of their studies before experiments begin. The following is a list of such methodological and statistical considerations that we urge you to consider in completing your application.

These are issues that we will be thinking about as we evaluate applications and that we will also encourage our external referees to consider. This list is by no means exhaustive but is designed to capture those issues we feel to be most important and relevant to autism research, many of which are often neglected. We hope this list will help you to prepare a competitive application.

1. How will sample sizes be chosen? Will a power analysis be performed?
   • For further information on the importance of adequately powered studies, see ref 2.
2. How will sample size be defined (e.g., number of mice, number of sections, number of cells)? Is this appropriate?
   • For further information on defining n’s and nested analyses, see ref 3.
3. Are the planned statistical tests appropriate for analysis of this type of data? For example, are the test assumptions about data distribution and variance appropriate for the analysis?
4. Are multiple comparisons planned? If so, consider that statistical adjustments will need to be made.
   • For further information on correcting for multiple testing, see ref 4.
5. Are criteria for excluding data points established? To avoid bias, these criteria should be established prior to data collection.
   • For more information on excluded animals, see ref 5.
6. Will subject/sample assignment to groups and data collection and processing be randomized?
   • For more information on batch effects and the importance of randomization, see ref 6.
7. Will data collection and analysis be performed blind to group allocation?
   • For more information on the importance of blind analysis, see ref 7.

1. At what age(s) will animals be assessed?
2. Will both females and males be used? If not, what is the justification?
3. For experiments in genetically modified animals, will homozygous or heterozygous animals be assessed? Please consider that in many cases, heterozygous animals may more closely mimic the human condition.

1. What background strain of mice/rats will be used? What is the rationale for using this particular strain?
   • For more information on the effects of mouse strain on behavior, see ref 8.
2. Will animals be housed alone or in groups? In an enriched environment? How might this influence the results?
   • For more information on the effects of rodent housing on experimental results, see ref 9.
3. Will behavioral studies be performed during the light or dark phase? Is this appropriate for the species and type of behavior being assessed?
   • For more information on the effects of light cycles on experimental results, see ref 9.
4. Will multiple behavioral tests be conducted in the same group of animals? If so, how might this affect the results?
   • For more information on the effects of repeated behavioral testing in mice, see ref 10.
5. Is maternal care an issue for some genetic mutations? If so, should foster mothers be used to avoid issues that might conflate the assessment of offspring?

1. Antibodies: Have antibodies been validated for use in the assay and species in which you plan to use them?
   • For more information on antibody reliability and validation, see ref 11.
2. Cell lines: What is the source of the cell lines you plan to use? Have these lines been authenticated, and have they been tested for mycoplasma contamination? Are any of the cell lines you plan to use listed in the database of commonly misidentified cell lines maintained by the International Cell Line Authentication Committee and the National Center for Biotechnology Information (NCBI) Biosample?
   • For more information on considerations for reproducible use of cell lines, see ref 12.
   • For more information on considerations for the use of induced pluripotent stem cells (iPSCs), please refer to SFARI’s experimental design considerations for working with iPSCs document, available here.

1. Challenges in irreproducible research. Nature Special Archive Collection
2. Button K.S., Ioannidis J.P., Mokrysz C., Nosek B.A., Flint J., Robinson E.S., Munafo M.R. Power failure: Why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376 (2013) PubMed
3. Aarts E., Verhage M., Veenvliet J.V., Dolan C.V., van der Sluis S. A solution to dependency: Using multilevel analysis to accommodate nested data. Nat. Neurosci. 17, 491–496 (2014) PubMed
4. Noble W.S. How does multiple testing correction work? Nat. Biotechnol. 27, 1135–1137 (2009) PubMed
5. Baker, M. Missing mice: Gaps in data plague animal research. Nature 526, 182–185 (2016) Article
6. Leek J.T., Scharpf R.B., Bravo H.C., Simcha D., Langmead B., Johnson W.E., Geman D., Baggerly K., Irizarry R.A. Tackling the widespread and critical impact of batch effects in high-throughput data. Nat. Rev. Genet. 11, 733–739 (2010) PubMed
7. MacCoun R., Perlmutter S. Blind analysis: Hide results to seek the truth. Nature 526, 187–189 (2015) PubMed
8. Sittig L.J., Carbonetto P., Engel K.A., Krauss K.S., Barrios-Camacho C.M., Palmer A.A. Genetic background limits generalizability of genotype-phenotype relationships. Neuron 91, 1253–1259 (2016) PubMed
9. Reardon S. A mouse’s house may ruin experiments. Nature 530, 264 (2016) PubMed
10. Võikar V., Vasar E., Rauvala H. Behavioral alterations induced by repeated testing in C57BL/6J and 129S2/Sv mice: Implications for phenotyping screens. Genes Brain Behav. 3, 27–38 (2004) PubMed
11. Baker M. Reproducibility crisis: Blame it on the antibodies. Nature 521, 274–276 (2015) PubMed
12. Time to tackle cells’ mistaken identity. Nature 520, 264 (2015) Article