Incredibility Calculator

This program examines the extent that a given set of results is either improbably significant or improbably non-significant.

For a given power level, studies should only be significant a fraction of the time (e.g. 80% if the type II error rate is.20)

Enter in the values from a series of studies below, and then press "Calculate" to determine how incredible the results are.
First, enter the statistical test result from a study into the appropriate box.

Then click "add"

The observed power, and sample size from the study should appear in the "Observed Power" and "Sample Sizes" text areas on the right side of the page.

Repeat this process until all relevant studies have been entered.

Note: You can also directly enter the power and samples in to the text areas.




Chi-square test

Fdf Numdf Denom

Once you have finished adding the study statistics, the observed power and sample size will appear in the following text areas. Additionally, the total number of studies and how many of those were significant will also appear.

If you make a mistake, you can press the "undo" button to clear the last value.

You can also enter the power and sample size manually by separating each value with a comma. You must make sure to to enter the number of significant studies as well.

When all of the power and sample sizes have been entered, press "Calculate" to compute how incredible the pattern of significant and non-significant results is.
Power of Studies

Observed Power
Study Sizes

# of significant studies: Total # of studies:

The incredibility of the results are shown in this area.

The first row displays the average power of the studies. The column next to it shows the average power, weighted by sample size.

The next row shows the probability of obtaining at least as many significant results as you observed. Values less than .10 indicate an implausibly high number of significant results.

The next row shows the probability of obtaining as many non-significant results as observed. Values less than .10 indicate an implsauibly high number of non-significant results.

The last row indicates how many studies in the current sample would be expected to be non-significant.
Results of Analyses
Description Unweighted Weighted
Average Power
Too many Significant Results?
Too many Non-Signficant Results?
Probabilities < .10 suggest highly incredible data

Expected # of Sig. Results