Before conducting a study, there are many important considerations to make.
Your ability to draw conclusions from data is limited by several factors such as reliability, validity, power, etc.
Click on one of the headings below to answer the relevant pre-analysis question.
[+] What is the reliability of my scale/test?
# of Rows = # of People
# of Columns = # of Items
Cronbach's α represents a lower-bound estimate for the reliability of a scale/test composed of different items.
The inter-item correlation represents the average relatedness of the items to each other
[+] How much more reliable will your scale/test will be if you lengthened/shortened it?
|How many times longer is new scale/test?||e.g. if your new scale will be twice as long, enter 2|
|What is the current reliability?||e.g. if Cronbach's alpha = .88, enter .88|
The reliability of the longer scale/test would be:
[+] How much longer/shorter does your scale/test need be for a desired reliabilty?
|Current Reliability?||e.g. the Cronbach's alpha is now = .50|
|Desired Reliability?||e.g. you want the Cronbach's alpha to be = .70|
The scale/test must be times longer
[+] What would the effect size between X and Y be if you took into account the unreliability of the measures?
|Effect Size between X and Y:||e.g. r = .40|
|Reliability of X:||e.g. .70;|
|Reliability of Y:||e.g. .80|
The effect size correcting for unreliability is:
[+] How big of an effect size can you find with your measures?
|Predicted Effect Size between X and Y:||e.g. you believe X and Y have a true correlation of .60|
|Reliability of X:||e.g. .20|
|Reliability of Y:||e.g. .50|
The maxmimum correlation you could observe is:
[+] How big of a(n) effect/sample size/type II error rate is needed to obtain a significant effect?
Leave "Sample size" blank to calculate how small your sample size can be to show the effect at the desired power level
Leave "Beta" blank to calculate the probability you have of detecting an effect size at a given sample size
[+] How many studies will end up detecting an effect?
Total # of Studies That Should Be Significant
[+] How confident can I be in my effect size?
Enter the Effect Size in one of the text areas
Press "Enter" to calculate the variance and confidence interval in terms of other effect sizes
Group 1 sample size Group 2 sample size for correlations, enter the total sample size into just the "Group 1 sample size"
[+] What sample size do I need for a replication study?
Studies powered below 33% are severely underpowered, and failures to replicate them suggest the original effect is too small to treat the original study as informative. Therefore, replication studies should compare their
estimate to what the original study's effect size would be if it were powered at 33%
Enter the total sample size of the original study, the power you would like for your replication, and the type of effect size
Click "Calculate" to determine the sample size your replication study should have
Specify the format of the original effect size to receive the effect size that your replication must show it is significantly greater than
|Original Sample Size:|
|Desired Power for Replication:|
[+] What percent of research findings are true?
The proportion of hypotheses tested by a field that are true effects
The average Power (1-TypeII error rate) of the typical study in the field
The probability of discovering a positive result in a study, when no actual effect exists (Type I error rate)
The total number of studies done in a field. Set as 1 to receive percentages
The bias, or proportion of hypotheses tested that would not have been significant but nevertheless end up presented and reported as such. It is the percent of studies that are significant because of multiple testing, running more participants until the finding is significant, discarding measures that did not attain significance, or other types of researcher degrees of freedom. If all analyses are conducted as planned, then set bias to 0
Proportion of Hypotheses |
Tested that are True
| e.g. if 1/4 of hypotheses tested are
actually examining true effects, proportion = .25
|Avg. Study Power||e.g. Low=.2; High=.8|
|Type I Error Rate (alpha)||e.g. Significance criteria for results (e.g. p < .05 )|
|Total # of studies||e.g. The total number of studies done|
|Bias||e.g. Low = .1; High = .8|
|True Relationship Exists|
Probability of a Positive Finding being True: