A practical guide

It combines the results of several studies into one estimate, weighting each study by how precise it is. A large, tightly-estimated study pulls the average harder than a small, noisy one.

That is the entire mechanical idea. Everything else — which effect measure, which model, which diagnostic — is about making sure that average means something.

Start from what the studies report.

• Events out of totals in two arms → risk ratio (easy to explain), odds ratio (better statistical properties, routinely misread as a risk ratio), or risk difference (gives you the number-needed-to-treat directly).
• Means and standard deviations in two arms→ mean difference if every study uses the same scale; standardised mean difference (Hedges' g) if they do not.
• Correlations→ Fisher's z, back-transformed for reporting. Pooling raw correlations understates the uncertainty.
• One arm only → proportions (logit or Freeman-Tukey) or incidence rates.
• Anything already computed — hazard ratios, regression coefficients — → the pre-computed option, with the effect and its standard error.

Ratio measures are pooled on the log scale and back-transformed. That is why a forest plot of odds ratios has 0.5 and 2 the same distance from 1.

Equal-effects (often called fixed-effect) assumes every study is estimating one identical true effect, and all the variation you see is sampling noise.

Random-effects assumes the true effect varies between studies — different populations, doses, follow-up lengths — and estimates how much it varies (that is τ²).

In practice, choose random-effects unless the studies are near-identical replications. Real reviews almost never are. The choice matters most when studies differ in size: equal-effects lets the biggest study dominate, while random-effects gives smaller studies more say.

τ² is the estimated variance of the true effects across studies, in the units of your effect measure. It is the one that tells you how much the effect really differs.

I² is the percentage of total variation attributable to that real difference rather than sampling error. It is a ratio, so it rises as studies get bigger even if τ² is unchanged — a high I² among enormous trials can represent a difference too small to care about.

The Cochrane rough bands (below 30% low, 30–60% moderate, 50–90% substantial, 75–100% considerable) deliberately overlap, because they were never meant to be a threshold.

The most useful number is often neither: it is the prediction interval, which says where the true effect of the next comparable study would fall. If that interval crosses the null while your confidence interval does not, you have found something important about your evidence base.

Usually yes. The standard random-effects confidence interval treats τ² as if it were known exactly, when in fact it was estimated from a handful of studies. Hartung-Knapp accounts for that extra uncertainty and switches the reference distribution from z to t.

The result is a wider, better-behaved interval. With few studies the difference can be large — which is the point. It occasionally produces a narrower interval than the standard method; that is a known quirk, and a reason to report which method you used.

Subgroups split studies into categories and compare the pooled effect in each. Natural for categorical characteristics, and easy to present.

Meta-regression fits the characteristic as a predictor. Necessary for continuous moderators such as mean age or dose, and it uses the whole range of the variable rather than chopping it into bins.

Both are observational analyses across studies. Studies were not randomly assigned to subgroups, so a difference between groups can reflect anything else those studies share. And both are badly underpowered: the usual rule of thumb is at least ten studies per moderator, which most reviews cannot meet.

Treat every moderator analysis you did not pre-specify as hypothesis-generating, and say so in the paper.

They test for funnel plot asymmetry — whether small studies report systematically different effects from large ones. Publication bias is one explanation for that. Genuine heterogeneity, poorer methodology in smaller studies, and plain chance are others.

With fewer than ten studies these tests have almost no power, so a non-significant result is not evidence of no bias. Nothing in the statistics can detect a study that was never written down.

Contour-enhanced funnel plots help: if the missing region is exactly where results would have been non-significant, selective publication is the natural reading. If the gap straddles the significance boundary, heterogeneity is more likely.

Trim-and-fill and PET-PEESE are sensitivity analyses, not corrections. Report them alongside the primary estimate, never instead of it.

REML is the general-purpose recommendation and the default here: near-unbiased and efficient across most realistic scenarios.

DerSimonian-Laird is the historical default and still the most reported. It is non-iterative and fast, but underestimates τ² when heterogeneity is large.

Paule-Mandel behaves better than DL for binary outcomes and is what several methodological reviews now recommend for them.

Sidik-Jonkman is robust when heterogeneity is substantial; ML is efficient but biased downwards with few studies; Hunter-Schmidt is common in psychology and known to be downward-biased.

If the choice of estimator changes your conclusion, that is itself a finding — report the sensitivity rather than quietly picking the estimator you prefer.

Ratio measures are undefined when an arm has no events. The default fix adds 0.5 to every cell of the affected study, which works but biases the estimate towards no effect and inflates its variance.

For rare events with balanced arms, the Peto odds ratio needs no correction at all and performs well. The risk difference is also defined at zero, though it has its own problems when baseline risks vary.

Whatever you choose, run it both ways and report whether the conclusion moved.

PRISMA 2020 expects, at minimum:

• the effect measure and the model, with the τ² estimator named;
• the pooled estimate with its confidence interval;
• heterogeneity statistics and, increasingly, a prediction interval;
• which subgroup and moderator analyses were pre-specified;
• how publication bias was assessed;
• a sensitivity analysis;
• the software and version used.

The report this app generates covers all of these. What it cannot supply is the risk-of-bias assessment for each included study and a GRADE certainty rating for each outcome — those are judgements, and they belong to the review team.

You have a questionnaire and a theory about it: these five items measure anxiety, those five measure depression. A factor analysis says what the correlations between all ten items should look like if that theory is true — items measuring the same thing should correlate more with each other than with anything else.

The model is then fitted, and the correlations it implies are compared against the ones actually in your data. Fit indices are summaries of that comparison. That is the entire mechanical idea.

Which means a confirmatory factor analysis can only ever fail to reject the model you brought to it. It cannot tell you the right number of factors, and it cannot tell you that no other model fits equally well.

The distinction is not about the software. It is about where the structure came from.

• Confirmatory— the structure came from theory, from the instrument's published validation, or from a different sample. You are testing it here.
• Exploratory — the structure came from looking at this data. Fitting it here and reporting the fit is circular: the model was chosen to fit, so of course it fits.

Deriving a structure from your data and then confirming it on the same data is the most common way this analysis gets misused. If that is where you are, either split the sample in two, or report the analysis as exploratory and say so.

This is why the workbench asks you to build the factor structure yourself and only guesses at which columns are items. The structure is the hypothesis, and it is not ours to invent.

There is no single number, and the rules of thumb you will have heard (“200”, “ten per item”) are weak because what actually matters is how well-determined the model is: strong loadings and three or more items per factor need far fewer respondents than weak loadings and two.

The practical version: aim for at least ten cases per estimated parameter, treat five as the floor, and read standard errors from anything smaller as optimistic. The results page shows this ratio and warns you when it is low.

Small samples do not usually produce obviously wrong answers. They produce plausible answers with intervals too narrow to believe, and occasionally a negative variance estimate — a Heywood case — which is the model telling you it has run out of data.

Neither, on its own. They are measuring different things and disagreeing is informative rather than a problem to be resolved.

• CFI and TLI compare your model against one where nothing correlates with anything. If your items are only weakly correlated to begin with, that baseline is easy to beat, and these indices look worse than the model deserves.
• RMSEA is misfit per degree of freedom, so it punishes small, simple models. With fewer than about ten degrees of freedom it is genuinely unstable and can look poor for a well-specified model.
• SRMR is the average correlation the model failed to reproduce. It is the most directly interpretable of the four, and the one to look at first when the others argue.

When they disagree, stop reading indices and look at the residual correlations. “The model reproduces every correlation to within 0.04 except items 3 and 7” is a real finding. “CFI = 0.94” is not.

And treat 0.95 and 0.08 as conventions, not as a pass mark. They come from one simulation study on continuous, normally distributed data, and they were never meant to be applied as a threshold.

Sometimes, carefully, and never in a long sequence.

A modification index says how much chi-square would fall if one fixed parameter were freed. It is computed on your sample, which means the largest ones are partly real misfit and partly noise — and freeing them one after another fits the noise beautifully.

The rule that keeps you honest: add it only if you can name the reason before you look at the number.Two items with near-identical wording, the same question asked at two time points, a shared response format — those are reasons. “It improved fit” is not.

Cross-loadings deserve more suspicion than correlated residuals. An item that loads on two factors is usually an item that measures two things, and the fix is in the instrument, not the model. That is why the workbench will apply a correlated residual for you but leaves cross-loadings for you to make in the factor builder deliberately.

Whatever you add, say in the write-up that the final model was modified post hoc, and report the fit of the original model too.

Omega, if you have fitted a factor model — which here you always have.

Alpha assumes every item measures the factor equally well. Your loadings table shows you whether that is true, and it almost never is. When it is not, alpha understates reliability. Omega uses the loadings you actually estimated and makes no such assumption.

Report both anyway. Readers and reviewers expect alpha, and the two numbers are usually close enough that showing them together costs nothing and pre-empts the question.

Two warnings. A coefficient near 0.95 is not better news than 0.85 — it often means your items are paraphrases of each other rather than that the construct is well measured. And if any item is reverse-keyed and you have not rescored it, both coefficients are meaningless; the workbench flags this as mixed-sign loadings rather than quietly reporting a reliability of 0.05.

The right to compare groups. Without it, a difference in scores between men and women, or between two countries, could just as easily be a difference in how they read the questions.

Each level licenses something different:

• Configural — the same items load on the same factors in every group. The instrument is measuring the same constructs, but nothing is comparable yet.
• Metric — the loadings are equal. You may now compare correlations and regression coefficients involving the factor across groups. You may not compare means.
• Scalar — the intercepts are equal too. Now you may compare latent means. This is the level most cross-group research actually needs, and the level at which instruments most often fail.
• Strict — residual variances are equal as well. Required to compare observed sum scores rather than latent means. Frequently too strong to hold, and often not needed.

Judge each step by the change in CFI, not by the chi-square difference test. With a few thousand respondents that test rejects constraints that make no practical difference at all. A drop of more than 0.01 in CFI is the usual threshold.

Failing at scalar is a finding, not a failure. It says the groups use the response scale differently — and that is worth reporting rather than working around.

Take it seriously. At that level the two sets of items are behaving almost identically, and calling them separate constructs is a claim your data does not support — whatever the theory says.

The check is to fit a model that merges them into one factor and compare the fit against the two-factor model. If fit barely changes, the simpler model is the honest one. If fit drops sharply, the factors are distinct despite the correlation, and you can say so with evidence.

Average variance extracted is the other angle: if the variance a factor shares with its own items is smaller than the variance it shares with the other factor, the two are not discriminating. The workbench reports AVE per factor for exactly this comparison.

The conventional alarm is r ≥ 0.85, but this is a judgement about your constructs, not a threshold to clear.

Strictly, they are ordinal: the distance from “disagree” to “neutral” is not guaranteed to equal the distance from “neutral” to “agree”. In practice, treating them as continuous works well enough that most published work does it.

The line most people draw:

• Five or more categories, roughly symmetric responses — treat as continuous, and use robust ML so the standard errors survive the non-normality.
• Four or fewer categories, or badly skewed responses — use an ordinal estimator. Robust WLS is the standard choice.

The trade-offs are real: the ordinal estimators model the response thresholds properly and give larger, more honest loadings for coarse items, but they are much slower, they cannot use full-information maximum likelihood for missing data, and they need larger samples to behave.

If it matters to your conclusion, fit both and say so. If the two agree, the question was never important; if they disagree, that is worth a sentence in the paper.

Three things. It fits every equation at once, so a variable can be an outcome in one place and a predictor in another. It lets a variable be latent — measured by several columns, with the measurement error separated out. And because the whole system is estimated together, it can put a confidence interval on a quantity that spans two equations, such as an indirect effect.

The classic mediation set-up shows all three. X predicts M, and both predict Y. Run that as two separate regressions and you get the coefficients, but nothing that tells you how uncertain their product is. Run it as one model and you do.

What it does not give you is licence. Every arrow is an assumption you brought with you, and the fit indices cannot tell you whether it points the right way.

Three numbers, and the relationship between them.

• Indirect effect — the product of the coefficients along the route. How much of X reaches Y by going through M.
• Direct effect — what is left of the X to Y path once M is in the model.
• Total effect — the two added together, which is the association you would have seen if you had never measured M.

Read the intervals rather than the p-values, and read the indirect effect against the direct one rather than on its own. An indirect effect that is significant but a tenth the size of the direct path is a different story from one that carries the whole association, and only the comparison tells you which you have.

The older language of “full” and “partial” mediation is best avoided. Full mediation means a direct path that failed to reach significance, which is a statement about your sample size as much as about the world.

Because it is a product of two coefficients, and products are not normally distributed even when their factors are. The distribution is skewed, so a symmetric interval of the form estimate plus or minus two standard errors is the wrong shape — it tends to be too narrow on one side and too wide on the other.

Bootstrapping refits the model on a few thousand resamples of your data and reads the interval off the resulting distribution directly. It assumes nothing about the shape. This is why the bootstrap intervals in the results table sit asymmetrically around the estimate, and it is why essentially every journal now expects one for a mediation claim.

Use 1,000 resamples while you are still deciding on the model and 5,000 for the run you report. Percentile intervals are the conservative default; bias-corrected intervals have more power but can be liberal in small samples.

No — it means the model had nothing left to be wrong about. If every variable is connected to every other one, the model has as many parameters as the data has variances and covariances, so it reproduces them exactly whatever the coefficients turn out to be. CFI comes out at 1.000 and RMSEA at 0.000 by construction, not by merit.

This is the normal state of a three-variable mediation model, and most published mediation analyses are in it. It is not a problem: the individual paths and the effect decomposition are still meaningful, and they are what you came for. What you must not do is report the fit indices as evidence that the structure is correct.

To test the structure itself, leave something out. A model that says X reaches Y only through M — no direct path — has one degree of freedom, and its fit is then a real test of that claim.

It means the data are consistent with it. So, usually, are several models that say something quite different.

For almost any model you can write down, there exist equivalent models — with arrows reversed, or a mediator and an outcome swapped — that reproduce the covariances exactly as well and therefore have identical fit on every index. No amount of fitting distinguishes them, because they make the same predictions about the only thing being fitted.

What distinguishes them is outside the data: the temporal order in which things were measured, a randomised manipulation, and the theory. If your mediator and outcome were measured in the same questionnaire at the same moment, the arrow between them is a decision you made, and the results section should say so.

The second assumption behind every path is that there is no unmeasured common cause of the two variables it connects. Nothing in the output tests that one at all.

It depends on whether the column measures the thing exactly. A treatment condition, an age in years, a count — those are measured without meaningful error, and a latent variable would add nothing.

A questionnaire scale is different. Averaging ten items into one column and treating it as the construct assumes the average is the construct, and it never quite is. That measurement error attenuates every path the variable is on, usually downward: real effects look smaller than they are, and sometimes disappear.

Modelling the items as indicators of a latent variable separates the shared signal from the item-specific noise and corrects for that. The price is parameters — a latent variable needs at least three indicators to be identified on its own, and the whole model needs a larger sample.

A reasonable rule: use latent variables for constructs measured by multi-item scales, use the columns directly for everything else, and if your sample is too small to afford the measurement model, say in the limitations that the paths are attenuated.

More than the rules of thumb suggest, and the honest answer depends on the effects you are trying to detect rather than on the model alone.

The old advice — ten cases per estimated parameter, or a flat 200 — is a starting point rather than a criterion. The workbench shows you the cases-per-parameter figure and warns below five, but that is a check against obvious over-reach, not a power calculation.

Two things push the requirement up sharply. Indirect effects are products of coefficients, so they are small, and detecting a small effect takes a large sample — a mediation model that comfortably estimates its individual paths at n = 150 may have little power for the product of two of them. And latent variables with few indicators, or with weak loadings, need considerably more data than the parameter count implies.

If the indirect effect is the point of the study, treat n = 200 as a floor rather than a target, and report the interval rather than leaning on whether it excluded zero.