Significance, test choice and the size of a claim

By UK A Level Psychology · independent study · myAudience

myAudience original rendered educational illustration. Image: illustration, not a documentary photograph. · Generated for this isolated pending curriculum pack; no third-party artwork licence claimed.

Significance, test choice and the size of a claim

An inferential test asks whether observed data are sufficiently incompatible with a specified null model under its assumptions. A p-value is the probability, under that model, of a result at least as extreme as the one observed by the chosen test. It is not the probability that the null is true, that the result was caused by chance, or that a theory is correct. The ASA statement cautions that significance does not measure effect size or importance. A large sample can make a small difference statistically detectable; a small sample can leave an important difference uncertain. Report the actual effect, data quality and limits alongside the decision.

Set a significance level before analysis. At alpha=.05, a procedure with appropriate assumptions aims to limit Type I error, rejecting a true null, to its specified long-run rate. Type II error is failing to reject a false null. Reducing alpha to.01 generally reduces Type I risk while increasing Type II risk for a fixed design and effect. A non-significant result is not proof of equivalence: it might reflect little information, noisy measurement or a small effect. Testing many outcomes and reporting only the one passing.05 increases false-positive opportunity. A preplanned primary outcome and transparent reporting help preserve the meaning of the test.

The sign test uses paired observations reduced to improvement versus decline. Remove zero differences, count positive and negative signs, and let N be the number of nonzero pairs and S the less frequent sign count for the usual two-sided test. Under the null illustration, independent participants have probability.5 of either sign. Significance occurs for a sufficiently small S; this is the opposite comparison direction from tests where a sufficiently large positive statistic is significant. A directional test needs a direction chosen in advance and a result in that direction. A small count in the opposite direction does not support the predicted increase simply because S is small.

Select a test by the question, design and measurement before inspecting which gives a preferred result. For an association, Spearman's rho works with ranks and a monotonic relationship; Pearson's r requires appropriate interval-level data and a linear relationship with relevant assumptions for inference. For a difference with related measurements, Wilcoxon uses ranked paired differences while the related t-test uses interval data and an appropriate difference distribution. With independent groups, Mann-Whitney uses ranks and the unrelated t-test uses interval data with suitable distributional assumptions. The sign test loses magnitude information but can analyse paired direction categories. Ordinary Chi-Squared tests analyse suitable independent nominal frequency counts, not paired changes treated as independent people.

Critical-value tables require the right N or degrees of freedom, alpha, tail and comparison rule. The educational sign-test table in the example is derived by exact enumeration, not copied from. At two-sided.05, N=10 gives critical S=1 and N=12 gives critical S=2; reject when S is at or below the threshold. For tests such as rho or t, read the supplied table's convention instead of applying the sign-test rule. An observed statistical decision is conditional on assumptions, including independence across people and the scoring protocol. It cannot repair volunteer bias or transform a behavioural measure into direct evidence of private belief.

The image's six hypothetical pairs give N=5 after removing the tie, S=1 and a two-sided exact probability 12/32=.375; they do not justify rejection at.05. The larger example below is a separate constructed dataset. This distinction prevents the picture being mistaken for the example's evidence. An extended conclusion integrates effect, uncertainty and causality: a consistent direction under controlled comparison is stronger evidence than a bare threshold, but neither establishes that a teaching method changes every learner or a memory model is uniquely true. Exact probability derivation is enrichment for reasoning; exam answers must still use the supplied test/table instructions.

Original extended evaluation model. The opposite-direction result does not support the predicted improvement. Nine declines and one increase describe a strong imbalance, but a directional hypothesis specifies which imbalance matters before examining the data. Using the smaller sign count without considering direction would answer a different question. For the original two-sided example, ten nonzero pairs with one minority sign give 22/1024, approximately.0215. That probability concerns the specified equal-sign null, not the chance that a learning theory is false. Ties remain part of the descriptive mean even though they are removed from the sign-test N. A statistically significant result is therefore only one component of an evaluation. Rejecting a true null is a Type I error; failing to reject a false null is Type II. At a fixed design and effect, tightening alpha usually trades lower Type I risk for greater Type II risk. It does not retrospectively tell us which error occurred in one study. A non-significant result can reflect insufficient information, and cannot establish equivalent performance. Prespecifying the outcome, tail and analysis prevents a preferred significant result from determining the research question; reporting all planned comparisons exposes selective interpretation.

Test choice also depends on the observations actually collected. Independent groups with ordered confidence outcomes could justify Mann–Whitney, subject to the intended distributional interpretation and assumptions. Independent nominal condition-by-agreement frequencies could justify Chi-Squared when expected counts are suitable. Neither supplies paired before/after observations, so inventing pairs for a sign test changes the unit and null model. Conversely, the repeated recall example permits a paired analysis, but its sign reduction discards magnitudes. The mean change of about 1.33 words conveys information that the sign statistic alone cannot; participant variability and uncertainty must also be reported. Finally, a before/after comparison cannot isolate an intervention from practice, fatigue or another change over time. An original improved design would randomly allocate consenting participants to the intervention or a time-matched active comparator, standardise materials and scoring, and define a participant-level primary outcome in advance. Baseline differences and differential attrition still need checking; random allocation is not random population sampling. Low-burden tasks, withdrawal and transparent exclusions address ethical as well as inferential quality. The defensible conclusion is conditional evidence about the measured outcome under this design, not proof of a mechanism, universal improvement or practical importance from crossing a probability threshold.

Worked example: Hypothetical paired recall differences (after−before) are +2,+1,+3,+1,−1,+2,+1,+4,+1,+2,0,0. A two-sided sign-test table at.05 gives N=10: critical S=1; N=12: critical S=2. Calculate N, S, mean change and the decision, and justify the table row.

Remove both zero differences, leaving ten nonzero pairs: nine positive and one negative. Thus N=10 and S=1, not N=12.

Using the given two-sided.05 row, S≤1 so reject the equal-sign-probability null. Exact enumeration gives 2×(1+10)/2^10=22/1024≈.0215.

The twelve differences sum to 16 words. Mean change across all twelve, including unchanged participants, is 16/12≈1.33 words. Ties are removed for N in the sign test, not silently deleted from descriptive change.

The sign result supports a directional imbalance under its assumptions, not a particular effect magnitude or a cause. A before/after study without a comparator leaves practice as an alternative.

Answer: N=10, S=1, mean change≈1.33 words; the two-sided test rejects at.05. Effect size, study design and sampling still limit the substantive claim.

Check your understanding: Transfer: the researcher prepredicted improvement but observes nine declines and one increase. Another study compares independent groups on ordered confidence ranks, and a third counts agree/disagree in independent participants by condition. Choose defensible interpretations/tests; explain one failure of treating all three as paired sign tests.

Answer: The opposite-direction imbalance cannot support the prepredicted improvement. Independent ordered confidence groups suggest Mann-Whitney with appropriate assumptions; independent condition-by-agreement counts suggest Chi-Squared with adequate expected counts. Their observations are not repeated measurements paired within people, so forcing a sign test invents pairing and changes the question.

Why: Indicative rubric: credit tie handling, correct critical comparison, distinction between significance and magnitude, tail/direction reasoning and scenario-specific test choice. A label alone is inadequate: state association/difference, relatedness and level of measurement. No significance decision is licensed without the needed observed statistic/table.