Measurement, observation and evidence quality

By UK A Level Psychology · independent study · myAudience

myAudience original rendered educational illustration. Image: illustration, not a documentary photograph. · Generated for this isolated pending curriculum pack; no third-party artwork licence claimed.

Measurement, observation and evidence quality

Reliability concerns consistency of measurement under relevant repeat conditions. Validity concerns whether the evidence supports the intended interpretation. A scale can consistently measure willingness to please a researcher rather than private agreement; reliability therefore does not guarantee validity. Face validity asks whether a measure appears to assess the intended construct. Concurrent validity compares it with a suitable current measure of that construct. Ecological validity concerns relevance to behaviour in everyday contexts, and temporal validity concerns whether an account generalises across time. No single positive correlation certifies all of these properties.

Test-retest reliability compares scores across occasions when the measured characteristic is expected to be sufficiently stable. A change could reflect measurement inconsistency or a real change, so choose an interval appropriate to the construct. Inter-observer reliability compares independent coding of the same behaviour. Categories need operational definitions, a coding manual and training. In a hypothetical discussion study, an agreement event might mean an audible endorsement of the stated majority answer; silence is not automatically agreement. Event sampling records occurrences of specified events, whereas time sampling records behaviour at planned intervals. Time sampling may miss brief events and should not be reported as a count of every occurrence.

Observation can be naturalistic or controlled, covert or overt, participant or non-participant. These dimensions answer different questions. A non-participant can observe a controlled task overtly; the labels are not exclusive alternatives in one list. Overt observation may change behaviour, while covert observation creates consent and privacy issues. Questionnaires can collect standardised closed responses efficiently, but wording and social desirability can distort answers. Open questions and unstructured interviews can reveal unanticipated explanations, though they require systematic interpretation and make comparisons harder. A leading question supplies an interpretation rather than neutrally measuring it.

Qualitative data preserve meanings expressed in words; quantitative data express amounts or categories numerically. An interview is not inherently qualitative if it asks for a number; an experiment can also collect qualitative descriptions. Content analysis turns material into systematically defined codes. Coding the occurrence of 'agreement with a peer' in a hypothetical transcript yields a quantitative frequency, but the code is an analytic choice. Keep unclear cases explicit and check coding independently. Removing thematic analysis from the revised list does not remove content analysis; these drafts teach defined coding without counting the removed method as a revised requirement.

Primary data are collected for the current investigation; secondary data were collected by someone else or for a different purpose. A case study provides intensive evidence about an individual or small bounded case, often through multiple methods. It can reveal a possibility and generate hypotheses without estimating a population frequency. Meta-analysis statistically combines results from relevant studies, ordinarily using specified inclusion criteria and effect sizes. Publication bias, overlapping samples and non-comparable measures can distort the combined picture; increasing the number of reported studies does not automatically remove a shared design problem. These are methodological possibilities, not claims about an invented review.

The public/private agreement illustration makes a validity problem visible: observable agreement is not direct access to belief. A thought bubble is an explanatory convention, never an observation. To assess private acceptance, use a separately justified, confidential later response and consider demand, forgetting and changing reasons. Peer review may ask whether the operational definition matches the claim and whether alternative coding changes the result. Economic implications should also be measured rather than assumed: a more elaborate interview may retrieve additional useful details but consume more staff time. Evaluating both benefits and costs prevents a simplistic 'more data is always better' conclusion.

Worked example: Hypothetical coding of eight utterances: observer A records agree, agree, disagree, neutral, agree, neutral, disagree, agree. Observer B records agree, neutral, disagree, neutral, agree, agree, disagree, agree. Calculate simple agreement and propose a specific improvement.

They match on utterances 1,3,4,5,7,8: six of eight. Simple percentage agreement is 6/8×100=75%.

Disputes concern neutral versus agree at utterances 2 and 6. The manual should distinguish acknowledgment from explicit endorsement with worked borderline examples.

Train on different material, then independently code a new pilot sample. Do not merely negotiate these two disputed labels and call the edited scores independent reliability.

Percentage agreement does not correct for agreement expected by chance or validate the underlying construct. A high figure alone cannot establish that private belief was measured.

Answer: Agreement is 75%. A clearer neutral/endorsement rule and independent recoding can improve consistency; construct validity still requires an argument connecting the codes to the intended psychological claim.

Check your understanding: Transfer: a researcher repeatedly obtains exactly the same 1–5 agreement ratings, and the ratings correlate with another approval questionnaire. Evaluate the claim that the measure validly records internalisation. Suggest a complementary observation and a weakness it retains.

Answer: Stable ratings support consistency if genuine stability is expected; concurrent agreement helps only if the comparator is valid for private belief. Both could measure socially desirable answers. Add a confidential delayed judgement without the original peers, but acknowledge demand, memory and context changes. Converging methods strengthen an argument rather than proving access to a mental state.

Why: Indicative rubric: distinguish reliability from validity; explain the common-method alternative; propose an operational private-response measure with its limits. Credit precise category improvement and reject the inference that repeatability or surface plausibility alone establishes internalisation.