Reliable coding needs a shared rule
By UK GCSE Psychology · independent study · myAudience
myAudience original generated educational concept illustration; fictional examples. Image: illustration, not a documentary photograph. · Original myAudience illustration; no affiliation or endorsement by an awarding body.
Reliable coding needs a shared rule
Observation records specified events or behaviour using a defined unit and rule. An observer cannot directly see another person’s private experience; recording an action and inferring a motive are different steps. Our illustration uses fictional toy-object events to make coding visible without collecting personal information. A code such as R or B should have an explicit meaning that another coder can apply. If the question concerns an experience or interpretation, an observed count alone may not answer it.
A coding scheme needs boundaries. Suppose “red object” and “warm-coloured object” are both available codes: one event could fit both, so disagreements may come from overlapping definitions rather than careless coders. A repair could use mutually exclusive colour categories with a stated rule for mixed-colour objects and an “unclear” option where appropriate. The unit must also be fixed: one object presentation, one interval or one action. Otherwise a difference in totals can reflect different units rather than disagreement about the same event.
Interobserver reliability concerns consistency between observers applying the scheme. In the fictional table, coder A assigns R, R, B, B, R, B, R, B; coder B assigns R, R, B, B, B, B, R, R. They match on events one, two, three, four, six and seven, and disagree on five and eight. Raw percentage agreement is (6 / 8) × 100 = 75%. Disagreement is (2 / 8) × 100 = 25%. The denominator is eight paired events, not sixteen separate entries.
This simple agreement measure does not adjust for agreement expected by chance. If nearly all events received one code, two coders might agree frequently without discriminating well between categories. Nor does agreement establish validity: both can apply the same mistaken rule consistently. A reliable count of red objects is not automatically a valid measure of enthusiasm, understanding or emotion. The interpretation needs its own justification. These examples explain why a high percentage is an informative check, not a universal quality threshold or psychological finding.
Method choice follows the question. An interview can explore what a person reports experiencing and allow clarification, although interviewer influence and the participant’s willingness or accuracy remain limitations. A questionnaire applies common prompts efficiently but may restrict nuance or be misunderstood. Observation can address visible actions without relying entirely on recall, yet expectations and reactivity can influence coding or behaviour. Combining methods can reveal differences in what they measure; agreement between them is not guaranteed and should not be forced.
The table deliberately has eight separate event columns and two aligned coding rows, with six ticks and two crosses. R means red object and B blue object; these are fictional category records, not judgements about people. The calculation verifies how the agreement was produced. The warning about a shared wrong rule highlights validity as a separate question. Neither the table nor its percentage shows that real observers were trained, that the rules were ethically applied or that a psychological explanation has been confirmed.
Evaluated model response to the transfer: I would repair the overlapping categories by defining one mutually exclusive code per event, stating the observation unit and a rule for ambiguous objects before coding begins. Two coders should apply the same scheme independently to the same fictional events. Their six matches out of eight give 75% raw agreement, not six out of sixteen. Disagreements on events five and eight would be useful for locating unclear boundaries, but changing the rules after seeing desired results should not be concealed. A revised scheme would need a new check rather than assuming that discussion alone proves reliability.
Both coders could still agree using a rule that measures the wrong construct. For example, colour coding cannot establish how someone felt about an activity, so a questionnaire or interview may better address reported experience. An interview allows clarification but may introduce interviewer influence; a questionnaire standardises questions but may miss nuance or be interpreted differently. Observation remains useful for defined visible actions. I would select the method for the evidence the question requires, explain self-report and observation limitations, and avoid interpreting agreement as truth. The 75% figure describes this invented table only; it is neither a prescribed universal standard nor evidence about a real person.
Worked example: Coder A: R, R, B, B, R, B, R, B. Coder B: R, R, B, B, B, B, R, R. Compute raw agreement and distinguish reliability from validity.
Compare paired events 1 through 8, not sixteen separate entries.
Matches occur at 1, 2, 3, 4, 6, 7: sixmatches.
Disagreements at 5 and 8: two.
Raw agreement=(6/8)×100=75%; disagreement 25%.
The simple rate is not chance-adjusted; shared wrong interpretation can still produce agreement.
Answer: Raw agreement is 75% for this fictional table only. It describes consistency, not valid psychological inference or a universal acceptability threshold.
Check your understanding: Repair overlapping red/warm-colour categories, evaluate shared wrong-rule agreement, and compare observation with interview/questionnaire when the question concerns reported experience.
Answer: Define exclusive categories, a fixed event unit and ambiguity rules. Shared errors can agree without validity. Interviews allow clarification with influence risks; questionnaires standardise but constrain nuance. Full evaluated response is embedded in the last two paragraphs.
Why: Distinguish what can be observed from what is reported; correctly interpret six of eight paired matches and raw 75%agreement.