Home/Work/Case study

Academic research · Instrument development · Psychometrics

Full-cycle development of a health literacy scale

No validated Russian-language instrument existed to measure health literacy in school-aged children. With Moscow City University, we built one in five stages. I acted as the psychometric research consultant: advising on how to build the scale, managing the process, and carrying out the data analysis and the results.

54items in the final instrument

9scales in three sections

1,095pupils in the final validation study

5development stages, from expert review to thresholds

The study

A model, then a measure.

The conceptual model came from a review of international ways of measuring health literacy in teenagers. More than 50 published instruments were screened.

What the instrument covers

Three functional sections — autonomy, motivation and practice — across four topics: hygiene, daily routine, nutrition and physical activity. Unlike most comparable tools, it also measures motivation to look after health.

My contribution

I was the psychometric research consultant. I advised on how to build and validate the scale, managed the development process, and ran the data analysis in R and prepared the results. We invited other experts in the field to work on the content of the items. I am second author of the 2023 article, with Elena Nekhorosheva and Daria Kasatkina.

Building it

Shorter, and more reliable.

Each pilot cut weak items and reworded unclear ones, with experts reviewing every round. Eleven draft scales became nine.

Item reduction

Number of items at each stage

Small pilot, N = 92120 items

Large pilot, N = 31370 items

Final study, N = 1,09569 items

Final instrument54 items

The last cut, from 69 to 54 items, came from a second expert review that removed near-duplicate wording. Each final scale has six items.

Reliability

Cronbach’s alpha of each scale in the final instrument

Autonomy

Access to information0.81

Trust0.83

Beliefs0.78

Motivation

Amotivation0.70

Intrinsic motivation0.80

Extrinsic motivation0.76

Practice

Critical appraisal0.86

Communication0.86

Action0.83

Every scale reached 0.70 or higher, and six of nine reached 0.80. Two draft scales, understanding of information and search and interest, were dropped during piloting.

Structure

Testing the structure, factor by factor.

Each section was tested on its own, then all together. Fit was good for autonomy, acceptable for practice, and below the usual thresholds for motivation, which the article reports openly.

Confirmatory factor analysis

The measurement model: 54 items, nine scales, three sections

Autonomy

Access to information0.61items 0.52–0.73

Beliefs0.89items 0.49–0.77

Trust0.88items 0.58–0.79

Motivation

Amotivation—items 0.42–0.64

Extrinsic motivation—items 0.45–0.77

Intrinsic motivation—items 0.49–0.71

Practice

Communication0.92items 0.56–0.82

Critical appraisal0.77items 0.45–0.81

Action0.96items 0.61–0.80

Second column: loading of the factor on its section. Motivation scales correlate: amotivation with extrinsic 0.65, extrinsic with intrinsic 0.18, amotivation with intrinsic −0.90. Autonomy and practice correlate 0.79.

Standardised loadings of the simplest model, in which each item loads on one factor only. In the full diagram on wider screens, each square is one item, shaded by the strength of its loading. Redrawn with English labels from the article. Fit of this model was slightly below conventional thresholds (CFI 0.867, RMSEA 0.042); allowing cross-loadings improved it (CFI 0.958, RMSEA 0.024). The article recommends the simple model for teachers because scores stay a plain average.

Levels

From scores to something a teacher can read.

Scores are the average of six items, from 0 to 3. Response patterns were grouped into five levels, using the content of the answers and the frequency of each response in the sample.

Final study, 1,095 pupils

Share of pupils at each level, by scale

Very lowLowMediumHighVery high

Access to information17%24%34%21%

Beliefs9%27%46%17%

Trust22%28%27%19%

Intrinsic motivation18%24%33%23%

Critical appraisal8%25%23%23%21%

Communication18%34%18%17%12%

Practice10%40%26%14%10%

Amotivation35%49%9%

Extrinsic motivation58%31%

For amotivation and extrinsic motivation, a high level means more of it, not better. Percentages are rounded.

Read together, the scales show gaps. In this sample 63% of pupils were at a high or very high level on beliefs about healthy living, but 24% were at that level on practice. The instrument is built to show that kind of gap, not just a total score.

In a training study

The same scale, used before and after a programme.

The scale was also used in a school training study: about 500 pupils in grades 5 to 10 at Moscow schools, tested twice, with an experimental group and a control group.

What the figures show

At the second test, the experimental group scored higher than the control group on access to information, understanding and critical appraisal. In the experimental group no scale changed significantly between the two tests. In the control group, scores on trust, practice, amotivation and extrinsic motivation fell.

Why it is not proof

The groups differed already at the first test, and the gaps on those three scales were as large or larger then. Only 228 of the 500 pupils could be matched across both tests. The data describe the groups; they do not show that the programme caused the differences.

Experimental group minus control group

Difference in mean score at each test

First testSecond test

−0.6−0.20+0.2+0.6

Access to information+0.27 → +0.19

Understanding of information+0.37 → +0.26

Trust-0.32 → -0.13

Beliefs-0.10 → -0.15

Amotivation-0.23 → +0.54

Intrinsic motivation-0.01 → -0.01

Extrinsic motivation+0.01 → +0.55

Critical appraisal+0.38 → +0.25

Communication-0.20 → -0.14

Practice-0.18 → +0.14

Mean scores on a 0 to 3 scale; a dot to the right of the centre line means the experimental group scored higher. Valid cases: 313 at the first test (207 experimental, 106 control) and 312 at the second (216 and 96). The test in this study had ten scales, including understanding of information, which the final instrument does not retain.

Limits

What the evidence does not show.

Stated up front, so the instrument is used for what it was built for.

The instrument relies on written self-report, so it cannot objectively measure what children can find, understand or do, and it is not suited to younger children.
The motivation section fits the data below the usual thresholds, and the levels were set partly from this sample’s response distribution, not from national norms.
The training-study comparison is descriptive, with unequal groups at the start and large loss of cases.

Publication: Nekhorosheva, E. V., Enchikova, E. S., & Kasatkina, D. A. (2023). Multidimensional Health Literacy Scale: Development and Psychometric Properties (Razrabotka i validizatsiya metodiki mnogomernoy otsenki gramotnosti v voprosakh zdorov’ya). Educational Studies Moscow, 1, 126–160. Original in Russian. doi.org/10.17323/1814-9545-2023-1-126-160

Where it sits in the method: Stage 02 · Build & validate · Stage 03 · Collect the data · Stage 04 · Analyse

← Previous case studyCustomer experience tracking and satisfaction drivers Next case study →How 102 universities support wellbeing beyond campus

Get in touch

Tell me what you are trying to research.

job.enchikova@gmail.com How I work