PISA and invariance
Selected work · Leer en español
Research · Comparability
PISA and invariance
Article in preparation
What measurement has to satisfy for a comparison between education systems to hold.
The question
The analysis examines differences in science-item functioning across education systems. Project documentation distinguishes 90 systems present in the file from the 84 with computer-based science testing covered by this analysis.
A key decision
The adaptive design required changes to the second stage of analysis. Matching on estimated ability allows comparisons among people who received different item sets.
My role
I lead the research question and psychometric interpretation, with AI assistance in pipeline implementation.
An inspectable check
The saved result contains two comparisons with mirt over 1,313 cells per variant. Threshold agreement and maximum difference answer different questions: identical classification does not require identical numerical values.
| Settings | Correlation | Maximum absolute difference | Threshold agreement |
|---|---|---|---|
| Settings matched to mirt | 1.000000¹ | 0.000113 | 100% |
| Pipeline’s own decisions | 0.985755 | 0.035050 | 99.24% |
¹ Rounded to six decimals. The CSV retains additional precision. These are previously computed results; no new calibration was run.
Download the aggregate comparison (CSV) · Method decisions and limits (Spanish excerpt) · Provenance
The analysis covers 84 systems with computer-based science testing, out of 90 in the file. It uses its own calibration, not the OECD’s, and point RMSD estimates without BRR intervals. This comparison checks an implementation; it does not establish invariance or reproduce the official calibration.
Evidence status
- Checked / documented
- The existing verificacion_mirt.rds file contains 1,313 compared cells per variant. With settings matched to mirt, the maximum difference was 0.000113 and threshold agreement was 100%. Under the pipeline’s own decisions, these were 0.035050 and 99.24%.
- Observed in use
- Analysis of assessment files. No institutional adoption or reproduction of the OECD’s official calibration is claimed.
- Pending
- Article in preparation. Point RMSD estimates without BRR intervals, system-level groups and the project’s own calibration (not the OECD’s) limit interpretation. Analyses were not rerun for this page.
Sources: verificacion_mirt.rds and pipeline README. Two aggregate rows are provided here; matched settings do not imply exact equality or a replication of the official assessment operation.