Silicon Invariance
Selected work · Leer en español
Research · AI evaluation
Silicon Invariance
Article in preparation
Can a synthetic population stand in for a human sample in a survey?
The question
The project examines whether synthetic populations preserve structure, comparability and predictive performance against a human reference.
A key decision
Reversing labels tests whether responses change with the meaning of the scale. In four models the profile stayed positive, so they did not follow the reversed labels; in two it reversed, as expected if the model reads the scale. The study kept all six models in the other analyses and flagged every result for the four that did not follow the reversal.
My role
I define the research question and methodological decisions. AI agents assist implementation and analysis; I am writing the final article.
Evidence you can inspect
The probe reversed the response labels for the same 300 synthetic personas across all six models, each built from a real ESS9 respondent. Four correlations remained positive (+0.50 to +0.94); the other two were negative (−0.91 and −0.96), in the direction expected under scale reversal. These are aggregate profiles, not identical individual answers or a measure of general comprehension.
Original study figure (Spanish labels), checked against the six exported records. “Reads/does not read” describes the operational behavior in this probe, not a general model capability.
Download all six correlations (CSV) · File provenance and hashes
The study uses six local models, the PVQ-21 and, as its human reference, the 2018 European Social Survey (round 9). This case presents a methodological control. The confirmatory analyses of structure, comparability and prediction are complete, and all six models failed them, including the two that followed the reversal; the article in preparation will present them with their interpretation.
Evidence status
- Checked / documented
- The existing figure was checked against all six aggregate records in gate.rds, and on Oct 28, 2026 the six correlations were recomputed from the study's saved readings, matching results/gate_test.rds. Raw-profile correlations ranged from +0.50 to +0.94 for four models; the other two were −0.91 and −0.96. Generation and analyses were not rerun for this page.
- Observed in use
- The evidence comes from experimental outputs. The study does not document institutional use of synthetic populations as survey replacements.
- Pending
- Davin is preparing the article. The confirmatory analyses are complete; the article with their interpretation and a shareable technical repository are still pending. Findings are limited to the instrument, models and conditions examined; frontier models through paid APIs were not tested.
Aggregate label-reversal result, using the same 300 synthetic personas for all six models, each built from a real ESS9 respondent. Figure and CSV available on this page. Correlation does not imply identical answers.