Validation & transparency report
What we tested on our CYP2C19 caller, how it did, and what we still do not know.
This is a transparency report on one part of our pharmacogenomic engine: the CYP2C19 star-allele caller, checked against a defined, synthetic reference set. It is not a claim of clinical validation, and it does not cover the rest of the product.
What was tested
A single gene, a small and clearly bounded reference set, run through the real engine.
Gene
CYP2C19 only
No other gene is in this comparison.
Input
12 synthetic, truth-anchored array fixtures
Fabricated 23andMe-format files engineered to encode a cited reference diplotype. Not real Coriell or GeT-RM reference-material arrays, and not a real customer genome.
Truth source
Pratt et al. 2010, Table 2
GeT-RM/AMP collaborative reference-material characterization.
Engine pin
47e0d0585
The exact commit under test.
The result
We report the result two ways on purpose: the engine's raw output string against the GeT-RM diplotype, and the same comparison after aligning notation between the two. Neither number stands alone.
Normalized concordance
100.0%
12/12 samples
Raw string concordance
33.3%
4/12 samples
Across all 12 synthetic CYP2C19 fixtures tested, our caller's diplotype matched the cited GeT-RM reference after aligning notation on 12 of 12 samples (100.0%); comparing the engine's raw output strings before that alignment, the match rate was 4 of 12 (33.3%).
Why the two numbers differ
Our engine's CYP2C19 output lists only the detected non-reference star allele, for example *2, which is the same genotype GeT-RM records as the full diplotype *1/*2. The raw comparison treats "*2" and "*1/*2" as a mismatch even though they describe the same result. Normalization parses both sides into a comparable two-allele form and fills in the implied reference (*1) complement, but only when the input demonstrably covered the star-allele-defining positions for that sample; a genuine no-call is never counted as a match.
Limitations, read before the number above
None of these are hidden in a footnote. Every one of them bounds what this benchmark can honestly say.
Synthetic input, not real reference material
Every fixture is a fabricated file: real public background loci with fabricated filler genotypes, plus engineered genotypes at the two real CYP2C19 star-allele-defining positions. No wet-lab reference material, no real Coriell array, and no real customer genome was read for this benchmark.
Array-callable subset only
CYP2D6 copy-number and structural variants are not covered at all. CYP2D6 is an outside-call gene on array input in our product, and this benchmark cannot and does not speak to its accuracy.
One gene
CYP2C19 is the only gene in this comparison. CYP2C9, VKORC1, and TPMT calls exist in the engine's evidence store at this pin but are not yet consumed by our product's normalizer, so they are correctly excluded here rather than padded in.
Small sample size
12 sample-by-gene rows, 5 distinct diplotypes, 3 CPIC phenotype classes. This is a targeted caller-correctness check, not a population-representative validation study.
No WGS or HLA coverage this round
Whole-genome sequencing concordance and the separate HLA hypersensitivity screen are both out of scope for this benchmark.
Phenotype is not scored
The engine returned a diplotype call for every non-*1/*1 case but no phenotype prediction on these minimal synthetic fixtures. This benchmark scores the diplotype call only.
Every sample, row by row
The full comparison behind the numbers above: sample ID, the cited GeT-RM diplotype, the engine's raw output, the canonical (normalized) form, and whether each notation matched.
| Sample | Truth (GeT-RM) | Engine output (raw) | Canonical (normalized) | Match, normalized | Match, raw |
|---|---|---|---|---|---|
| GM17227 | *1/*1 | no call | *1/*1 | match | no match |
| GM17221 | *1/*1 | no call | *1/*1 | match | no match |
| GM17129 | *1/*1 | no call | *1/*1 | match | no match |
| GM17058 | *1/*2 | *2 | *1/*2 | match | no match |
| GM17245 | *1/*2 | *2 | *1/*2 | match | no match |
| GM02016 | *1/*2 | *2 | *1/*2 | match | no match |
| GM12273 | *1/*2 | *2 | *1/*2 | match | no match |
| GM17052 | *1/*3 | *3 | *1/*3 | match | no match |
| GM17289 | *2/*2 | *2/*2 | *2/*2 | match | match |
| GM16689 | *2/*2 | *2/*2 | *2/*2 | match | match |
| GM17263 | *2/*2 | *2/*2 | *2/*2 | match | match |
| GM16688 | *2/*3 | *2/*3 | *2/*3 | match | match |
Citation
Pratt VM, et al. Characterization of 107 genomic DNA reference materials for CYP2D6, CYP2C19, CYP2C9, VKORC1, and UGT1A1: a GeT-RM and Association for Molecular Pathology collaborative project. J Mol Diagn. 2010;12(6):835-846. PMID 20889555, Table 2.
Provenance
- Engine pin
- 47e0d0585073450a810d78a5235428b91b5d746d
- Truth asset version
- getrm_truth.v1
- Run mode
- live engine (not a mock)
- Run date
- 2026-07-16