Accuracy of Diagnostic Codes and Algorithms Used to Identify Rheumatoid Arthritis and Juvenile Idiopathic Arthritis in Administrative Claims and Electronic Health Records: Systematic Review and Meta-Analysis.

Save

Objective

This systematic review aimed to assess the diagnostic accuracy of algorithms used to identify rheumatoid arthritis and juvenile idiopathic arthritis in electronic health records.

Methods

We searched Medline, Embase, and Cochrane Central Register for Controlled Trials databases and included studies that validated case definitions against a reference standard, such as rheumatologist‐confirmed diagnosis or American College of Rheumatology/EULAR classification criteria. Title and abstract screening, full‐text review, data extraction, and quality assessment were all completed in duplicate. Results were synthesized narratively and using a bivariate random‐effects meta‐analysis of sensitivity and specificity.

Results

A total of 35 studies were included. Algorithms varied widely in complexity, ranging from single International Classification of Diseases (ICD) codes to combinations including disease‐modifying antirheumatic drugs (DMARDs), hospitalization records, and specialist diagnosis. Algorithms combining ICD codes with DMARD prescriptions (pooled sensitivity 0.79 [95% confidence interval (CI) 0.61–0.90], specificity 0.96 [95% CI 0.72–1.00], positive predictive value [PPV] 0.78 [95% CI 0.63–0.88]) or requiring an ICD code assigned by a rheumatologist (pooled sensitivity 0.91 [95% CI 0.70–0.98], specificity 0.94 [95% CI 0.49–1.00], PPV 0.70 [95% CI 0.64–0.75]) showed the highest accuracy, with balanced sensitivity, specificity, and PPV.

Less restrictive algorithms demonstrated high sensitivity but lower PPV. Substantial heterogeneity was observed across studies, likely due to differences in algorithm structure, data sources, and validation methods. Despite this variability, we used conceptually coherent categories to allow for meaningful synthesis, prioritizing clinical interpretability.

Conclusions

These findings support the use of more specific algorithms when diagnostic certainty is essential and highlight the need for further validation of high‐performing algorithms across diverse health care systems.

INTRODUCTION

Rheumatoid arthritis (RA) and juvenile idiopathic arthritis (JIA) are autoimmune diseases characterized by joint inflammation and systemic complications leading to disability and reduced quality of life.1 Despite differences in onset and classification criteria, they share key pathogenic mechanisms, clinical manifestations, and therapeutic strategies.2, 3 A challenge in RA and JIA research is obtaining robust epidemiologic data to understand the natural history of disease and the impact of therapy on disease outcomes.4 Electronic health records (EHRs), have greatly facilitated studying these conditions in large populations, providing comprehensive data on patient demographics, symptoms, diagnoses, and treatment history5, 6 in a real‐world setting.

SIGNIFICANCE & INNOVATIONSThis review provides an updated and expanded synthesis of validated algorithms to identify rheumatoid arthritis (RA) and juvenile idiopathic arthritis (JIA) in electronic health records, addressing a growing need as real‐world data become increasingly central in rheumatology research. Compared to earlier reviews, it includes more recent studies, a broader range of geographic and health care settings, both adult and pediatric populations, and structured categorization of algorithm components. It is also the first to perform meta‐analysis stratified by algorithm components and to assess the risk of bias using the QUADAS‐2 tool.

These advances offer practical guidance for researchers for selecting or developing algorithms tailored to their setting and data source.This review offers evidence‐based guidance for selecting RA and JIA identification algorithms based on their accuracy across diverse settings. By categorizing algorithms and summarizing accuracy measures, the findings support researchers and clinicians in choosing or adapting definitions depending on research purposes and available data sources. These findings may also inform future algorithm development and validation studies.The review highlights gaps in validation efforts and emphasizes the need to validate high‐performing algorithms across diverse health care settings and evolving coding systems, ensuring accurate disease identification in current and future research.

However, the reliability and validity of research based on EHR data depend on the accuracy of the diagnostic codes and algorithms used to identify the condition.7, 8 Combining multiple diagnostic codes with clinical data, such as prescriptions and laboratory results, enhances RA identification accuracy. For instance, algorithms incorporating at least three physician diagnostic codes achieve specificity and positive predictive values (PPVs) of over9 90%. The inclusion of disease‐modifying antirheumatic drug (DMARD) prescriptions with diagnostic codes, significantly improves case identification.10 Standardized algorithms that integrate International Classification of Diseases (ICD) codes with other clinical parameters have also been assessed for their utility in diverse health care systems.11

Despite advances in identifying RA and JIA cases, limitations persist due to coding inaccuracy, population heterogeneity, and differences across health care systems and methodologic approaches. A 2013 systematic review, part of the Mini‐Sentinel pilot project sponsored by the US Food and Drug Administration, reported substantial variability in the accuracy of algorithms (PPV 34%–97%) applied to EHRs.12 The data sources of interest were limited to the United States or Canada, reflecting Mini‐Sentinel's focus on US‐based data for surveillance system,13 thus limiting the global applicability of the findings.

The aim of this systematic review is to synthesize and critically assess evidence on the accuracy of diagnostic codes and algorithms used to identify RA and JIA in EHRs and other administrative databases. By expanding and updating the scope, this review provides a more comprehensive evaluation of the tools used to identify the disease across various key data sources in the context of the significant changes in RA and JIA treatment and outcomes over the past decade.

METHODS

The study protocol was registered on PROSPERO (registration ID: CRD42024619130). The systematic review is reported in line with the Preferred Reporting Items for Systematic Reviews and Meta‐Analyses of Diagnostic Test Accuracy (PRISMA‐DTA) statement.14

Eligibility criteria

We included studies assessing the accuracy of codes and algorithms identifying RA and JIA in EHRs or other administrative databases against a reference standard based on clinician‐documented diagnosis, clinician‐conducted chart review, another form of clinician‐confirmed diagnosis, or application of the American College of Rheumatology (ACR)/EULAR criteria15 (eg, to general practitioner records, medical charts, office records) regardless of study design. Studies relying on self‐reported diagnoses as the reference standard were excluded, as previously recommended.16 Eligible studies had to clearly report algorithm components and describe methods for accuracy assessment. We excluded case reports, editorials, commentaries, and narrative reviews.

Although systematic reviews were not included, relevant ones were used for reference checking. No restrictions were placed on language or publication date.

Information sources

We searched Medline (PubMed interface), Embase, and the Cochrane Central Register for Controlled Trials. Search strategies used medical subject headings (MeSH) terms and text‐words aligned with our eligibility criteria (Supplementary Material S1) and were adapted for each database. To ensure literature saturation, we manually reviewed references of eligible studies and included articles from prior systematic reviews.

Search strategy

Our strategy was based on the Mini‐Sentinel project search strategies,12, 17 updated with additional terms. We reviewed the PubMed MeSH terms, as well as abstracts and full texts of primary studies from a prior systematic review, to identify relevant terminology. The search was developed collaboratively and was validated by confirming that it retrieved all articles from the Chung et al12 review.

After the Medline search was finalized, it was adapted to the other databases. Full search strategies and validation process details are in Supplementary Material S2. Searches were conducted on November 11, 2024.

Study records

Data management

We used Covidence software for record management and deduplication. Screening questions for title and abstract review were developed and tested. Reviewers completed training and pilot screening (50 abstracts by 5 reviewers) to calibrate eligibility criteria.

Study selection

Of the reviewers CS‐H, JB, YA, M. Fatir, and FM, two independently screened titles and abstracts against eligibility criteria. Full texts of potentially eligible studies were reviewed in duplicate.

Disagreements were resolved by discussion or a third reviewer. Reasons for full‐text exclusions were documented.

Data collection process and items

Two reviewers independently extracted data using a standardized template based on the Standards for Reporting of Diagnostic Accuracy Studies criteria,18 adapted for administrative databases. Discrepancies were resolved through discussion or a third reviewer. Extracted data included study characteristics, algorithms definitions, validation methods, and accuracy measures.

The template is available as a data supplement (see Availability of data section). Classifications for the target condition are provided in Supplementary Material S3.

Diagnostic accuracy measures

Primary accuracy measures were sensitivity, specificity, PPV, and negative predictive value. When not reported, these were calculated from two‐by‐two tables. For practical interpretation, accuracy was categorized is high (≥80%), moderate (60%–80%), or low (<60%).19 The unit of assessment was the individual.

Further details are provided in Supplementary Material S3.

Risk of bias and applicability

Quality was assessed using the QUADAS‐2 tool, which examines risk of bias and applicability across four domains: patient selection, index test, reference standard, and flow/timing.20 Domains were rated as low, unclear, or high. Overall study bias followed QUADAS‐2 guidance (ie, any high‐risk domain rendered the study high risk overall).14 Two reviewers independently assessed study quality, resolving disagreements through discussion.

Synthesis of results

We categorized algorithms based on the number and type of codes used to identify RA and JIA (eg, ICD, prescriptions, diagnostic tests, procedures). Classification followed the review by Shrestha et al,21 with modifications made as necessary:Less restrictive algorithms: required a single diagnostic code from an outpatient visit or unspecified source (eg, single ICD code)Restrictive algorithms: required multiple codes or inclusion of specific types (eg, procedures, prescriptions, tests, or hospitalization codes)

Algorithms were categorized as either restrictive or less restrictive. For further analysis, we developed subcategories among restrictive algorithms based on expected accuracy and usage frequency in included studies. These were RA diagnosis codes and DMARD prescription codes, two or more ICD codes for RA, one or more ICD codes for RA by rheumatologist, and one or more ICD hospitalization codes for RA (full descriptions are in Supplementary Material S3).

Meta‐analysis

We performed a bivariate random‐effects meta‐analysis using a generalized linear mixed‐effects model to jointly synthesize sensitivity and specificity estimates, accounting for within‐ and between‐study variability and their correlation.22, 23, 24 Pooled PPV estimates were obtained via meta‐analysis of proportions using the meta package in R, applying random‐effects models within subgroups defined by algorithm and reference standard categories (metaprop function).

For each study, sensitivity and specificity with 95% confidence intervals (CIs) were calculated. Forest plots were generated for PPV, sensitivity, and specificity, displaying individual and pooled estimates by algorithm subgroup to capture between‐study heterogeneity. Heterogeneity was assessed using I2 statistics derived from variance components of the random‐effects model.

Analyses were conducted using R (version 4.4.3).

Availability of data

This research is based on published literature. All data used in this research are already included in the article or the supplementary material. The data extraction template is available at the Open Science Framework (https://osf.io/xfpzg).

RESULTS

Study selection

The search yielded 4,109 records; after removing duplicates, 3,452 were screened based on eligibility criteria, and 108 underwent full‐text review. Seventy‐seven articles were excluded (reasons provided in Figure 1). Reference checks of the 31 eligible articles identified 4 additional studies, totaling 35 included in the review (Figure 1).

Study characteristics

Most studies were from the United States and Canada (n = 21), followed by Europe, (mainly Italy, the United Kingdom, Denmark, and Sweden; n = 10), Asia (Japan and South Korea; n = 3), and Australia (n = 1). The majority were retrospective cohort studies (n = 25), followed by cross‐sectional (n = 6) and case‐control studies (n = 4). Two‐thirds (n = 23, 66%) were published since the 2013 review (Supplementary Table S1).

The included studies (n = 35) used diverse electronic databases, including national health insurance claims (eg, Korean National Health Insurance database,25 Medicare,10, 26, 27, 28 and Ontario Health Insurance Plan9, 29), hospital discharge records (eg, Stockholm County Medical Information System,30 Danish National Patient Registry,31, 32 Partners Healthcare database used by Brigham and Women's Hospital and Massachusetts General Hospital,33, 34, 35 and Canadian Institute for Health Information Discharge Abstract Database9, 29), and institutional EHRs (eg, Vanderbilt University Medical Center's Synthetic Derivate33, 36, 37; University of California, Los Angeles [UCLA] Health System38; Mayo Clinic Biobank39; and EHRs from specific rheumatologic institutions or disease registries11, 40, 41, 42). Some also incorporated pharmacy data31, 43, 44 and physician billing records.42, 45, 46

Most studies focused on patients with RA, although five examined patients with JIA.36, 40, 46, 47, 48 Samples sizes ranged from small cohorts, for example, 31 and 94 participants,47, 49 to large datasets exceeding 55,000 individuals.25 Women made up 50% to 80% of participants. Mean age varied, typically between 50 and 70 years for RA and <19 years for JIA. Full study characteristics are detailed in Supplementary Table S1.

RA case identification relied mainly on ICD codes (Table 1), with ICD‐9 (62.8%) and ICD‐10 (48.6%) codes being the most used (Supplementary Table S1). Some older datasets also used ICD‐8 codes.30, 31 Additional criteria were applied in several studies, including medications (54.3%) and laboratory markers (22.8%) such as rheumatoid factor (RF) and anti–cyclic citrullinated peptide (anti‐CCP) antibodies35, 50 (Table 1). Some studies implemented complex algorithms, including the eMERGE algorithm, which combined ICD codes with RF values and other autoimmune disease exclusions.38, 39 The algorithm developed by Liao et al,35 incorporating ICD codes, prescriptions, and laboratory values, was validated in other settings by Carroll et al33 and Huang et al34 (Supplementary Table S1).

Most studies (71.4%) used a rheumatologist's clinical diagnosis as the reference standard (Table 1), whereas others applied established classification criteria, including the 1987 ACR criteria51 and the 2010 ACR/EULAR criteria15 and for JIA, the 2001 International League of Associations for Rheumatology criteria52 (Supplementary Table S1). Four studies relied on medical records review by trained nurses or researchers, without clear specification of the reference standard27,28,37,50 (Table 1).

Risk of bias and applicability

Risk of bias and applicability concerns varied across studies, with patient selection (unclear risk 54%, high risk 26%) and flow/timing being the most affected domains (unclear risk 57%, high risk 26%) (Table 2, Supplementary Figure S1). Patient selection showed the highest risk, mainly due to nonrandom inclusion, use of ICD codes without additional verification, and concerns regarding representativeness. The index test domain showed the lowest risk (63%), generally due to clearly defined and consistently applied algorithms, though some studies lacked clarity (eg, specific medications included in the algorithm) or did not report blinding to the reference standard.

The reference standard domain showed low risk (57%), using established standards such as rheumatologist diagnosis or ACR/EULAR criteria,15 though some had unclear validation methods. Flow and timing had mixed results, with high risk in studies that did not include all participants in the analysis or did not assess all diagnostic accuracy measures due to study design.

Overall, six studies were rated high risk of bias26, 27, 28, 30, 41, 49 due to limitations in multiple domains, potentially leading to misclassification and reduced generalizability. In contrast, studies by Carroll et al,33 Cho et al,25 Liao et al,35 Singh et al,53 and Widdifield et al9 had low risk, with strong design across all domains (Table 2).

Applicability concerns were generally low (patient selection 51%, index test 66%, reference standard 66%), suggesting good relevance to real‐world settings. However, studies using specialized databases or narrow inclusion criteria (eg, pediatric populations, single‐center databases, incident RA cases) had moderate concerns because they may not be generalizable to broader RA populations (Table 2, Supplementary Figure S1).

Results of individual studies

RA codes and algorithms

Individual study results are detailed in Supplementary Table S2. Across studies, over 180 definitions/algorithms were evaluated. Percentage of studies that used less restrictive or restrictive algorithms and their categories are presented in Figure 2.

Less restrictive algorithms typically had high sensitivity but low specificity, increasing false‐positives. For example, Almutairi et al54 found 90% sensitivity but only 28.5% specificity using one or more RA primary codes in hospital discharge records, highlighting the risk of overidentification. Other studies29, 45, 53 showed similar trends, with sensitivities >90% and specificities <65%.

Adding DMARD prescriptions to ICD codes improved specificity and PPV while maintaining good sensitivity. The study by Cho et al25 achieved 85% sensitivity and 70% specificity using ICD‐10 codes with biologic, DMARD, or nonsteroidal anti‐inflammatory drug prescriptions, maintaining >85% PPV values in all developed algorithms. Similarly, Singh et al53 reported 85% sensitivity, 83% specificity, and 81% PPV when using a similar approach.

The study by Linauskas et al31 showed PPV increased from 61.9% to 87.7% when requiring one or more DMARD prescriptions.

Requiring two or more ICD RA codes, separated over time, improved specificity while maintaining good sensitivity, though decreasing PPV. Hanly et al45 reported 83% sensitivity, 81% specificity, and 52% PPV using two RA physician codes two or more months apart. Similar trends were seen in the studies by Kubota et al55 (84% sensitivity, 99% specificity, and 59% PPV) and Widdifield et al9 (84% sensitivity, 99% specificity, and 46% PPV).

Requiring one or more hospitalization records improved specificity and PPV but greatly reduced sensitivity. Convertino et al56 found 95% specificity and 81% PPV but only 37% sensitivity when requiring one RA hospitalization or emergency admission code. Similarly, the study by Hanly et al45 achieved 98.5% specificity, 77% PPV, and 20.7% sensitivity when using one or more hospitalization codes.

Similar patterns were noted by Widdifield et al.9, 29

Algorithms requiring at least one rheumatologist diagnosis code showed overall moderate to high sensitivity and specificity, with moderate PPV. The studies by Widdifield et al9, 29 achieved 81% and 99% sensitivity, 99% and 77% specificity, and 51% and 68% PPV, respectively. Similarly, Hanly et al45 reported 88% sensitivity, 75.5% specificity, and 47% PPV when using one or more RA codes by a rheumatologist.

In contrast, Carrara et al11 reported 99.8% specificity but only 13% sensitivity when combining one RA rheumatologist certification with other ICD‐9 codes.

The eMERGE algorithm, combining ICD codes, RF values, and other autoimmune disease exclusions, demonstrated high specificity (95%–99%) and PPV (94%–97%) in both studies evaluating it (studies by Zheng et al38 and Kronzer et al,39 respectively). However, sensitivity varied likely due to differences in sample size and study population. Zheng et al38 (n = 4,766, UCLA Health System) reported a higher sensitivity (72%), whereas the study by Kronzer et al39 (n = 497, Mayo Clinic Biobank included in the Rochester Epidemiology Project) achieved a sensitivity of 53% in a smaller and more geographically limited sample.

However, Kronzer et al39 found that sensitivity increased with RA duration and EHR history, reaching 71% for patients with ≥10 years’ disease duration. Overall, the eMERGE algorithm performed well, with sensitivity possibly influenced by dataset scope.

The RA algorithm initially developed by Liao et al,35 validated by Carroll et al33 and Huang et al,34 consistently showed moderate sensitivity (51%–79%) and high PPV (>87%) across different settings, showing a good balance between measures for identifying RA cases. Huang et al34 enhanced the model by incorporating ICD‐10 codes and newer biologic treatments, achieving 77% sensitivity, 95% specificity, and 91% PPV. Overall, the algorithm maintained high accuracy, with sensitivity improving slightly as case definitions and coding systems evolved.

Both the eMERGE and Liao algorithms used laboratory data (RF, anti‐CCP), which improved specificity and PPV, and had limited effect on sensitivity. Paltta et al43 reported 98% PPV when incorporating anti–citrullinated protein antibody positivity in their algorithm, compared with ICD codes alone (82%) or ICD codes and DMARDs (89%). Singh et al53 also showed improved sensitivity (88%), specificity (91%), and PPV (93%) when adding positive RF to their algorithm, compared to ICD codes alone (sensitivity 100%, specificity 55%, PPV 66%) or ICD codes and DMARDs (sensitivity 85%, specificity 83%, PPV 81%), suggesting that laboratory data helps reduce false‐positives while maintaining a balanced improvement in the accuracy measures.

JIA codes and algorithms

JIA studies evaluated similar EHR‐based algorithms (Supplementary Table S3). Less restrictive algorithms, such as one or more JIA diagnosis codes, showed high sensitivity but moderate to low PPV. Harrold et al40 and Peterson et al36 both reported 100% sensitivity, with PPVs of 45% and 48%, respectively.

Only one study assessed the accuracy of DMARD prescription alone as a definition. Thomas et al48 found 40% sensitivity but 100% specificity and PPV when requiring one or more DMARD prescriptions without prior alternative indication.

Requiring two or more ICD codes for JIA improved PPV while maintaining good sensitivity. Reported sensitivity ranged from 72% to 93% and PPV ranged from 60% to 100% across studies.36, 40, 46, 47, 48 Peterson et al36 noted that increasing the number of JIA codes (four or more) raised PPV (83%) without substantial decrease in sensitivity (87%), compared to using one or more codes (PPV 48%). Shiff et al46 found that specificity improved (85%–92%) with more ICD codes over longer intervals.

Two studies assessed algorithms requiring one or more JIA codes from a rheumatologist. The study by Harrold et al40 showed that requiring a rheumatologist visit improved PPV from 69% to 90% while maintaining good sensitivity (81%). The study by Stringer et al47 showed similar sensitivity (86%) but decreased PPV (58%), which increased to 65% when using two JIA codes eight or more weeks apart within two years, maintaining good sensitivity (81%).

Meta‐analysis

Meta‐analyses showed substantial variability in the accuracy of RA identification algorithms in EHRs, depending on algorithm components and reference standards. Using rheumatologist diagnosis as a reference, algorithms requiring two or more ICD codes had the highest pooled sensitivity (0.89 [95% CI 0.75–0.95]; n = 4 studies) and specificity (0.96 [95% CI 0.74–1.00]; n = 4), though with high heterogeneity (I2 = 91.3% and I2 = 99.8%, respectively). Algorithms combining ICD codes with DMARD prescriptions also showed good pooled sensitivity (0.79 [95% CI 0.61–0.90], I2 = 90.6%; n = 5) and high specificity (0.96 [95% CI 0.72–1.00], I2 = 98.9%; n = 5) (Table 3, Supplementary Figure S2).

Algorithms using one or more ICD codes by a rheumatologist also showed high performance (sensitivity 0.91 [95% CI 0.70–0.98], specificity 0.94 [95% CI 0.49–1.00]; n = 3) but were based on only three studies, warranting cautious interpretation. A meta‐analysis of sensitivity and specificity using ACR/EULAR classification criteria15 as the reference standard was not feasible due to insufficient data.

PPV results varied by reference standard and showed substantial heterogeneity. Using rheumatologist diagnosis as the reference, the highest pooled PPVs were seen in algorithms combining ICD and DMARD codes (0.78 [95% CI 0.63–0.88]; n = 10), those including one or more ICD codes by a rheumatologist (0.64 [95% CI 0.51–075]; n = 4), and those requiring one or more hospitalization codes (0.79 [95% CI 0.77–0.81]; n = 7). In contrast, when using ACR/EULAR classification criteria,15 PPVs were slightly lower when requiring two or more ICD codes (0.62 [95% CI 0.14–0.94]; n = 2) or ICD and DMARD codes (0.73 [95% CI 0.45–0.90]; n = 6).

Algorithms requiring one or more ICD codes by a rheumatologist demonstrated the highest pooled PPV (0.85 [95% CI 0.35–0.98]; n = 3), though with high heterogeneity (I2 = 96.3%) (Table 3, Supplementary Figure S3).

DISCUSSION

This systematic review analyzed the accuracy of algorithms for identifying RA and JIA in EHRs. Overall, algorithms combining ICD codes with DMARD prescriptions or those that include one or more ICD codes assigned by a rheumatologist demonstrated the highest accuracy, balancing sensitivity, specificity, and PPV. These algorithms are more reliable for identifying confirmed RA cases, though rheumatologist‐assigned codes may miss cases diagnosed in primary care, potentially leading to underestimation of disease prevalence.

Less restrictive algorithms are useful for initial case identification (highly sensitive) but may require additional criteria to enhance specificity and PPV. Requiring two or more ICD codes is a well‐balanced approach, reducing false‐positives (higher specificity) without significantly lowering sensitivity, though achieving moderate PPV. Hospitalization‐based codes are highly specific for identifying severe RA cases but may exclude many nonhospitalized, mild/moderate cases, making it less suitable for general RA case identification.

For JIA, a single ICD code is highly sensitive but may overestimate prevalence due to false‐positives. Requiring two or more spaced ICD codes improves accuracy and seems to be the best approach, providing a good balance of sensitivity, specificity, and PPV in EHRs or administrative data. However, validation evidence for JIA remains limited, highlighting the need for further research.

Our findings align with and extend previous systematic reviews on RA and connective tissue disease algorithms in EHRs. Compared to the 35 studies we included, the 2013 review by Chung et al12 included 9 studies and found that algorithms combining diagnostic and prescription data—particularly those including specialist codes, medication data, or laboratory data—achieved higher PPVs (>80%), whereas ICD‐only algorithms had more variable performance across settings, generally with low PPVs. Similar trends were seen in studies for other conditions, but there were generally lower PPVs (<65%) in general populations,21, 57 probably reflecting the challenges of identifying these conditions due to nonspecific coding and overlap with other autoimmune conditions.

Similarly, a systematic review by Crossfield et al58 on gout also highlighted variability in methods and performance, reflecting a lack of standardization in defining rheumatologic conditions using real‐world data. Compared to these conditions, RA appears to be more accurately identified in EHRs, especially when algorithms include medication data or are supported by specialist confirmation.

The accuracy of hospitalization‐based algorithms must be interpreted within the context of evolving health care delivery. RA hospitalizations have declined significantly over the past two decades due to earlier diagnosis, improved outpatient care, and widespread use of biologic therapies.59, 60, 61 As a result, performance of algorithms using hospitalization codes in historic data may be less representative of current patient populations. Similarly, treatment patterns have evolved, with earlier DMARD initiation, combined therapies (eg, biologics and DMARDs),62, 63, 64 and the development of targeted‐synthetic DMARDs (eg, JAK inhibitors), which may affect the predictive value of medication‐based algorithms applied to contemporary prescribing data.

The transition from ICD‐9 to ICD‐10 introduced more specific diagnosis codes, potentially influencing algorithm performance. In this review, 62.8% of studies examined ICD‐9 codes and 48.6% examined ICD‐10 codes, reflecting variability in study periods, country‐level implementation, differences in capacity to adopt new coding systems, and the use of historical data sources. Algorithms developed with ICD‐9 codes may not translate directly to the ICD‐10 without revalidation,65, 66 particularly for specific or rare disease subtypes.

These changes underscore the need for regular update and validation of algorithms to align with current coding systems and clinical practice.

From a practical perspective, the selection of an algorithm for identifying RA or JIA should be guided by the purpose of the study and the nature of the data source. For studies aiming to evaluate treatment outcomes, assess adverse events, or conduct comparative effectiveness studies, algorithms with high specificity may be preferable to ensure the correct identification of true cases. In contrast, for surveillance or studies assessing prevalence or burden of disease, algorithms prioritizing higher sensitivity may be preferable, even at lower PPV.

To further support practical application, we provide a summary table (Table 4) outlining algorithm definitions, accuracy, and considerations for their use depending on study purposes and data source, which may assist researchers and clinicians in selecting or adapting algorithms in future EHR‐based studies.

Future research should focus on validating existing algorithms, such as the eMERGE38, 39 and Liao et al35 algorithms, across diverse health care systems and populations. Although both show consistent diagnostic accuracy in diverse settings, the generalizability of the Liao algorithm beyond academic health centers remains uncertain. Validation using community‐based data is needed to ensure broader applicability.

Clear and transparent reporting of algorithm components is essential for replication, external validation, and broader implementation in different contexts. Additionally, the development and validation of algorithms using machine learning or natural language processing techniques hold promise for further improving case ascertainment, particularly in settings where structured coding is limited.

The studies reviewed exhibited substantial methodologic variability, making direct comparisons difficult. Differences in sample size, sampling methods, settings, population, and data sources likely contributed to the high statistical heterogeneity in several analyses. Algorithm definitions varied widely, even within similar categories, highlighting a lack of standardization in algorithm development and validation methods.

Despite this, meta‐analyses were feasible by grouping algorithms into broad but conceptually coherent categories, prioritizing simplicity and clinical interpretability. Importantly, algorithm performance is highly context dependent and may not be generalizable across different health care systems, data sources, or patient populations. As such, the pooled estimates should not be used to directly select algorithms for a given setting but rather to identify overall trends and guide clinicians and researchers in choosing or adapting algorithms appropriate to their specific context and data source.

The results may also support future validation efforts in more targeted contexts.

Although many studies incorporated medication use, most grouped DMARDs as a single category, precluding an evaluation of the contribution of specific drugs or therapeutic classes to the accuracy. Several studies did not report full accuracy measures due to sampling designs, which involved evaluating only algorithm‐positive cases and thus only the PPV. The PPV depends on disease prevalence within the population being studied.

Several studies selected their populations from hospital settings or rheumatologic clinics, where the prevalence of RA is likely to be higher than in the general population, potentially leading to an overestimation of the PPVs. Another source of variability may be the data used in each study and the purpose for which the data were originally generated (ie, claims/reimbursement vs clinical care). Therefore, findings should be interpreted cautiously and, ideally, validated in general populations to better understand their performance across varying levels of disease prevalence.

Additionally, sensitivity and specificity may be influenced by the characteristics of the population included in the reference standard. For example, algorithms validated using rheumatologist‐confirmed diagnoses may differ in performance from those validated against classification criteria, which may be more stringent or require specific data elements. These methodologic differences highlight the need to consider context when comparing accuracy across studies.

Finally, the search focused on studies explicitly reporting diagnostic validation, potentially excluding relevant studies that addressed this in broader investigations (ie, assessed algorithm accuracy as an intermediate objective as part of an EHR‐based study of RA outcomes). A protocol deviation was the omission of a gray literature search due to the nature of the review and to the expectation that relevant studies would be published primarily in peer‐reviewed journals.

This systematic review incorporates comprehensive and contemporary evidence from diverse geographic areas. It comprehensively synthesizes and quantitatively analyzes RA and JIA algorithm accuracy in EHRs across diverse health care settings. It adheres to PRISMA guidelines, systematically categorizes algorithms by restrictiveness to facilitate clinically meaningful comparisons, and uses the QUADAS‐2 tool for transparent quality assessment.

This review highlights the variability in accuracy of algorithms used to identify RA and JIA in EHRs. Algorithms combining diagnostic codes and prescription data, particularly DMARDs, or a rheumatologist diagnosis demonstrated the highest overall performance, with a good balance of sensitivity, specificity, and PPV. Less restrictive algorithms may aid initial case identification but require refinement for specific research purposes.

The choice of algorithm should be tailored to the study objective and data source. Future efforts should prioritize external validation of high‐performing algorithms across diverse settings to improve standardization and reliability in real‐world data research.

AUTHOR CONTRIBUTIONS

All authors contributed to at least one of the following manuscript preparation roles: conceptualization AND/OR methodology, software, investigation, formal analysis, data curation, visualization, and validation AND drafting or reviewing/editing the final draft. As corresponding author, Prof. Moriarty confirms that all authors have provided the final approval of the version to be published and takes responsibility for the affirmations regarding article submission (eg, not under consideration by another journal), the integrity of the data presented, and the statements regarding compliance with institutional review board/Declaration of Helsinki requirements.

Supporting information

Related articles

PubMed

Macrophage Activation Syndrome Complicating Systemic Juvenile Idiopathic Arthritis in a Child With Coronary Artery Dilation and SARS‐CoV‐2 Positivity: Diagnostic Overlap With MIS‐C. A Case Report.

ABSTRACT Macrophage activation syndrome (MAS) can complicate systemic juvenile idiopathic arthritis (sJIA) evolving over weeks. With SARS‐CoV‐2 positivity and coronary dilation, MIS‐C cannot be excluded, and here the timeline and serology do not establish SARS‐CoV‐2 as the trigger. Fibrinogen is an acute‐phase reactant and should not be required to recognize MAS.

PubMed

Juvenile idiopathic arthritis is associated with reduced levels of biochemical bone status indices reflecting altered bone formation.

Introduction Individuals with juvenile idiopathic arthritis (JIA) are at increased risk of impaired bone health and fractures. This prospective case-control study assessed bone mineral density (BMD) and bone and cartilage biomarkers in children with JIA and healthy controls. Methods Fifty-one individuals with well-characterized JIA (age 1.6–17.9 years) and 61 age-matched controls from the same region were included. Joint erosions were assessed radiographically, and BMD was measured by digital...