The reporting quality of studies investigating the diagnostic accuracy of anti-CCP antibody in rheumatoid arthritis and its impact on diagnostic estimates

Rheumatoid arthritis (RA) is a chronic, systemic inflammatory disorder that affects many tissues and organs, mainly synovial joints [1]. The disease leads progressively to the destruction of articular cartilage and ankylosis of the joints [2]. Although the cause of RA is unknown, autoimmunity plays a pivotal role in both its chronicity and progression [3]. RA affects females more frequently than males and it is diagnosed mainly in age 40–60 years [4].

The diagnosis of RA is based on clinical criteria and laboratory tests. Regarding the later tests, the presence of the rheumatoid factor (RF), an autoantibody, consists one of the American College of Rheumatology (ACR) criteria for presence and severity of RA [5]. However, RF has a limited specificity since it can be detected in other autoimmune or infectious diseases, and in the healthy elderly. Anti-cyclic citrullinated protein antibodies (anti-CCP) are other autoantibodies that may be detected in RA patients. Recently anti-CCP testing has become substantial part of ACR-EULAR classification criteria for RA [6]. There is evidence that CCP-assays provide comparable performance with that of RF [7]. However, analysis of the association between anti-CCP antibody titre and RA activity produced contradictory results [8, 9]. Anti-CCP2 assay is the most popular because of its high diagnostic specificity and its predictive and prognostic value in RA [10–12].

Currently, diagnostic studies on anti-CCP assays are publishing with a high rate [13]. However, overestimated and biased results from poorly designed and reported studies may direct scientists into making false decisions [14–16]. The reporting information on design and conduct of diagnostic studies is crucial, though, its absence has already been noticed [17, 18]. Nevertheless, appropriate reporting may allow researchers to detect potential bias in studies’ internal validity, to assess generalizability and applicability of their results [19]. A survey of published studies of diagnostic accuracy showed that the methodological quality was not optimal. In addition, information on issues like study design, conduct and data analysis was often not reported [20, 21].

Inadequate reporting of the published diagnostic accuracy studies may restrict the generalizability, applicability and credibility of studies’ results. A number of guidelines and statements have been developed to improve the quality of a variety of study designs [22], including the diagnostic accuracy studies [19]. In particular, in order to improve the reporting of diagnostic accuracy studies, the Standards for Reporting of Diagnostic Accuracy (STARD) statement has been proposed (http://www.stard-statement.org/ )[19]. The STARD statement is a checklist of 25 criteria that diagnostic accuracy studies should conform to in order to make their conclusions easier to assess, interpret and generalize, and lead as a result to better decisions in diagnosis. However, STARD does not assess the actual quality of the research study but the reporting quality, two issues which are not necessarily correlated. In addition to STARD, another tooled has been proposed, called QUADAS, for assessing the methodological quality of diagnostic accuracy studies [23]. Recently, QUADAS was used to evaluate the quality of anti-CCP RA studies in a meta-analysis [13].

The aim of the present study was twofold: first, to evaluate the reporting quality of studies used anti-CCP2 for the diagnosis of RA, according to the STARD statement, and second, to investigate whether quality of reporting is associated with the effect size of diagnostic metrics using meta-analytic techniques (data synthesis). The analysis was focused on the reporting of methods and results sections of the STARD statement. The effect of quality on diagnostic accuracy was focused on studies scored as “high quality” and “low quality”, and for specific items of STARD.

Table 1 shows the overall proportion of reporting of the 22 items in the methods and results sections of the STARD statement and the corresponding proportions for high and low quality articles.

Overall, 10 items (six and four items in methods and results sections, respectively) were reported by 85% or more of the studies (Table 1). In methods, the items include the reporting of 1) study population (inclusion/exclusion criteria, setting, location), 2) participants recruitment (eg. based on symptoms, previous testing), 3) participant sampling, 4) data collection (prospective or retrospective study), 5) methods for calculating or comparing measures of diagnostic accuracy and statistical methods used to quantify uncertainty and 6) methods for calculating reproducibility, if done. In results, the items include the reporting of 1) clinical and demographic characteristics of the study population (age, sex, presenting symptoms, comorbidity, current treatment), 2) the cross tabulation or the distribution of the test results by the results of the reference standard, 3) estimates of variability of diagnostic accuracy between subgroups of participants, centers, if done and 4) estimates of test reproducibility, if done.

Furthermore, 13 items (including the ten items already mentioned above) were reported by 70% or more of the studies. The 3 additional items were the reporting of 1) reference standard and its rationale of, 2) definition of and rationale for the units, cut-offs and/or categories of tests results and 3) estimates of diagnostic accuracy and measures of statistical uncertainty.

In contrast, some items were reported only by a small fraction of articles. For example, 20% of articles provided the number, training and expertise of persons executing the tests, 18% reported the blinding status, 13% provided information on recruitment, 12% reported adverse events and finally, 8% provided details about handling of missing responses and outliers.

Impact of study quality on diagnostic estimates

Table 2 shows the meta-analysis’ overall results (pooled sensitivities and specificities), the results according to STARD score (high/low quality) and the results for specific STARD items (comparison of outcome “yes” vs. “no”).

Overall, the pooled sensitivity and specificity were 70.8% (95% ci, 68.3% to 73.2%) and 95.8% (95% ci, 94.9% to 96.6%), respectively. Studies with high quality score produced similar results with studies with low quality score [sensitivity: 70.9% (95% ci, 67.5% to 74.1%) vs. 70.7% (95% ci, 67.0% to 74.1%) and specificity: 95.7% (94.4% to 96.7%) vs. 96.0% (94.8% to 97.0%)]. In HSROC analysis, sensitivity and specificity in high vs. low quality studies were found approximately equal; though, a slightly reduced variance of logit-transformed sensitivity and specificity was found in the low quality studies (0.39 and 0.93, respectively) (Figures 2a and 2b).

In comparing specific items (“yes” vs. “not”), the estimates of pooled sensitivities were statistically significant for items 10 and 13 [p = 0.03 and p = 0.06 (marginal), respectively]. In addition, the estimates of pooled specificities were statistically significant for items 13 and 16 (p = 0.01 and p = 0.01, respectively).

The present study investigated the quality of reporting of studies using the anti-CCP2 assay in RA patients according to the STARD statement. The differences between high and low quality studies were explored. The effect of reporting quality on pooled estimates of diagnostic metrics was also examined. Our analysis focused on the reporting of methodological items (items in method and results’ sections). In total, the 103 articles (corresponding to 132 studies) covered a publication period of 23 years. Almost the articles used in our analysis were published after the introduction of STARD statement (only 4 of them were published during 2003, year of STARD appearance).

Although the overall reporting quality was relatively good (13 items were reported by 70% or more of the studies) there are some essential methodological aspects of the studies (such as number/training/expertise of persons executing the tests, readers’ blinding to results, information on recruitment, adverse events from performing the tests, handling of missing responses and outliers) that are seldom reported making it difficult for the reader to assess explicitly the validity of a study. Comparing the quality of reporting in high versus low quality articles, significant differences were seen in a relatively large number of methodological items (11 items referred to: study population, data collection, reference standard, definition of units/cut-offs, number/training/expertise of persons executing and reading the tests, methods for calculating diagnostic measures, dates of recruitment, clinical/demographic characteristics, information on recruitment, time interval between tests, estimates of diagnostic accuracy).

Overall, the STARD quality score (high/low) has no effect on pooled sensitivity and pooled specificity. However, the meta-analysis showed an effect for specific STARD items. Studies not reporting sufficiently the methods used in calculating the measures of diagnostic accuracy (item 1), may have overestimated the sensitivity. In addition, the reporting of demographic and clinical characteristics/features of the study population (items 13 and 16) has affected the effect size of specificity, i.e. they have overestimated it, indicating also a spectrum bias [19].

However, the findings of the present synthesis (sensitivity of anti-CCP2, 71% and specificity, 96%) are compatible with those of earlier reviews (Nishimura et al. [27]: sensitivity, 67% and specificity, 95%, Whiting et al. [13]: sensitivity, 67%, specificity, 96%). An overestimation of our overall sensitivity might be resulted because of the lack of stratification by study design or disease duration in the analysis.

In a recent review, Whiting et al. [13] compared the accuracy of ACPA with that of RF in diagnosing RA in patients with early symptoms of the disease. They also assessed their studies for methodological quality by using a modification of the QUADAS criteria (items related to reporting quality, were removed). However, the impact of quality effect in diagnostic accuracy was not evaluated further. Nevertheless, the primary aim of the present study was to evaluate the effect of quality of reporting (according to STARD) in diagnostic accuracy rather than evaluating the effect of methodological quality (according to QUADAS); though, both tools can be useful for assessing the quality of diagnostic studies in a different perspective [28].

Applications of the STARD statement guidelines for assessing the quality of reporting in diagnostic accuracy studies, have been conducted in various medical fields such as in the field of diagnostic endoscopy [29], of juvenile idiopathic arthritis in peripheral joints [30], of diabetic retinopathy screening [31], of glucose monitor studies [32], of optical coherence tomography in glaucoma [33], of ultrasonography for the diagnosis of developmental dysplasia of the hip [34] and in the field of screening ultrasonography for trauma [35].

A limitation of the present study is that the literature search was restricted to PubMed. In addition, some studies may have been missed since we included only studies that provided data to estimate both sensitivity and specificity. However, the number of articles used is relatively large and an overview of reporting quality of studies may be obtained and the reached conclusions are unlikely to be affected by omitted studies. We would like to stress that lack of reporting of a STARD item does not necessarily implies that this item was not performed. Thus, a badly performed but well reported study will necessarily receive full credit. Finally, the published studies have had different design settings, and involved different stages of rheumatoid arthritis (study design, disease duration) which may question the synthesis of information, and therefore, the generalizability of results.

In conclusion, our attempt to assess the reporting quality of diagnostic accuracy studies in RA highlights the need for further improvement. Implementation of the quality reporting statements (e.g. CONSORT) have already improved the quality of reporting in other fields of medical research [36]. Thus, guidelines on the reporting of diagnostic accuracy studies are expected to improve the quality of reports of diagnostic studies as well. Finally, the study quality has no effect on the pooled estimates of diagnostic accuracy.