Malignancy validation in a United States registry of rheumatoid arthritis patients

Background

Physician reporting is commonly used to ascertain adverse events or outcomes measured in epidemiologic studies. However, little is known on the accuracy of physician reported malignancies compared to pertinent medical record review in large cohort studies.

Methods

The Consortium of Rheumatology Researchers of North America (CORRONA) registry gathers physician-completed questionnaires for rheumatoid arthritis (RA) patients, including request for information on incident malignancies, approximately every three months. For incident malignancies reported from October 1st, 2001, through December 31st, 2007, we retrospectively requested completion of a Targeted Adverse Event (TAE) form for additional information as well as primary source documents to adjudicate the malignancy reports. CORRONA has employed a prospective request for source documentation for these events since 2008. We classified each malignancy as definite, probable, possible, or not a malignancy.

Results

From 20,837 RA patients enrolled in CORRONA, 461 incident malignancies were initially reported on physician questionnaires. After review of returned source documents with adjudication, 234 were deemed definite, 69 probable, 101 possible, and 57 not an incident malignancy. The positive predictive value (PPV) of initial physician report of a malignancy versus “definite or probable” malignancy based on adjudication was 0.66 (95% CI 0.61 - 0.70). The PPV was 0.68 (95% CI 0.63 – 0.72) when the subsequent TAE form also confirmed the presence of malignancy. When possible malignancies were included, the PPV of physician-reported malignancies without a subsequent TAE form increased to 0.86 (0.83 – 0.89), and with a subsequent TAE form, 0.89 (0.85-0.91).

Conclusion

Twelve percent of initial physician reports of incident malignancy could not be confirmed with review of source documents. The most common reason for lack of confirmation was inability to obtain documents or insufficient data in source materials. These results suggest that timely collection of relevant medical records and an adjudication process are required to improve the accuracy of cancer reporting in epidemiologic studies.

Rheumatoid Arthritis (RA) is an auto-immune, multi-system inflammatory disease with significant morbidity and mortality. In addition to its articular manifestations, RA has a variety of extra-articular complications, including malignancy [1, 2]. The impact of immunosuppressive treatments used to manage patients with RA may also influence the development of malignancies, as each of these treatments cause perturbations of the immune system that may lead to malignancy [3–6]. Because of the influences of disease activity and therapy on the immune system of patients with RA, as well as the relative infrequent occurrence of many types of malignancy, it can be difficult to assess the impact of a specific medication on the risk of malignancy.

To date, several epidemiologic and pharmacoepidemiology studies have been performed to assess the rate of malignancy in RA patients treated with Tumor Necrosis Factor a inhibitors (TNFi) [7–18]. Sources for these datasets include national registries, prospective cohorts, and administrative databases. Cancer outcomes in these studies have been identified based on national cancer registers, patient self-report, physician report, administrative claims, and medical records. In the United States, common methods for ascertainment of malignancy among RA patients include patient self-report or physician report. Another approach is mining of centralized administrative databases, often using claims data or ICD-9 codes to document presence of malignancy, as in the US, national malignancy registries do not exist. The Surveillance Epidemiology and End Results (SEER) registry does capture data on incident malignancies in the US, but it only collects data for a handful of states as well as some additional, separate urban areas. There are also state specific cancer registries, however, between the overall number of RA patients and the overall low rate of malignancy among RA patients, they often do not have the power necessary for pharmacoepidemiologic studies. In addition, individual patient information is not available in many of these registries to permit linkage to observational registries of other diseases. Outside the United States, in addition to the above methods, many countries have national cancer registries where reporting of all incident malignancies to a central database with confirmation is mandatory.

To classify incident cancers in a large cohort of patients with RA where a cancer registry was not available, primary records were requested to validate the malignancy. We developed an adjudication process and tested the accuracy of physician-reported incident cancers, using pertinent medical records as the gold standard.

Study population

The Consortium of Rheumatology Researchers of North America (CORRONA) is an independent registry of RA patients that has collected clinical, laboratory, imaging, medication, and toxicity data since 2001[19]. To date, it has collected data from over 100 rheumatology practices and over 300 participating rheumatologists throughout the United States, both academic and private, with over 20,000 RA patients enrolled. Data are collected from both patients and their treating rheumatologists using questionnaires, which gather information on disease duration, prognostic information, physician and patient-determined standardized disease severity and activity measures, medical comorbidities, use of medications including DMARDs, laboratory values, and adverse events [20]. Follow-up assessments are requested at four month intervals and completed during routine clinical encounters. Approvals for participation in the CORRONA registry are obtained from the respective Institutional Review Boards of participating academic sites and a central Institutional Review Board for private practice sites.

At each visit, physicians complete questionnaires that include information about new comorbidities, including cancer. For the period of this study (10/01 to 12/07), follow-up Targeted Adverse (TAE) Forms and request for source documents were performed retrospectively. Since 2008, TAE forms and requests for source documents are initiated at the time of the initial physician report. Only the reporting rheumatologists’ office can request medical records due to privacy requirements established by the CORRONA registry. The medical records requested include pathology reports, hospital discharge summaries, notes from an oncologist, and/or notes from a primary care physician.

Source documents

After the adverse event forms were returned, all sites were asked to provide source documents to corroborate the diagnosis of an incident malignancy. If source documents were not received within a specified period of follow-up, a second request was made. If source documents were still not submitted, a second questionnaire asking for validation specifically of the organ of the malignancy, pathology report, and date of onset was requested.

Medical record review

All malignancies reported between 10/1/2001 and 12/31/2007 were evaluated for this analysis. Once all data, including the adverse event forms and primary source documents, were received, they were reviewed using a structured abstraction form. In the analyses reported here, all data were reviewed and abstracted separately by two investigators (MF and VF). The abstracted data were then compared, and inconsistencies were resolved by returning to the source documents and assessing which record was accurate. As some data were subject to interpretation, any discrepancies between the two records were noted, and a third party (JG) reviewed the data. Three investigators (MF, VF, JG) then discussed the information. If consensus between MF and VF could not be reached, the third party (JG) adjudicated how the data would be recorded.

Source documents were ranked by two of the authors (MF and VF) for confidence in confirmation of the presence and type of malignancy. These rankings were then submitted to other members of the research team (DHS, JG, JK, GR, MH), and a final hierarchy of the level of confidence in the cancer diagnosis was constructed. The hierarchy gave greater weight to objective evidence of malignancy (pathology report) or note from the appropriate type of expert (e.g. oncologist, radiation oncologist, dermatologist for skin cancers, etc.). Other sources included death certificates, admission notes, and discharge summaries ( 1: Table S1).

Adjudication of malignancies

Source documents were reviewed for each malignancy; the document with the highest hierarchical value was recorded (see above). Malignancies were defined as “definite” if a malignancy was reported on a biopsy report or from an oncologist or radiation oncologist. To be considered an incident malignancy, the date of diagnosis (month and year) also had to be submitted, and had to have occurred after enrollment in CORRONA. For certain malignancies, note from an appropriate specialty was considered sufficient to designate the case as “definite” (ex: skin cancer and dermatologist’s note). If the supporting documentation was a note from another type of specialist, and the date of diagnosis and histology were included either in the note or hand written on adverse event form, then the diagnosis was designated “probable”. If either the histology or the date of diagnosis were not present, then the case was designated “possible.” In some cases, the source documents or adverse event form noted that the case was not a malignancy. Disposition of reported cases is shown in Figure 1.

Disposition of physician reported cases*.* This represents the results of the adjudication of the 461 reported malignancies in 401 patients

Full size image

Statistical analysis

Outcomes for this analysis compared a gold standard (documentation of confirmed incident malignancies) to the CORRONA questionnaire as well as the adverse event form report of malignancy. Positive predictive value (PPV) of follow-up visit report and adverse event form was calculated compared to the gold standard with 95% confidence intervals (CI).

The demographics of patients in this analysis are shown in Table 1. Of the 20,839 RA patients in the CORRONA Registry, 401 had at least one rheumatologist-reported malignancy during follow up, and some patients had more than one incident malignancy reported during follow up. RA patients with malignancies differed from the subjects without malignancy: they were older (65.2 vs. 57.6 years, p < 0.0001), less likely to be female (63 vs. 73%, p <0.0001), and were more likely to be non-Hispanic White (89 vs. 84%, p = 0.0005). In addition, they also had longer disease duration at enrollment (12.6 vs. 9.6 years, p < 0.0001) and longer duration of follow-up within CORRONA (2.3 vs. 1.8 years, p < 0.0001). There was no significant difference in seropositive status between the two groups (71 vs. 68%, p = 0.31).

The various documents used to determine confidence in the diagnosis of malignancy are shown in Table 2. Of the 461 cases, there were corroborating documents for 303 (65.7%). For cases with multiple documents, the document with the highest validity as deemed by the authors was used. Source documents that were able to definitely confirm the diagnosis were available for about half of the reported malignancies. For 158 cases (34.3% of the reported malignancies), there was no additional confirmation beyond an adverse event form. Of these, 36 (22.8% without confirmation, 7.8% overall) were adjudicated as probable, as the AE form submitted both a histology and an incident date. An additional 79 (50% without confirmation, 17.1% overall) were adjudicated as possible incident malignancy. The remaining 43 (27.2% without confirmation, 9.3% overall) had insufficient data on the TAE Form and they were deemed unlikely to be an incident malignancy.

Reasons why cases were excluded (N = 57) from categorization as incident malignancies are shown in Figure 1. The most common reason for exclusion was determining, usually via source documents, that there was insufficient data to determine whether an incident malignancy had occurred (43 cases, 75.4% of excluded, 9.3% overall). Six cases were excluded as they were prevalent, but not incident malignancies (10.5% of excluded, 1.3% overall). Five cases were excluded as on corroboration, they were confirmed to not be malignancies (8.8% of excluded, 1.1% overall). Lastly, 3 cases were excluded as they were submitted twice (5.3% of excluded, 0.7% overall). All 234 definite malignancies had corroborating data to confirm date of onset and histology. Of the 69 probable malignancies, 36 had no corroborating records submitted, but had an incident date and history on the TAE form. Of the 101 possible malignancies, 79 had no records submitted. Of the possible malignancies, 60 had a histology submitted but no date of the malignancy. 13 had a date of the malignancy but no histology, and 28 had both no histology and no date.

The number and type of malignancies stratified by adjudicated status are found in Table 3. The most common reported malignancies were non-melanoma skin cancers (NMSC), 154 total cases, comprising 33.4% of all malignancies. Using the gold standard of medical records review, just under half (46.1%) of these reported NMSC could be classified as definite malignancies; adding probable cases increased that number to 68.8%. In addition, there were 9 melanomas reported, one of which was ultimately confirmed not a melanoma. The second most common reported malignancy was breast cancer (59 cases, 13% of all reported malignancies). Five different histologies were noted on records review (data not shown). About three-quarters (74.6%) of reported breast cases were adjudicated as definite or probable. Hematologic, prostate, lung, and colon cancers all accounted for between 4.6% - and 9.1% of the total reported malignancies. Lastly, 57 cases (11.1%) were proven on either adverse event forms or from source documents to actually not be an independent incident malignancy. This was most common with cancers that were not able to be specified (71.9% of all cases deemed not an incident malignancy). These cases were included in the analysis as on the TAE they were confirmed as malignancies by the primary site. In one case, a histology and incident date was listed but the primary organ was listed as “unknown.” This case was deemed a definite incident malignancy.

A comparison of the predictive value of the CORRONA questionnaire and the adverse event form to the gold standard of medical records review is shown in Table 4. Using only definite and probable cases, the questionnaire had a positive predictive value (PPV) of 0.66 (95% CI 0.61 – 0.70). If possible cases were added, this increased to 0.86 (95% CI 0.83 – 0.89). If information from the TAE form was also used, the PPV was 0.68 (95% CI 0.63 – 0.72) for definite and probable cases. If possible cases were included, the PPV increased to 0.89 (95% CI 0.85 – 0.91).

Accurate classification of incident malignancies is essential for pharmacoepidemiologic studies, as well as to compare rates between different cohort studies and assess the relationship between disease characteristics and treatments with the development of cancer. Patient self-report may be an imperfect method for ascertaining incident malignancies, as the difference between a benign mass and true neoplasm is not always clearly communicated or understood. Even physician report may be unreliable, as the treating rheumatologist may not have the primary information about an incident malignancy. Administrative databases based on claims data are also not ideal, as the methodology used to define malignancy can cause the positive predictive value to vary widely [21, 22]. We attempted to validate physician report of incident malignancy based on physician questionnaires and adverse event forms using a gold standard of pertinent medical records review in a large cohort of patients with RA. We were able to confirm approximately two-thirds of reported incident malignancies. We found that having a positive report of incident malignancy on the adverse event forms increased the PPV only slightly compared to questionnaires alone. However, the PPV of the physician report alone may not be sufficiently high enough to be relied on for epidemiologic purposes. Hence, use of non-validated reports of incident malignancy might overestimate the true incidence in patients with RA and might bias the results of pharmacoepidemiologic studies assessing causal associations with specific treatments.

Our second notable finding was the exclusion of 14 malignancies (3.0%) because they were either duplicate entries, established cancers that were not incident, or proven by corroborating documents to not be malignant. An additional 43 cases (9.3%) had insufficient data and could not even be deemed possible malignancies. Clearly some of those are incident cases, although it is not clear how many. In addition, 101 cases (21.9%) could at best be classified as possible incident malignancies (either histology report or date of diagnosis missing). The accuracy of classifying cancer in a study assessing the risk of different treatments is important. If the possible malignancies are dismissed as non-cancers, it might artificially create a sense of decreased risk for a given medication, when at least some of those cases are likely to be true malignancies. Alternatively, including all possible malignancies might change the perception of the risk/benefit ratio and prevent patients or physicians from choosing a medication that could have benefit. The first approach would provide greater specificity; however, this is at the sacrifice of sensitivity. Assessing both rates would provide a range within which the true value most likely exists, but also illustrates the degree of uncertainty generated by physician report and even follow-up adverse event forms.

Our third significant finding was that despite follow-up with the primary treating rheumatologist, almost a third of the time, we could not obtain source documents to corroborate the malignancy. In some cases, this may be due to lack of diligence by the site. However, in most cases, this was because records truly could not be obtained. As a result, since 2008 the registry now requests source documents at the time the cancer is initially reported, to avoid requesting records of events from multiple years earlier. This proactive approach has improved the proportion of cases with source documents.

The methodology defines the classification of the malignancies, as it does depend on appropriate documentation submitted from the patient’s rheumatologist. The use of appropriate specialists for specific malignancies, such as a dermatologist for a skin cancer or a urologist for a genitourinary cancer, was deemed appropriate as for those malignancies where that specialist will treat the patient directly, as opposed to having an oncologist manage treatment. All ‘possible’ malignancies were classified as such because of some inadequacy of the requisite data to have confidence that it was truly an incident case.

The type of dataset itself also can impact the rate of malignancy found in a given population. The intensity of evaluation is much higher in a clinical trial, and virtually all cases of malignancy during the follow-up period would be expected to be reported. However, Phase II and III randomized controlled trials only follow patients with a single intervention for a finite period, and thus cannot offer the kind of data that can be possible in a long-term disease registry. It is possible that the intensity of surveillance itself can impact clinical care and outcome (Hawthorne effect), but it is still unclear if it will affect the overall incidence of a specific comorbidity such as malignancy in patients with RA.

The strengths of this analysis include the use of a very large dataset which includes information gathered from both providers and patients in the context of an established infrastructure which permitted the identification and follow-up of all reported cases. This allowed us to procure source documents in most cases. In addition, we used a standardized record review process with multiple adjudicators. All of these steps increased the validity of the results.

This study did have certain weaknesses as well. Although standard questions on the development of new malignancies were included in both the physician and patient questionnaires at each visit, it is still possible that some malignancies were not reported. As previously noted, the absence of a national cancer registry in the US makes independent assessment of new malignancies extremely difficult if not impossible. As such, we cannot calculate a negative predictive value of the default evaluation of “no malignancy.” Individual state cancer registries are available, are sponsored by both the NCI and the CDC, and their reliability continues to improve, though they are not reflective of nationwide rates, as the SEER data is. However, identification of individual patients requires the use of personal health information (PHI), and the CORRONA consent does not permit the use of PHI to cross-link with other registries. In addition, despite repeated attempts to obtain source documents, in many cases none were available. This led to the classification of many malignancies as “possible,” even while reported independently on the primary questionnaire from either the patient or treating physician. Certainly many of these cases likely were true malignancies. We believe that this problem is likely to be endemic to all observational registry studies in the absence of a national cancer registry. The response rate for pertinent records did improve when cases were reported to CORRONA more recently. However, there was no significant difference in rate of excluded cases (either not an incident malignancy or confirmed to be not a malignancy) from earlier versus more recent cases. A cancer registry where all malignancies are reported and validated, as is frequently done outside the United States, is superior to our methodology. However, information from within the United States is still of great value, as drug utilization patterns in the US are quite different from European registries where penetration of biologic agents is significantly greater than in Europe[23, 24]. We therefore believe that it is critical to appropriately analyze data on major comorbidites from a US source.

In conclusion, our process of confirming malignancies started with the identification of possible cases reported at the time of a routine clinic visit. It was further refined through subsequent hierarchical steps which included a targeted adverse event form and subsequent review and ranking of available source documents by a team of physicians. As a result of these rigorous adjudication steps, we found that routine reports of malignancy by patients and physicians were not always accurate. The implications of over and underreporting in large disease registry may be epidemiologically significant. We believe that it would be ideal if uniform standards for reporting of these events could be adopted in observational disease registries where cancer registries are not available.

Affiliations

1. Massachusetts General Hospital, 55 Fruit St, Boston, MA, 02114, USA

   • Mark C Fisher

2. Department of Rheumatology, New York University (NYU) Hospital for Joint Diseases, 301 East 17th Street, Suite 1410, New York, NY, 10003, USA

   • Victoria Furer

3. Departments of Medicine and Epidemiology and Preventive Medicine, University of Maryland School of Medicine, 10 S. Pine St., MSTF 8-34, Baltimore, MD, 21201, USA

   • Marc C Hochberg

4. Department of Rheumatology, New York University (NYU) Hospital for Joint Diseases, 301 East 17th Street, Suite 1410, New York, NY, 10003, USA

   • Jeffrey D Greenberg

5. Center for Rheumatology, Albany Medical College, State University of New York, 1367 Washington Avenue, Albany, NY, 12206, USA

   • Joel M Kremer

6. Department of Medicine, University of Alabama, 510 20th St. South FOT 805D, Birmingham, AL, 35294, USA

   • Jeff R Curtis

7. Division of Preventive and Behavioral Medicine, University of Massachusetts Medical School, 377 Plantation Street, Worcester, MA, 01605, USA

   • George Reed

8. Department of Medicine, University of Massachusetts Medical School, 377 Plantation Street, Worcester, MA, 01605, USA

   • Leslie Harrold

9. Division of Rheumatology, Immunology and Allergy, Brigham and Women’s Hospital, 1620 Tremont Street, Suite 3030, Boston, MA, 02120, USA

   • Daniel H Solomon

Corresponding author

Correspondence to Mark C Fisher.

Competing interests

Dr. Fisher has nothing to disclose. Dr. Furer has nothing to disclose. Dr. Hochberg has consulted for Abbott Laboratories, Amgen, Bristol Myers Squibb, Genentech, UCB Inc. Dr. Greenberg receives salary support from research grants from the NIH (K23AR054412), the Arthritis Foundation and the National Arthritis Research Foundation, serves as Chief Scientific Officer for CORRONA and has served on Advisory Boards for Genentech and Pfizer. Dr. Kremer is the President of CORRONA, receives salary support. Dr. Curtis has consulted for or received honoraria from Roche/Genentech, UCB, Centecor, CORRONA, Amgen, and Pfizer, and has received research support from Amgen, Roche/Genentech, Centecor and CORRONA. Dr. Curtis also receives salary support from the NIH (AR053351). Dr. Harrold received support from the NIH (K23AR053856). She also has consulted for CORRONA. Dr. Solomon has received research support from Amgen and Abbott for rheumatoid arthritis research and has received support for an educational course from Bristol Myers Squibb. Dr. Solomon also receives salary support from the NIH for mentoring (K24 AR 055989).

Authors’ contributions

MCF, VF, JD, and DH participated in the study design, data acquisition, data analysis, and manuscript preparation. MCH, JK, JC, GR, and LRH participated in data analysis and manuscript preparation. All authors read and approved the final manuscript.

Reprints and Permissions