Malignancy validation in a United States registry of rheumatoid arthritis patients
© Fisher et al.; licensee BioMed Central Ltd. 2012
Received: 16 January 2012
Accepted: 31 May 2012
Published: 31 May 2012
Physician reporting is commonly used to ascertain adverse events or outcomes measured in epidemiologic studies. However, little is known on the accuracy of physician reported malignancies compared to pertinent medical record review in large cohort studies.
The Consortium of Rheumatology Researchers of North America (CORRONA) registry gathers physician-completed questionnaires for rheumatoid arthritis (RA) patients, including request for information on incident malignancies, approximately every three months. For incident malignancies reported from October 1st, 2001, through December 31st, 2007, we retrospectively requested completion of a Targeted Adverse Event (TAE) form for additional information as well as primary source documents to adjudicate the malignancy reports. CORRONA has employed a prospective request for source documentation for these events since 2008. We classified each malignancy as definite, probable, possible, or not a malignancy.
From 20,837 RA patients enrolled in CORRONA, 461 incident malignancies were initially reported on physician questionnaires. After review of returned source documents with adjudication, 234 were deemed definite, 69 probable, 101 possible, and 57 not an incident malignancy. The positive predictive value (PPV) of initial physician report of a malignancy versus “definite or probable” malignancy based on adjudication was 0.66 (95% CI 0.61 - 0.70). The PPV was 0.68 (95% CI 0.63 – 0.72) when the subsequent TAE form also confirmed the presence of malignancy. When possible malignancies were included, the PPV of physician-reported malignancies without a subsequent TAE form increased to 0.86 (0.83 – 0.89), and with a subsequent TAE form, 0.89 (0.85-0.91).
Twelve percent of initial physician reports of incident malignancy could not be confirmed with review of source documents. The most common reason for lack of confirmation was inability to obtain documents or insufficient data in source materials. These results suggest that timely collection of relevant medical records and an adjudication process are required to improve the accuracy of cancer reporting in epidemiologic studies.
KeywordsMalignancy Rheumatoid arthritis Registry Validation
Rheumatoid Arthritis (RA) is an auto-immune, multi-system inflammatory disease with significant morbidity and mortality. In addition to its articular manifestations, RA has a variety of extra-articular complications, including malignancy [1, 2]. The impact of immunosuppressive treatments used to manage patients with RA may also influence the development of malignancies, as each of these treatments cause perturbations of the immune system that may lead to malignancy [3–6]. Because of the influences of disease activity and therapy on the immune system of patients with RA, as well as the relative infrequent occurrence of many types of malignancy, it can be difficult to assess the impact of a specific medication on the risk of malignancy.
To date, several epidemiologic and pharmacoepidemiology studies have been performed to assess the rate of malignancy in RA patients treated with Tumor Necrosis Factor a inhibitors (TNFi) [7–18]. Sources for these datasets include national registries, prospective cohorts, and administrative databases. Cancer outcomes in these studies have been identified based on national cancer registers, patient self-report, physician report, administrative claims, and medical records. In the United States, common methods for ascertainment of malignancy among RA patients include patient self-report or physician report. Another approach is mining of centralized administrative databases, often using claims data or ICD-9 codes to document presence of malignancy, as in the US, national malignancy registries do not exist. The Surveillance Epidemiology and End Results (SEER) registry does capture data on incident malignancies in the US, but it only collects data for a handful of states as well as some additional, separate urban areas. There are also state specific cancer registries, however, between the overall number of RA patients and the overall low rate of malignancy among RA patients, they often do not have the power necessary for pharmacoepidemiologic studies. In addition, individual patient information is not available in many of these registries to permit linkage to observational registries of other diseases. Outside the United States, in addition to the above methods, many countries have national cancer registries where reporting of all incident malignancies to a central database with confirmation is mandatory.
To classify incident cancers in a large cohort of patients with RA where a cancer registry was not available, primary records were requested to validate the malignancy. We developed an adjudication process and tested the accuracy of physician-reported incident cancers, using pertinent medical records as the gold standard.
The Consortium of Rheumatology Researchers of North America (CORRONA) is an independent registry of RA patients that has collected clinical, laboratory, imaging, medication, and toxicity data since 2001. To date, it has collected data from over 100 rheumatology practices and over 300 participating rheumatologists throughout the United States, both academic and private, with over 20,000 RA patients enrolled. Data are collected from both patients and their treating rheumatologists using questionnaires, which gather information on disease duration, prognostic information, physician and patient-determined standardized disease severity and activity measures, medical comorbidities, use of medications including DMARDs, laboratory values, and adverse events . Follow-up assessments are requested at four month intervals and completed during routine clinical encounters. Approvals for participation in the CORRONA registry are obtained from the respective Institutional Review Boards of participating academic sites and a central Institutional Review Board for private practice sites.
At each visit, physicians complete questionnaires that include information about new comorbidities, including cancer. For the period of this study (10/01 to 12/07), follow-up Targeted Adverse (TAE) Forms and request for source documents were performed retrospectively. Since 2008, TAE forms and requests for source documents are initiated at the time of the initial physician report. Only the reporting rheumatologists’ office can request medical records due to privacy requirements established by the CORRONA registry. The medical records requested include pathology reports, hospital discharge summaries, notes from an oncologist, and/or notes from a primary care physician.
After the adverse event forms were returned, all sites were asked to provide source documents to corroborate the diagnosis of an incident malignancy. If source documents were not received within a specified period of follow-up, a second request was made. If source documents were still not submitted, a second questionnaire asking for validation specifically of the organ of the malignancy, pathology report, and date of onset was requested.
Medical record review
All malignancies reported between 10/1/2001 and 12/31/2007 were evaluated for this analysis. Once all data, including the adverse event forms and primary source documents, were received, they were reviewed using a structured abstraction form. In the analyses reported here, all data were reviewed and abstracted separately by two investigators (MF and VF). The abstracted data were then compared, and inconsistencies were resolved by returning to the source documents and assessing which record was accurate. As some data were subject to interpretation, any discrepancies between the two records were noted, and a third party (JG) reviewed the data. Three investigators (MF, VF, JG) then discussed the information. If consensus between MF and VF could not be reached, the third party (JG) adjudicated how the data would be recorded.
Source documents were ranked by two of the authors (MF and VF) for confidence in confirmation of the presence and type of malignancy. These rankings were then submitted to other members of the research team (DHS, JG, JK, GR, MH), and a final hierarchy of the level of confidence in the cancer diagnosis was constructed. The hierarchy gave greater weight to objective evidence of malignancy (pathology report) or note from the appropriate type of expert (e.g. oncologist, radiation oncologist, dermatologist for skin cancers, etc.). Other sources included death certificates, admission notes, and discharge summaries ( 1: Table S1).
Adjudication of malignancies
Outcomes for this analysis compared a gold standard (documentation of confirmed incident malignancies) to the CORRONA questionnaire as well as the adverse event form report of malignancy. Positive predictive value (PPV) of follow-up visit report and adverse event form was calculated compared to the gold standard with 95% confidence intervals (CI).
Total CORRONA Population
Total CORRONA Patients Without Malignancy
Total CORRONA Patients With Malignancies
(N = 20,839)
(N = 20,438)
(N = 401)**
Age (Mean, SD)
Female Gender (N, %)
Race/Ethnicity (N, %)
Seropositivity (N, %)
RA Disease Duration at Enrollment in CORRONA (mean years, SD)
Years of Follow-up After CORRONA Enrollment Until Report of Cancer* (mean, SD)
Hierarchy of Source Documents
Type of Document
Level of Confidence*
N, % Used**
Radiation Oncologist Note
Dermatologist note (skin cancers only)
OB-GYN Note (for OB-GYN cancers only)
Urologist Note (for GU cancers only)
Other Physician Note
Pharmaceutical Company adverse event form
Reasons why cases were excluded (N = 57) from categorization as incident malignancies are shown in Figure 1. The most common reason for exclusion was determining, usually via source documents, that there was insufficient data to determine whether an incident malignancy had occurred (43 cases, 75.4% of excluded, 9.3% overall). Six cases were excluded as they were prevalent, but not incident malignancies (10.5% of excluded, 1.3% overall). Five cases were excluded as on corroboration, they were confirmed to not be malignancies (8.8% of excluded, 1.1% overall). Lastly, 3 cases were excluded as they were submitted twice (5.3% of excluded, 0.7% overall). All 234 definite malignancies had corroborating data to confirm date of onset and histology. Of the 69 probable malignancies, 36 had no corroborating records submitted, but had an incident date and history on the TAE form. Of the 101 possible malignancies, 79 had no records submitted. Of the possible malignancies, 60 had a histology submitted but no date of the malignancy. 13 had a date of the malignancy but no histology, and 28 had both no histology and no date.
Reported Incident Malignancy Types by Adjudication
Organ or Type of Malignancy
Records Review – Definite*
Records Review – Probable**
Records Review – Possible***
Records Review – Not Malignancy or Not An Incident Malignancy****
Skin – NMSC1
Skin – Melanoma
Skin – Other2
Head and Neck6
Predictive Values versus Gold Standard of Records Review
PPV Physician Questionnaire (95% CI)
PPV Adverse Event Form (95% CI)
Accurate classification of incident malignancies is essential for pharmacoepidemiologic studies, as well as to compare rates between different cohort studies and assess the relationship between disease characteristics and treatments with the development of cancer. Patient self-report may be an imperfect method for ascertaining incident malignancies, as the difference between a benign mass and true neoplasm is not always clearly communicated or understood. Even physician report may be unreliable, as the treating rheumatologist may not have the primary information about an incident malignancy. Administrative databases based on claims data are also not ideal, as the methodology used to define malignancy can cause the positive predictive value to vary widely [21, 22]. We attempted to validate physician report of incident malignancy based on physician questionnaires and adverse event forms using a gold standard of pertinent medical records review in a large cohort of patients with RA. We were able to confirm approximately two-thirds of reported incident malignancies. We found that having a positive report of incident malignancy on the adverse event forms increased the PPV only slightly compared to questionnaires alone. However, the PPV of the physician report alone may not be sufficiently high enough to be relied on for epidemiologic purposes. Hence, use of non-validated reports of incident malignancy might overestimate the true incidence in patients with RA and might bias the results of pharmacoepidemiologic studies assessing causal associations with specific treatments.
Our second notable finding was the exclusion of 14 malignancies (3.0%) because they were either duplicate entries, established cancers that were not incident, or proven by corroborating documents to not be malignant. An additional 43 cases (9.3%) had insufficient data and could not even be deemed possible malignancies. Clearly some of those are incident cases, although it is not clear how many. In addition, 101 cases (21.9%) could at best be classified as possible incident malignancies (either histology report or date of diagnosis missing). The accuracy of classifying cancer in a study assessing the risk of different treatments is important. If the possible malignancies are dismissed as non-cancers, it might artificially create a sense of decreased risk for a given medication, when at least some of those cases are likely to be true malignancies. Alternatively, including all possible malignancies might change the perception of the risk/benefit ratio and prevent patients or physicians from choosing a medication that could have benefit. The first approach would provide greater specificity; however, this is at the sacrifice of sensitivity. Assessing both rates would provide a range within which the true value most likely exists, but also illustrates the degree of uncertainty generated by physician report and even follow-up adverse event forms.
Our third significant finding was that despite follow-up with the primary treating rheumatologist, almost a third of the time, we could not obtain source documents to corroborate the malignancy. In some cases, this may be due to lack of diligence by the site. However, in most cases, this was because records truly could not be obtained. As a result, since 2008 the registry now requests source documents at the time the cancer is initially reported, to avoid requesting records of events from multiple years earlier. This proactive approach has improved the proportion of cases with source documents.
The methodology defines the classification of the malignancies, as it does depend on appropriate documentation submitted from the patient’s rheumatologist. The use of appropriate specialists for specific malignancies, such as a dermatologist for a skin cancer or a urologist for a genitourinary cancer, was deemed appropriate as for those malignancies where that specialist will treat the patient directly, as opposed to having an oncologist manage treatment. All ‘possible’ malignancies were classified as such because of some inadequacy of the requisite data to have confidence that it was truly an incident case.
The type of dataset itself also can impact the rate of malignancy found in a given population. The intensity of evaluation is much higher in a clinical trial, and virtually all cases of malignancy during the follow-up period would be expected to be reported. However, Phase II and III randomized controlled trials only follow patients with a single intervention for a finite period, and thus cannot offer the kind of data that can be possible in a long-term disease registry. It is possible that the intensity of surveillance itself can impact clinical care and outcome (Hawthorne effect), but it is still unclear if it will affect the overall incidence of a specific comorbidity such as malignancy in patients with RA.
The strengths of this analysis include the use of a very large dataset which includes information gathered from both providers and patients in the context of an established infrastructure which permitted the identification and follow-up of all reported cases. This allowed us to procure source documents in most cases. In addition, we used a standardized record review process with multiple adjudicators. All of these steps increased the validity of the results.
This study did have certain weaknesses as well. Although standard questions on the development of new malignancies were included in both the physician and patient questionnaires at each visit, it is still possible that some malignancies were not reported. As previously noted, the absence of a national cancer registry in the US makes independent assessment of new malignancies extremely difficult if not impossible. As such, we cannot calculate a negative predictive value of the default evaluation of “no malignancy.” Individual state cancer registries are available, are sponsored by both the NCI and the CDC, and their reliability continues to improve, though they are not reflective of nationwide rates, as the SEER data is. However, identification of individual patients requires the use of personal health information (PHI), and the CORRONA consent does not permit the use of PHI to cross-link with other registries. In addition, despite repeated attempts to obtain source documents, in many cases none were available. This led to the classification of many malignancies as “possible,” even while reported independently on the primary questionnaire from either the patient or treating physician. Certainly many of these cases likely were true malignancies. We believe that this problem is likely to be endemic to all observational registry studies in the absence of a national cancer registry. The response rate for pertinent records did improve when cases were reported to CORRONA more recently. However, there was no significant difference in rate of excluded cases (either not an incident malignancy or confirmed to be not a malignancy) from earlier versus more recent cases. A cancer registry where all malignancies are reported and validated, as is frequently done outside the United States, is superior to our methodology. However, information from within the United States is still of great value, as drug utilization patterns in the US are quite different from European registries where penetration of biologic agents is significantly greater than in Europe[23, 24]. We therefore believe that it is critical to appropriately analyze data on major comorbidites from a US source.
In conclusion, our process of confirming malignancies started with the identification of possible cases reported at the time of a routine clinic visit. It was further refined through subsequent hierarchical steps which included a targeted adverse event form and subsequent review and ranking of available source documents by a team of physicians. As a result of these rigorous adjudication steps, we found that routine reports of malignancy by patients and physicians were not always accurate. The implications of over and underreporting in large disease registry may be epidemiologically significant. We believe that it would be ideal if uniform standards for reporting of these events could be adopted in observational disease registries where cancer registries are not available.
- Cash JM, Klippel JH: Second-line drug therapy for rheumatoid arthritis. N Engl J Med. 1994, 330: 1368-1375. 10.1056/NEJM199405123301908.View ArticlePubMedGoogle Scholar
- Gridley G, McLaughlin JK, Ekbom A, Klareskog L, Adami HO, Hacker DG, Hoover R, Fraumeni JF: Incidence of cancer among patients with rheumatoid arthritis. J Natl Cancer Inst. 1993, 85: 307-311. 10.1093/jnci/85.4.307.View ArticlePubMedGoogle Scholar
- Biancone L, Calabrese E, Petruzziello C, Pallone F: Treatment with biologic therapies and the risk of cancer in patients with IBD. Nat Clin Pract Gastroenterol Hepatol. 2007, 4: 78-91.View ArticlePubMedGoogle Scholar
- Bongartz T, Sutton AJ, Sweeting MJ, Buchan I, Matteson EL, Montori V: Anti-TNF antibody therapy in rheumatoid arthritis and the risk of serious infections and malignancies: systematic review and meta-analysis of rare harmful effects in randomized controlled trials. JAMA. 2006, 295: 2275-2285. 10.1001/jama.295.19.2275.View ArticlePubMedGoogle Scholar
- Brown SL, Greene MH, Gershon SK, Edwards ET, Braun MM: Tumor necrosis factor antagonist therapy and lymphoma development: twenty-six cases reported to the Food and Drug Administration. Arthritis Rheum. 2002, 46: 3151-3158. 10.1002/art.10679.View ArticlePubMedGoogle Scholar
- Kanik KS, Cash JM: Does methotrexate increase the risk of infection or malignancy?. Rheum Dis Clin N Am. 1997, 23: 955-967. 10.1016/S0889-857X(05)70368-9.View ArticleGoogle Scholar
- Askling J, Baecklund E, Granath F, Geborek P, Fored M, Backlin C, Bertilsson L, Coster L, Jacobsson LT, Lindblad S: Anti-tumour necrosis factor therapy in rheumatoid arthritis and risk of malignant lymphomas: relative risks and time trends in the Swedish Biologics Register. Ann Rheum Dis. 2009, 68: 648-653. 10.1136/ard.2007.085852.View ArticlePubMedGoogle Scholar
- Askling J, Fored CM, Baecklund E, Brandt L, Backlin C, Ekbom A, Sundstrom C, Bertilsson L, Coster L, Geborek P: Haematopoietic malignancies in rheumatoid arthritis: lymphoma risk and characteristics after exposure to tumour necrosis factor antagonists. Ann Rheum Dis. 2005, 64: 1414-1420. 10.1136/ard.2004.033241.View ArticlePubMedPubMed CentralGoogle Scholar
- Askling J, Fored CM, Brandt L, Baecklund E, Bertilsson L, Feltelius N, Coster L, Geborek P, Jacobsson LT, Lindblad S: Risks of solid cancers in patients with rheumatoid arthritis and after treatment with tumour necrosis factor antagonists. Ann Rheum Dis. 2005, 64: 1421-1426. 10.1136/ard.2004.033993.View ArticlePubMedPubMed CentralGoogle Scholar
- Chakravarty EF, Michaud K, Wolfe F: Skin cancer, rheumatoid arthritis, and tumor necrosis factor inhibitors. J Rheumatol. 2005, 32: 2130-2135.PubMedGoogle Scholar
- Geborek P, Bladstrom A, Turesson C, Gulfe A, Petersson IF, Saxne T, Olsson H, Jacobsson LT: Tumour necrosis factor blockers do not increase overall tumour risk in patients with rheumatoid arthritis, but may be associated with an increased risk of lymphomas. Ann Rheum Dis. 2005, 64: 699-703. 10.1136/ard.2004.030528.View ArticlePubMedPubMed CentralGoogle Scholar
- Setoguchi S, Solomon DH, Weinblatt ME, Katz JN, Avorn J, Glynn RJ, Cook EF, Carney G, Schneeweiss S: Tumor necrosis factor alpha antagonist use and cancer in patients with rheumatoid arthritis. Arthritis Rheum. 2006, 54: 2757-2764. 10.1002/art.22056.View ArticlePubMedGoogle Scholar
- Wolfe F, Michaud K: Lymphoma in rheumatoid arthritis: the effect of methotrexate and anti-tumor necrosis factor therapy in 18,572 patients. Arthritis Rheum. 2004, 50: 1740-1751. 10.1002/art.20311.View ArticlePubMedGoogle Scholar
- Wolfe F, Michaud K: Biologic treatment of rheumatoid arthritis and the risk of malignancy: analyses from a large US observational study. Arthritis Rheum. 2007, 56: 2886-2895. 10.1002/art.22864.View ArticlePubMedGoogle Scholar
- Wolfe F, Michaud K: The effect of methotrexate and anti-tumor necrosis factor therapy on the risk of lymphoma in rheumatoid arthritis in 19,562 patients during 89,710 person-years of observation. Arthritis Rheum. 2007, 56: 1433-1439. 10.1002/art.22579.View ArticlePubMedGoogle Scholar
- Amari W, Zeringue AL, McDonald JR, Caplan L, Eisen SA, Ranganathan P: Risk of non-melanoma skin cancer in a national cohort of veterans with rheumatoid arthritis. Rheumatology (Oxford). 50: 1431-1439.
- Khurana R, Wolf R, Berney S, Caldito G, Hayat S, Berney SM: Risk of development of lung cancer is increased in patients with rheumatoid arthritis: a large case control study in US veterans. J Rheumatol. 2008, 35: 1704-1708.PubMedGoogle Scholar
- Parikh-Patel A, White RH, Allen M, Cress R: Risk of cancer among rheumatoid arthritis patients in California. Cancer Causes Control. 2009, 20: 1001-1010. 10.1007/s10552-009-9298-y.View ArticlePubMedPubMed CentralGoogle Scholar
- Kremer J: The CORRONA database. Ann Rheum Dis. 2005, 64 (Suppl 4): v37-v41.Google Scholar
- Felson DT, Anderson JJ, Boers M, Bombardier C, Chernoff M, Fried B, Furst D, Goldsmith C, Kieszak S, Lightfoot R: The American College of Rheumatology preliminary core set of disease activity measures for rheumatoid arthritis clinical trials. The Committee on Outcome Measures in Rheumatoid Arthritis Clinical Trials. Arthritis Rheum. 1993, 36: 729-740. 10.1002/art.1780360601.View ArticlePubMedGoogle Scholar
- Setoguchi S, Solomon DH, Glynn RJ, Cook EF, Levin R, Schneeweiss S: Agreement of diagnosis and its date for hematologic malignancies and solid tumors between medicare claims and cancer registry data. Cancer Causes Control. 2007, 18: 561-569. 10.1007/s10552-007-0131-1.View ArticlePubMedGoogle Scholar
- Ramsey SD, Scoggins JF, Blough DK, McDermott CL, Reyes CM: Sensitivity of administrative claims to identify incident cases of lung cancer: a comparison of 3 health plans. J Manag Care Pharm. 2009, 15: 659-668.PubMedGoogle Scholar
- Curtis JR, Jain A, Askling J, Bridges SL, Carmona L, Dixon W, Finckh A, Hyrich K, Greenberg JD, Kremer J: A comparison of patient characteristics and outcomes in selected European and U.S. rheumatoid arthritis registries. Semin Arthritis Rheum. 2010, 40: 2-14 e11. 10.1016/j.semarthrit.2010.03.003.View ArticlePubMedPubMed CentralGoogle Scholar
- Kremer JM, Greenberg J: Interpreting registry-derived drug studies: does societal context matter?. Arthritis Rheum. 2009, 60: 3155-3157. 10.1002/art.24880.View ArticlePubMedGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2474/13/85/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.