Systematic assessment of the quality of osteoporosis guidelines
BMC Musculoskeletal Disorders volume 3, Article number: 20 (2002)
Numerous agencies have developed clinical practice guidelines for the management of postmenopausal osteoporosis. The study objective was to conduct a systematic assessment of the quality of osteoporosis guidelines produced since 1998.
Guidelines were identified by searching MEDLINE (1998+), the world wide web, known guideline developer websites, bibliographies of retrieved guidelines, and through consultation with content experts. Each guideline was then assessed by three independent appraisers using the 'Appraisal Instrument for Clinical Guidelines' (version 1) by Cluzeau.
We identified 26 unique guidelines from 1998–2001 and 21 met our inclusion criteria. Of the 21 guidelines reviewed, 8 were developed by medical societies, 6 by national groups, 6 by government agencies, and 1 by an international group. Twelve of the guidelines were published, 7 were organizational reports, and 2 were accessible only from the web. Half or more of the 20 items assessing the rigor of guideline development were met by 15% (median quality score 23%, range 5–80%, (95% CI 16.5, 34.7)), 81% met at least half of the 12 items assessing guideline content and context (median score 58%, range 17–83%, (95% CI 50.8, 65.5)), and none met half or more of the items assessing guideline application (median score 0%, range 0–47%, (95% CI -0.5 to 12.6)). Eight guidelines described the method used to assess the strength of evidence, and in 6 there was an explicit link between recommendations and the supporting evidence. Ten guidelines were judged not suitable for use in practice, 10 were acceptable with modification, and one was acceptable for use without modification.
The methodological quality of current osteoporosis guidelines is low, although their scores for clinical content were higher. Virtually no guidelines covered dissemination issues. Few guidelines were judged as acceptable for use in their current format.
Clinical practice guidelines are "systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances" . In many respects this is an ideal time to develop guidelines in the management of postmenopausal osteoporosis, since large randomized controlled trials of various therapies with clinically important outcome measures such as fractures have been conducted . In the past few years, numerous agencies and organizations have developed clinical practice guidelines (CPGs) for the management of post-menopausal osteoporosis. For the primary care physician, it is difficult to assimilate all the information contained in the numerous guidelines and decide which guideline to use in clinical practice. Recent research demonstrated that the application of two different guidelines  and  to a cohort of women who had sustained a fracture resulted in quite different estimates of the proportion of women who should be treated (87.9 versus 58.9%) .
CPGs may have the potential to improve the use of research findings and ultimately the care of patients, but only if the guidelines are associated with rigorous development. Special consideration should be given to the implementation of guidelines, including the needs of the users. A recent needs assessment conducted by Jaglal et al. revealed that family physicians wanted guidelines or information aids about osteoporosis that have significant input from family physicians . Additionally, with looming health care shortages, CPGs are increasingly used as the basis for the rationing of resources. It is therefore important that the guidelines are developed with methodological rigour, then appropriately disseminated and implemented .
The objective of this study was to identify and conduct a systematic assessment of the most recent postmenopausal osteoporosis guidelines produced from 1998–2001 that made recommendations on therapy. This process involved the systematic evaluation of the quality of the guidelines using a validated quality assessment tool .
Guidelines pertaining to postmenopausal osteoporosis were identified by searching MEDLINE (1998+), the world wide web, known guideline developer websites, references of retrieved guidelines and through consultation with content experts. We restricted the search to English language items. Search terms included the following medical headings and text words: osteoporosis practice guidelines, osteoporosis, consensus, consensus statements, and osteoporosis standards. We also searched known websites listing guidelines: such as: http://www.guideline.gov, http://www.gacguidelines.ca, http://www.cma.ca/cpgs/index.asp.
We included those documents that had clinical recommendations, that used a group process to formulate recommendations and that included a bibliography of relevant references.
For each guideline, the following descriptive information was extracted: the year of development, type of developer, publication status (peer-reviewed or not published) stated industry sponsorship, and clinical recommendations of pharmaceutical and non-pharmaceutical interventions.
The standardized quality instrument used to assess the guidelines was the 'Appraisal Instrument for Clinical Guidelines' (version 1, 1999) by Cluzeau and colleagues . This instrument is currently widely used in the United Kingdom and Canada, and has been shown to have acceptable reliability and criterion validity . This instrument uses the Yes, No and Not Sure format, consists of 37 items and is divided into 3 dimensions which assess the adequacy of documentation (Table 1). Dimension 1 (20 items) focuses on the rigour of guideline development and reflects attributes necessary to enhance guideline validity and reliability. It inquires about the responsibility and endorsement of the guidelines, composition of the development group, identification and interpretation of the evidence, the link between the evidence and recommendations, and plans to update the guideline. Dimension 2 (12 items) focuses on the context and content and addresses attributes of reliability, applicability, flexibility and clarity. The 12 items appraise the objectives of the guidelines, the presentation and format, the estimated outcomes-benefits, costs and harms. Dimension 3 (5 items) addresses the dissemination, implementation, and monitoring of the guidelines.
We included an evaluation of the global assessment of guidelines according to a similar strategy used in another study . Appraisers were asked whether they would "strongly recommend this guideline for use in practice without modifications", "recommend this guideline for use in practice on the condition of some alterations", or "not recommend this guideline". We also included a global quality rating using an 11-point scale, with 0 indicating the lowest possible quality and 10 representing the highest possible quality . Where the information pertaining to background development was lacking, we searched listed websites and attempted to contact authors for further details.
Three appraisers using the Cluzeau instrument independently assessed each guideline. The three appraisers consisted of a rheumatologist/epidemiologist (AC), physiotherapist/methodologist (LW) and one of either: geriatrician/epidemiologist (MM-S-H), geriatrician (AB) and clinical biochemist (DO). Using the assessments, we calculated the frequency with which the guidelines adhered to each of the 37 appraisal items. Adherence was defined by agreement by at least two of the three appraisers.
Next, using the values assigned by each appraiser for each item, we calculated a summary score for each dimension: Dimension 1 – Rigour of development, Dimension 2-Context and content and Dimension 3-Application. A "yes" response by an appraiser was assigned a value of 1 and all other responses 0.
The dimension quality score was derived by calculating the mean of the appraisers' scores. Each dimension was expressed as a score out of 100% (95% CI) so that scores could be compared across the three dimensions.
Appraiser agreement was evaluated by calculating the percentage of guidelines for which the three appraisers scored the quality of each dimension within 20 percentage points of each other and by calculating intraclass correlation coefficients (ICC) for the three dimensions. Reliability was assessed by internal consistency (Cronbach's alpha) of each dimension by calculating the correlation between all items within a dimension to test to what extent they measured the same underlying concept. Pearson's correlation coefficients were used to evaluate the correlation between appraisers' dimension scores and global quality scores. Differences in mean quality were analyzed using the Mann-Whitney test. All analyses were completed using SPSS/PC Windows version 10.0.
We identified 26 unique guidelines from 1998–2001: 21 were retrieved and on assessment met our inclusion criteria [3, 4, 9–27]. Five were excluded for the following reasons: three did not include clinical recommendations or were review articles [28–30] and two were not retrievable [31, 32]. Consultation with content experts identified seven guidelines, results of a Medline search identified four further guidelines; search of the World Wide web yielded three guidelines, and known guideline websites yielded six potential guidelines and review of references a further seven guidelines. Eight of the 21 guidelines reported receiving outside funding, and 6/8 of the guidelines acknowledged funding from industry.
Twelve guidelines were published, seven were organizational reports, and two were accessible only from the web. Guideline developers included medical societies (n = 8), national groups (n = 6), international groups (n = 1), and government agencies (n = 6). Table 1 presents the percentage of the guidelines that satisfied each of the 37 quality criteria. Figure 1 illustrates the distribution of the guidelines by dimension quality score. Twenty of the 21 guidelines did not include patients/consumers on the development panel. Only 11 of the 21 included family physicians.
Half or more of the 20 items assessing the rigor of guideline development were met by 15% of the guidelines (median quality score 23%, (95% CI 16.5, 34.7, range 5–80%), 81% met at least half of the 12 items assessing guideline content and context (median score 58%, (95% CI 50.8, 65.5), range 17–83%), and none met half or more of the items assessing guideline application (median score 0%, (95% CI -0.5, 12.6) range 0–47%).
Table 2 presents the mean dimension quality scores for the 21 osteoporosis CPGs. Eight guidelines described the method used to interpret and assess the strength of evidence. Six of the 21 guidelines provided an explicit link between major recommendations and supporting evidence.
The mean global quality score was 5.23 (95% CI 4.52, 5.94; median 5.33, standard deviation 1.56), suggesting that the appraisers felt the guidelines to be of medium quality.
The mean global assessment rating demonstrated that 11 of the 21 guidelines were recommended for use in practice (1 without change, 10 with modifications) and 10 guidelines were not recommended for use in practice.
Assessment of the reliability of the guideline appraisal instrument showed that D1 and D3 had good internal consistency, and D2 was borderline. The Cronbach's alpha was 0.87 for rigour of guideline development, 0.64 for context and content, and 0.71 for application.
For agreement between appraisers, the intraclass correlation coefficients were 0.91 for D1, 0.63 for D2 and 0.93 for D3. The appraisers' scores were within 20 percentage points for 95%, 67% and 100% of the guidelines for D1, 2, and 3, respectively. The Pearson's correlation coefficients between appraisers' dimension scores and their global quality scores (0–10) were 0.71 for D1, 0.60 for D2 and 0.46 for D3, with all coefficients being significant. The Pearson's correlation coefficients between appraisers' dimensions scores and their overall global assessment (recommend or not) were 0.59, 0.61 and 0.49 for D1, D2 and D3 respectively, with all coefficients being significant.
Eighteen of 21 guidelines included both pharmaceutical and non-pharmaceutical options (fall prevention, exercise or hip protectors). Ten of the guidelines included recommendations on fall prevention, and nine included recommendations on hip protectors.
There was no significant difference in the mean quality assessment when analyzed according to industry funded versus non-funded guidelines, type of publication (published, web or report), or the type of organization. There was no significant difference in quality according to year of publication, although there appeared to be a trend to improved quality from 1998 to 2001. Only 1/8 of the guidelines that acknowledged funding gave evidence that the potential biases were taken into account.
The methodological quality with respect to the rigor of development and implementation of current osteoporosis guidelines is low, although their scores with respect to clinical content are higher. Virtually no guidelines covered the application/dissemination issues. In particular, they did not define outcomes that could be monitored and failed to identify key elements that could be monitored by local guideline groups. Only one guideline was judged acceptable for use in its current format . Clinical practice guideline developers are usually clinicians, or methodologists who have little or no training in the behavioural and organization sciences. Tremendous time and effort are devoted to the development of evidence-based guidelines and the dissemination and implementation are often considered to be a separate component which has only recently thought to be the jurisdiction of the CPG developers.
Our findings are similar to other published reviews of guidelines [8, 33]. We also chose to limit our study to one clinical area and to focus on recent guidelines from 1998–2001, recognizing that guidelines currently being developed may follow a more rigorous methodology. Our results indicate that it is important to search both the Internet and Medline as some guidelines were only posted on the web and that retrieving all relevant guidelines can be a difficult task that requires a comprehensive search strategy.
Another potential limitation is that very few of the guidelines encouraged patient input into the development of CPGs. Potential benefits of including patients in the guideline development process include incorporation of priority setting from the patient perspective, improved adherence with recommendations, and reinforcement of the shared decision-making process.
Twenty-nine percent of the guidelines acknowledged involvement or support from industry. Choudry et al. conducted a survey of CPG authors and found 87% had some form of interaction with the pharmaceutical industry, with 58% receiving financial support for research. Given the heavy involvement of industry in the osteoporosis field, one would anticipate that the majority of the guideline developers would have relationships with industry that predated the guideline creation process. Industry has a vested interest in supporting the recommendations of osteoporosis guidelines, which raises concerns about industry's potential to influence the guideline development process. Acknowledgement of the degree of support/involvement with industry in guideline development is important and Choudry et al.  recommended that there be a formal process for discussing these conflicts prior to CPG development.
In addition to the goal of promoting good clinical practice and enhancing appropriate management, another potential goal of developing clinical practice guidelines is to improve the use of health care resources. However, we found that over two-thirds of guidelines neglected to include costs. The majority of guideline recommendations are based on results from efficacy, not effectiveness studies and therefore are not always useful in clinical practice . While the methodological quality of guidelines is a prerequisite to the development of good guidelines, it is not sufficient to ensure improvements in clinical practice. In addition, research methodologies on the impact of evidence-based guidelines in clinical practice would be valuable.
Educational material, including the dissemination of practice guidelines has been shown to have little or no effect on promoting behavioural change among health professionals . In contrast, strategies that encourage the development of a local guideline initiative have been shown to result in an increased uptake and dissemination. Only one of the osteoporosis guidelines that we reviewed encouraged development of a local initiative .
It is clear from the results of our systematic assessment that the developers of future osteoporosis guidelines need to incorporate greater methodological rigor and patient involvement into the development process. It would also be beneficial if guideline developers could follow common standard of reporting similar to that used by CONSORT for RCTs .
The multitude of guidelines dealing with postmenopausal osteoporosis makes it difficult for individual clinicians to decide which guideline to follow. Perhaps greater collaboration and sharing of resources between developer groups would result in fewer guidelines, but with local input as appropriate. More of the resources and energy could then be directed towards the dissemination process.
Clinical Practice Guidelines: Directions for a New Program. Committee to Advise the Public Health Service on Clinical Practice. Edited by: Field MJ and Lohr KN. 1990, Washington, National Academic Press
Black DM, Cummings SR, Karpf DB: Randomized trial of effect of alendronate on risk of fracture in women with existing vertebral fractures. Fracture Intervention Trial Research Group. Lancet. 1996, 348: 1535-1541. 10.1016/S0140-6736(96)07088-2.
Eddy DM, Johnston CC, Cummings SR, Dawson-Hughes B, Lindsay R, Melton LJ: Osteoporosis: Review of the evidence for prevention, diagnosis and treatment and cost-effectiveness analysis. Osteoporosis Int. 1998, 8: 1-88. 10.1007/s001980050040.
Royal College of Physicians: Osteoporosis: Clinical guidelines for prevention and treatment. London. 1-110. November 2001, [http://www.rcplondon.ac.uk/pubs/wp_osteo_update.htm]
McLellan AR, Fraser M: Fractures in women over 50 years: Implications for the secondary prevention of osteoporotic fractures of the application of the NOF and RCP London treatment guidelines. J Bone Miner Res. 2001, 16 (1): S290-
Jaglal SB, Hawker G, Carroll J, McIsacc W, Jaakkimainen L, Cadarette S: Information needs of family physicians for the management of osteoporosis. J Bone Miner Res. 2001, 16 (1): S290-
Cluzeau FA, Littlejohns P, Grimshaw JM, Feder G, Moran S: Development and application of a generic methodology to assess the quality of clinical guidelines. Int J for Quality in Health Care. 1999, 11: 21-28. 10.1093/intqhc/11.1.21.
Graham ID, Beardall S, Carter AO, Glennie J, Hebert PC, Tetroe JM: What is the quality of drug therapy clinical practice guidelines in Canada. CMAJ. 2001, 165: 157-63.
American Association of Clinical Endocrinologists 2001. Medical guidelines for clinical practice for the prevention and management of postmenopausal osteoporosis. Endocrine Practice. 2001, 7 (4): 293-311.
Clinical management guidelines for osteoporosis in Hong Kong. Hong Kong MJ. 1998, 4: 423-431.
Guidelines of care on osteoporosis for the primary care physician. Foundation for Osteoporosis Research and Education. 1998, 1-59.
Adelizzi RA, Shapses SA: Recommended practice guidelines for the diagnosis and treatment of osteoporosis. Osteoporosis Coalition of New Jersey. 1998, 1-12.
American Medical Directors Association: Osteoporosis: Clinical practice guideline. 1998, 1-12.
Brigham and Womens Hospital: Osteoporosis: Guide to prevention, diagnosis and treatment. October 2001, [http://is.partners.org/handbook/quarkfiles/osteo/osteo_print.asp]
ACOG educational bulletin – Osteoporosis. Int J Gynecology and Obstetrics. 1998, 62: 193-201.
Baran RW, Kiel DP, Patterson H, Briesacher B, Erwin WG: Diagnosis and treatment of osteoporosis in long-term care facilities. Consult Pharm. 1998, 6: 685-699.
Brown JP, Fortier M, Khan A, Rowe T: An Evidence based review of the management of osteoporosis. J SOGC. 1-7. April 2001
Clinical Resource Efficiency Support Team: Guidance on the prevention and treatment of osteoporosis. Belfast. 2001, 1-35.
Kin Yuen Chui, Kendler D, Khan A, Brown J, Fortier M: Osteoporosis. J Obstet Gynaecol Can. 2001, 23: 978-9.
Klibanski A, Adams-Campbell L, Bassford T: Osteoporosis prevention, diagnosis, and therapy. NIH Consensus Conference. 2000, 1-45.
London department of health. Osteoporosis. November 2001, [http://www.doh.gov.uk/pub/docs/doh/osteo.pdf]
Osteoporosis; diagnostics and pharmacotherapy. Finland: Evidence-based guidelines. November 2001, [http://www.duodecem.fi/lcdnet/ENG/pdg/p20066.htm]
Meunier PJ, Delmas PD, Eastell R, McClung MR, Papapoulos S, Rizzoli R: Diagnosis and management of osteoporosis in postmenopausal women: clinical guidelines. Clinical Therapeutics. 1999, 21: 1025-1044. 10.1016/S0149-2918(99)80022-8.
Ontario Program for Optimal Therapeutics: Ontario guidelines for the prevention and treatment of osteoporosis. 1-49. November 2001, [http://www.opot.org/guidelines/osteoporosis.pdf]
Siebzehner MI: Consensus statement on prevention and treatment of osteoporosis. IMAJ. 2000, 2: 397-401.
Ullom-Minnich P: Prevention of osteoporosis and fractures. American Family Physician. 1999, 60: 194-202.
Osteoporosis clinical guideline. SAMJ. 2002, 90 (9): 905-944.
Compston J, Papapoulos SE, Blanchard I, on behalf of working party for European member states: Report on osteoporosis in the European community: current status and recommendations for the future. Osteoporos Int. 1998, 8 (6): 531-534. 10.1007/s001980050094.
Genant HK, Cooper C, Poor G, Reid I, Ehrlich G, Kanis J: Interim report and recommendation of the World Health Organization Task Force for Osteoporosis. Osteoporos Int. 1999, 10 (4): 259-64. 10.1007/s001980050224.
Miller P: Management of postmenopausal osteoporosis for primary care. Menopause. 1998, 5: 123-131.
Health Council of Netherlands: Prevention of osteoporotic fractures publication no 1998/05. 1998
Osteoporosis practice guidelines. Lebanese Medical Journal. 2002
Shaneyfelt T, Mayo-Smith M, Rothwangl J: Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature. JAMA. 2002, 281 (20): 1900-5.
Choudry NK, Stelfox HT, Detsky AS: Relationships between authors of clinical practice guidelines and the pharmaceutical industry. JAMA. 2002, 287 (5): 612-617.
Pilote L, IB Tager: Outcomes research in the development and evaluation of practice guidelines. BMC Health Serv Res. 2002, 2: 7-10.1186/1472-6963-2-7.
Bero LA, Grilli R, Grimshaw JM, Harvey E, Oxman AD, Thomson MA: Closing the gap between reserach and practice: an overview of systematic reviews of interventions to promote the implementation of research findings. BMJ. 1998, 317: 465-468.
Grilli R, Magrini N, Penna A, Mura G, Liberati A: Practice guidelines developed by specialty societies: the need for a critical appraisal. Lancet. 2000, 355: 103-6. 10.1016/S0140-6736(99)02171-6.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2474/3/20/prepub
We would like to thank Jennifer Clinch for her assistance with the ICC analysis. Dr. Cranney is funded by an Arthritis Society of Canada scholarship and Dr. Graham is a CIHR New Investigator.
AC participated in design of the study, evaluation of guidelines, data analysis, and drafting the manuscript. LW participated in the design, coordination of the study, evaluation of guidelines and commented on the paper. IG participated in design of the study, statistical analysis and writing of manuscript. MM participated in the evaluation of guidelines and provided feedback on the manuscript. AB participated in the evaluation of guidelines and provided feedback on manuscript. DSO participated in evaluation phase and was involved in preparation of the manuscript. All authors read and approved the final manuscript.
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.
About this article
Cite this article
Cranney, A., Waldegger, L., Graham, I. et al. Systematic assessment of the quality of osteoporosis guidelines. BMC Musculoskelet Disord 3, 20 (2002). https://doi.org/10.1186/1471-2474-3-20