Systematic assessment of the quality of osteoporosis guidelines
© Cranney et al; licensee BioMed Central Ltd. 2002
Received: 22 May 2002
Accepted: 12 August 2002
Published: 12 August 2002
Numerous agencies have developed clinical practice guidelines for the management of postmenopausal osteoporosis. The study objective was to conduct a systematic assessment of the quality of osteoporosis guidelines produced since 1998.
Guidelines were identified by searching MEDLINE (1998+), the world wide web, known guideline developer websites, bibliographies of retrieved guidelines, and through consultation with content experts. Each guideline was then assessed by three independent appraisers using the 'Appraisal Instrument for Clinical Guidelines' (version 1) by Cluzeau.
We identified 26 unique guidelines from 1998–2001 and 21 met our inclusion criteria. Of the 21 guidelines reviewed, 8 were developed by medical societies, 6 by national groups, 6 by government agencies, and 1 by an international group. Twelve of the guidelines were published, 7 were organizational reports, and 2 were accessible only from the web. Half or more of the 20 items assessing the rigor of guideline development were met by 15% (median quality score 23%, range 5–80%, (95% CI 16.5, 34.7)), 81% met at least half of the 12 items assessing guideline content and context (median score 58%, range 17–83%, (95% CI 50.8, 65.5)), and none met half or more of the items assessing guideline application (median score 0%, range 0–47%, (95% CI -0.5 to 12.6)). Eight guidelines described the method used to assess the strength of evidence, and in 6 there was an explicit link between recommendations and the supporting evidence. Ten guidelines were judged not suitable for use in practice, 10 were acceptable with modification, and one was acceptable for use without modification.
The methodological quality of current osteoporosis guidelines is low, although their scores for clinical content were higher. Virtually no guidelines covered dissemination issues. Few guidelines were judged as acceptable for use in their current format.
Clinical practice guidelines are "systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances" . In many respects this is an ideal time to develop guidelines in the management of postmenopausal osteoporosis, since large randomized controlled trials of various therapies with clinically important outcome measures such as fractures have been conducted . In the past few years, numerous agencies and organizations have developed clinical practice guidelines (CPGs) for the management of post-menopausal osteoporosis. For the primary care physician, it is difficult to assimilate all the information contained in the numerous guidelines and decide which guideline to use in clinical practice. Recent research demonstrated that the application of two different guidelines  and  to a cohort of women who had sustained a fracture resulted in quite different estimates of the proportion of women who should be treated (87.9 versus 58.9%) .
CPGs may have the potential to improve the use of research findings and ultimately the care of patients, but only if the guidelines are associated with rigorous development. Special consideration should be given to the implementation of guidelines, including the needs of the users. A recent needs assessment conducted by Jaglal et al. revealed that family physicians wanted guidelines or information aids about osteoporosis that have significant input from family physicians . Additionally, with looming health care shortages, CPGs are increasingly used as the basis for the rationing of resources. It is therefore important that the guidelines are developed with methodological rigour, then appropriately disseminated and implemented .
The objective of this study was to identify and conduct a systematic assessment of the most recent postmenopausal osteoporosis guidelines produced from 1998–2001 that made recommendations on therapy. This process involved the systematic evaluation of the quality of the guidelines using a validated quality assessment tool .
Guidelines pertaining to postmenopausal osteoporosis were identified by searching MEDLINE (1998+), the world wide web, known guideline developer websites, references of retrieved guidelines and through consultation with content experts. We restricted the search to English language items. Search terms included the following medical headings and text words: osteoporosis practice guidelines, osteoporosis, consensus, consensus statements, and osteoporosis standards. We also searched known websites listing guidelines: such as: http://www.guideline.gov, http://www.gacguidelines.ca, http://www.cma.ca/cpgs/index.asp.
We included those documents that had clinical recommendations, that used a group process to formulate recommendations and that included a bibliography of relevant references.
For each guideline, the following descriptive information was extracted: the year of development, type of developer, publication status (peer-reviewed or not published) stated industry sponsorship, and clinical recommendations of pharmaceutical and non-pharmaceutical interventions.
Proportion of Guidelines Meeting Quality Criteria
% CPGs where 2 or more appraisers said "yes"
Items included in Dimension 1: Rigour of Method Development
Is the agency responsible for the development and/or endorsement of the guidelines clearly identified?
Was funding or other support received for developing the guidelines?
If funding or other support was received, is there evidence that the potential biases of the funding body(ies) were taken into account?
Is there a description of the individuals (e.g. professionals, interest groups – including patients) who were involved in the guidelines development group?
If so, did the group contain representatives of all key disciplines?
Is there a description of the sources of information used to select the evidence on which the recommendations are based?
If so, are the sources of information adequate?
Is there a description of the method(s) used to interpret and assess the strength of the evidence?
If so, is(are) the method(s) for rating the evidence satisfactory?
Is there a description of the methods used to formulate the recommendations?
If so, are the methods satisfactory?
Is there an indication of how the views of interested parties not on the panel (or the consensus conference participants) were taken into account?
Is there an explicit link between the major recommendations and the level of supporting evidence?
Did the guidelines receive an independent external review prior to their publication/release?
If so, is explicit information given about methods and how comments were addressed?
Were the guidelines piloted?
If the guidelines were piloted, is explicit information given about the methods used and the results adopted?
Is there a mention of a date for reviewing or updating the guideline?
Is the body responsible for the reviewing and updating clearly identified?
Overall, have the potential biases of guideline development been adequately dealt with, i.e. have they attempted to minimize the introduction of bias adequately?
Items included in Dimension 2: Context and Content
Are the reasons for developing the guidelines clearly stated?
Are the objectives of the guidelines clearly defined?
Is there a satisfactory description of the patients to which the guidelines are meant to apply?
Is there a description of the circumstances (clinical or non-clinical) in which exceptions might be made in using the guidelines?
Is there an explicit statement of how patient preferences should be taken into account in applying the guidelines?
Do the guidelines describe the condition to be detected, treated, or prevented in unambiguous terms?
Are there different possible options for management of the condition clearly stated in the guidelines?
Are the recommendations clearly presented?
Is there an adequate description of the health benefits that are likely to be gained from the recommended management?
Is there an adequate description of the potential harms or risks that may occur as a result of the recommended management?
Is there an estimate of the costs or expenditures likely to incur from the recommended management?
Are the recommendations supported by the estimated benefits, harms and costs of the intervention?
Items included in Dimension 3: Application
Does the guideline document suggest possible methods for dissemination and implementation?
Does the guideline document specify criteria for monitoring adherence to the guidelines?
Does the guideline document identify clear standards or targets for adherence to the guidelines?
Does the guideline document define measurable outcomes (e.g. health, process, economic, outcomes) that can be monitored?
Does the guideline document identify key elements which need to be considered by local guideline groups?
We included an evaluation of the global assessment of guidelines according to a similar strategy used in another study . Appraisers were asked whether they would "strongly recommend this guideline for use in practice without modifications", "recommend this guideline for use in practice on the condition of some alterations", or "not recommend this guideline". We also included a global quality rating using an 11-point scale, with 0 indicating the lowest possible quality and 10 representing the highest possible quality . Where the information pertaining to background development was lacking, we searched listed websites and attempted to contact authors for further details.
Three appraisers using the Cluzeau instrument independently assessed each guideline. The three appraisers consisted of a rheumatologist/epidemiologist (AC), physiotherapist/methodologist (LW) and one of either: geriatrician/epidemiologist (MM-S-H), geriatrician (AB) and clinical biochemist (DO). Using the assessments, we calculated the frequency with which the guidelines adhered to each of the 37 appraisal items. Adherence was defined by agreement by at least two of the three appraisers.
Next, using the values assigned by each appraiser for each item, we calculated a summary score for each dimension: Dimension 1 – Rigour of development, Dimension 2-Context and content and Dimension 3-Application. A "yes" response by an appraiser was assigned a value of 1 and all other responses 0.
The dimension quality score was derived by calculating the mean of the appraisers' scores. Each dimension was expressed as a score out of 100% (95% CI) so that scores could be compared across the three dimensions.
Appraiser agreement was evaluated by calculating the percentage of guidelines for which the three appraisers scored the quality of each dimension within 20 percentage points of each other and by calculating intraclass correlation coefficients (ICC) for the three dimensions. Reliability was assessed by internal consistency (Cronbach's alpha) of each dimension by calculating the correlation between all items within a dimension to test to what extent they measured the same underlying concept. Pearson's correlation coefficients were used to evaluate the correlation between appraisers' dimension scores and global quality scores. Differences in mean quality were analyzed using the Mann-Whitney test. All analyses were completed using SPSS/PC Windows version 10.0.
We identified 26 unique guidelines from 1998–2001: 21 were retrieved and on assessment met our inclusion criteria [3, 4, 9–27]. Five were excluded for the following reasons: three did not include clinical recommendations or were review articles [28–30] and two were not retrievable [31, 32]. Consultation with content experts identified seven guidelines, results of a Medline search identified four further guidelines; search of the World Wide web yielded three guidelines, and known guideline websites yielded six potential guidelines and review of references a further seven guidelines. Eight of the 21 guidelines reported receiving outside funding, and 6/8 of the guidelines acknowledged funding from industry.
Half or more of the 20 items assessing the rigor of guideline development were met by 15% of the guidelines (median quality score 23%, (95% CI 16.5, 34.7, range 5–80%), 81% met at least half of the 12 items assessing guideline content and context (median score 58%, (95% CI 50.8, 65.5), range 17–83%), and none met half or more of the items assessing guideline application (median score 0%, (95% CI -0.5, 12.6) range 0–47%).
Mean Quality Scores of Osteoporosis Practice Guidelines (CPG), Maximum score = 100 for each Dimension (D)
Guideline Developer (Ref)
Acceptable for use
Rigor of Development
Context and Content
American College Obstetrics Gynecology 
American Family Physician 
Prevention of osteoporosis and fractures
Cdn Consensus Menopause and Osteoporosis 
Hong Kong PG 
Clinical management guidelines for osteoporosis in Hong Kong
ASCP American Society of Consultant Pharmacists 
Diagnosis and Treatment of OP in long term care Facilities
Israeli Centre for Technology Assessment 
Consensus statement on prevention and treatment of osteoporosis
Osteoporosis: Diagnostics and pharmacotherapy
Foundation for Osteoporosis Research and Education 
Guidelines of care on osteoporosis for the primary care physician
International Committee for Osteoporosis Clinical Guidelines 
Diagnosis and management of OP in postmenopausal women: clinical Guidelines
Brigham & Womens Hospital 
Osteoporosis: guide to prevention, Diagnosis and Treatment
Osteoporosis Coalition of New Jersey 
Recommended practice guidelines for the diagnosis and treatment of osteoporosis
American Medical Directors Association 
Osteoporosis: clinical practice guideline
Society for Obstetrician and Gynecologists of Canada 
An Evidence-based review of the management of osteoporosis
National Osteoporosis Foundation 
Osteoporosis; evidence for prevention, diagnosis and treatment and cost-effectiveness
American Association of Clinical Endocrinologists 
Guidelines for clinical practice for prevention and management of postmenopausal osteoporosis
NIH Consensus Statement 
Osteoporosis Prevention, Diagnosis and Therapy
CREST  Clinical Resource Efficiency Support Team
Guidance on the prevention and treatment of osteoporosis
London Department of Health 
South African Medical Journal 
Osteoporosis Clinical Guideline
Ontario Program of Optimal Therapeutics 
Ontario Guidelines for the prevention and treatment of osteoporosis
Royal College of Physicians 
Osteoporosis: clinical guidelines for prevention and treatment
The mean global quality score was 5.23 (95% CI 4.52, 5.94; median 5.33, standard deviation 1.56), suggesting that the appraisers felt the guidelines to be of medium quality.
The mean global assessment rating demonstrated that 11 of the 21 guidelines were recommended for use in practice (1 without change, 10 with modifications) and 10 guidelines were not recommended for use in practice.
Assessment of the reliability of the guideline appraisal instrument showed that D1 and D3 had good internal consistency, and D2 was borderline. The Cronbach's alpha was 0.87 for rigour of guideline development, 0.64 for context and content, and 0.71 for application.
For agreement between appraisers, the intraclass correlation coefficients were 0.91 for D1, 0.63 for D2 and 0.93 for D3. The appraisers' scores were within 20 percentage points for 95%, 67% and 100% of the guidelines for D1, 2, and 3, respectively. The Pearson's correlation coefficients between appraisers' dimension scores and their global quality scores (0–10) were 0.71 for D1, 0.60 for D2 and 0.46 for D3, with all coefficients being significant. The Pearson's correlation coefficients between appraisers' dimensions scores and their overall global assessment (recommend or not) were 0.59, 0.61 and 0.49 for D1, D2 and D3 respectively, with all coefficients being significant.
Eighteen of 21 guidelines included both pharmaceutical and non-pharmaceutical options (fall prevention, exercise or hip protectors). Ten of the guidelines included recommendations on fall prevention, and nine included recommendations on hip protectors.
There was no significant difference in the mean quality assessment when analyzed according to industry funded versus non-funded guidelines, type of publication (published, web or report), or the type of organization. There was no significant difference in quality according to year of publication, although there appeared to be a trend to improved quality from 1998 to 2001. Only 1/8 of the guidelines that acknowledged funding gave evidence that the potential biases were taken into account.
The methodological quality with respect to the rigor of development and implementation of current osteoporosis guidelines is low, although their scores with respect to clinical content are higher. Virtually no guidelines covered the application/dissemination issues. In particular, they did not define outcomes that could be monitored and failed to identify key elements that could be monitored by local guideline groups. Only one guideline was judged acceptable for use in its current format . Clinical practice guideline developers are usually clinicians, or methodologists who have little or no training in the behavioural and organization sciences. Tremendous time and effort are devoted to the development of evidence-based guidelines and the dissemination and implementation are often considered to be a separate component which has only recently thought to be the jurisdiction of the CPG developers.
Our findings are similar to other published reviews of guidelines [8, 33]. We also chose to limit our study to one clinical area and to focus on recent guidelines from 1998–2001, recognizing that guidelines currently being developed may follow a more rigorous methodology. Our results indicate that it is important to search both the Internet and Medline as some guidelines were only posted on the web and that retrieving all relevant guidelines can be a difficult task that requires a comprehensive search strategy.
Another potential limitation is that very few of the guidelines encouraged patient input into the development of CPGs. Potential benefits of including patients in the guideline development process include incorporation of priority setting from the patient perspective, improved adherence with recommendations, and reinforcement of the shared decision-making process.
Twenty-nine percent of the guidelines acknowledged involvement or support from industry. Choudry et al. conducted a survey of CPG authors and found 87% had some form of interaction with the pharmaceutical industry, with 58% receiving financial support for research. Given the heavy involvement of industry in the osteoporosis field, one would anticipate that the majority of the guideline developers would have relationships with industry that predated the guideline creation process. Industry has a vested interest in supporting the recommendations of osteoporosis guidelines, which raises concerns about industry's potential to influence the guideline development process. Acknowledgement of the degree of support/involvement with industry in guideline development is important and Choudry et al.  recommended that there be a formal process for discussing these conflicts prior to CPG development.
In addition to the goal of promoting good clinical practice and enhancing appropriate management, another potential goal of developing clinical practice guidelines is to improve the use of health care resources. However, we found that over two-thirds of guidelines neglected to include costs. The majority of guideline recommendations are based on results from efficacy, not effectiveness studies and therefore are not always useful in clinical practice . While the methodological quality of guidelines is a prerequisite to the development of good guidelines, it is not sufficient to ensure improvements in clinical practice. In addition, research methodologies on the impact of evidence-based guidelines in clinical practice would be valuable.
Educational material, including the dissemination of practice guidelines has been shown to have little or no effect on promoting behavioural change among health professionals . In contrast, strategies that encourage the development of a local guideline initiative have been shown to result in an increased uptake and dissemination. Only one of the osteoporosis guidelines that we reviewed encouraged development of a local initiative .
It is clear from the results of our systematic assessment that the developers of future osteoporosis guidelines need to incorporate greater methodological rigor and patient involvement into the development process. It would also be beneficial if guideline developers could follow common standard of reporting similar to that used by CONSORT for RCTs .
The multitude of guidelines dealing with postmenopausal osteoporosis makes it difficult for individual clinicians to decide which guideline to follow. Perhaps greater collaboration and sharing of resources between developer groups would result in fewer guidelines, but with local input as appropriate. More of the resources and energy could then be directed towards the dissemination process.
We would like to thank Jennifer Clinch for her assistance with the ICC analysis. Dr. Cranney is funded by an Arthritis Society of Canada scholarship and Dr. Graham is a CIHR New Investigator.
- Clinical Practice Guidelines: Directions for a New Program. Committee to Advise the Public Health Service on Clinical Practice. Edited by: Field MJ and Lohr KN. 1990, Washington, National Academic PressGoogle Scholar
- Black DM, Cummings SR, Karpf DB: Randomized trial of effect of alendronate on risk of fracture in women with existing vertebral fractures. Fracture Intervention Trial Research Group. Lancet. 1996, 348: 1535-1541. 10.1016/S0140-6736(96)07088-2.View ArticlePubMedGoogle Scholar
- Eddy DM, Johnston CC, Cummings SR, Dawson-Hughes B, Lindsay R, Melton LJ: Osteoporosis: Review of the evidence for prevention, diagnosis and treatment and cost-effectiveness analysis. Osteoporosis Int. 1998, 8: 1-88. 10.1007/s001980050040.Google Scholar
- Royal College of Physicians: Osteoporosis: Clinical guidelines for prevention and treatment. London. 1-110. November 2001, [http://www.rcplondon.ac.uk/pubs/wp_osteo_update.htm]
- McLellan AR, Fraser M: Fractures in women over 50 years: Implications for the secondary prevention of osteoporotic fractures of the application of the NOF and RCP London treatment guidelines. J Bone Miner Res. 2001, 16 (1): S290-Google Scholar
- Jaglal SB, Hawker G, Carroll J, McIsacc W, Jaakkimainen L, Cadarette S: Information needs of family physicians for the management of osteoporosis. J Bone Miner Res. 2001, 16 (1): S290-Google Scholar
- Cluzeau FA, Littlejohns P, Grimshaw JM, Feder G, Moran S: Development and application of a generic methodology to assess the quality of clinical guidelines. Int J for Quality in Health Care. 1999, 11: 21-28. 10.1093/intqhc/11.1.21.View ArticleGoogle Scholar
- Graham ID, Beardall S, Carter AO, Glennie J, Hebert PC, Tetroe JM: What is the quality of drug therapy clinical practice guidelines in Canada. CMAJ. 2001, 165: 157-63.PubMedPubMed CentralGoogle Scholar
- American Association of Clinical Endocrinologists 2001. Medical guidelines for clinical practice for the prevention and management of postmenopausal osteoporosis. Endocrine Practice. 2001, 7 (4): 293-311.Google Scholar
- Clinical management guidelines for osteoporosis in Hong Kong. Hong Kong MJ. 1998, 4: 423-431.Google Scholar
- Guidelines of care on osteoporosis for the primary care physician. Foundation for Osteoporosis Research and Education. 1998, 1-59.Google Scholar
- Adelizzi RA, Shapses SA: Recommended practice guidelines for the diagnosis and treatment of osteoporosis. Osteoporosis Coalition of New Jersey. 1998, 1-12.Google Scholar
- American Medical Directors Association: Osteoporosis: Clinical practice guideline. 1998, 1-12.Google Scholar
- Brigham and Womens Hospital: Osteoporosis: Guide to prevention, diagnosis and treatment. October 2001, [http://is.partners.org/handbook/quarkfiles/osteo/osteo_print.asp]
- ACOG educational bulletin – Osteoporosis. Int J Gynecology and Obstetrics. 1998, 62: 193-201.Google Scholar
- Baran RW, Kiel DP, Patterson H, Briesacher B, Erwin WG: Diagnosis and treatment of osteoporosis in long-term care facilities. Consult Pharm. 1998, 6: 685-699.Google Scholar
- Brown JP, Fortier M, Khan A, Rowe T: An Evidence based review of the management of osteoporosis. J SOGC. 1-7. April 2001Google Scholar
- Clinical Resource Efficiency Support Team: Guidance on the prevention and treatment of osteoporosis. Belfast. 2001, 1-35.Google Scholar
- Kin Yuen Chui, Kendler D, Khan A, Brown J, Fortier M: Osteoporosis. J Obstet Gynaecol Can. 2001, 23: 978-9.Google Scholar
- Klibanski A, Adams-Campbell L, Bassford T: Osteoporosis prevention, diagnosis, and therapy. NIH Consensus Conference. 2000, 1-45.Google Scholar
- London department of health. Osteoporosis. November 2001, [http://www.doh.gov.uk/pub/docs/doh/osteo.pdf]
- Osteoporosis; diagnostics and pharmacotherapy. Finland: Evidence-based guidelines. November 2001, [http://www.duodecem.fi/lcdnet/ENG/pdg/p20066.htm]
- Meunier PJ, Delmas PD, Eastell R, McClung MR, Papapoulos S, Rizzoli R: Diagnosis and management of osteoporosis in postmenopausal women: clinical guidelines. Clinical Therapeutics. 1999, 21: 1025-1044. 10.1016/S0149-2918(99)80022-8.View ArticlePubMedGoogle Scholar
- Ontario Program for Optimal Therapeutics: Ontario guidelines for the prevention and treatment of osteoporosis. 1-49. November 2001, [http://www.opot.org/guidelines/osteoporosis.pdf]
- Siebzehner MI: Consensus statement on prevention and treatment of osteoporosis. IMAJ. 2000, 2: 397-401.PubMedGoogle Scholar
- Ullom-Minnich P: Prevention of osteoporosis and fractures. American Family Physician. 1999, 60: 194-202.PubMedGoogle Scholar
- Osteoporosis clinical guideline. SAMJ. 2002, 90 (9): 905-944.Google Scholar
- Compston J, Papapoulos SE, Blanchard I, on behalf of working party for European member states: Report on osteoporosis in the European community: current status and recommendations for the future. Osteoporos Int. 1998, 8 (6): 531-534. 10.1007/s001980050094.View ArticlePubMedGoogle Scholar
- Genant HK, Cooper C, Poor G, Reid I, Ehrlich G, Kanis J: Interim report and recommendation of the World Health Organization Task Force for Osteoporosis. Osteoporos Int. 1999, 10 (4): 259-64. 10.1007/s001980050224.View ArticlePubMedGoogle Scholar
- Miller P: Management of postmenopausal osteoporosis for primary care. Menopause. 1998, 5: 123-131.View ArticlePubMedGoogle Scholar
- Health Council of Netherlands: Prevention of osteoporotic fractures publication no 1998/05. 1998Google Scholar
- Osteoporosis practice guidelines. Lebanese Medical Journal. 2002Google Scholar
- Shaneyfelt T, Mayo-Smith M, Rothwangl J: Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature. JAMA. 2002, 281 (20): 1900-5.View ArticleGoogle Scholar
- Choudry NK, Stelfox HT, Detsky AS: Relationships between authors of clinical practice guidelines and the pharmaceutical industry. JAMA. 2002, 287 (5): 612-617.View ArticleGoogle Scholar
- Pilote L, IB Tager: Outcomes research in the development and evaluation of practice guidelines. BMC Health Serv Res. 2002, 2: 7-10.1186/1472-6963-2-7.View ArticlePubMedPubMed CentralGoogle Scholar
- Bero LA, Grilli R, Grimshaw JM, Harvey E, Oxman AD, Thomson MA: Closing the gap between reserach and practice: an overview of systematic reviews of interventions to promote the implementation of research findings. BMJ. 1998, 317: 465-468.View ArticlePubMedPubMed CentralGoogle Scholar
- Grilli R, Magrini N, Penna A, Mura G, Liberati A: Practice guidelines developed by specialty societies: the need for a critical appraisal. Lancet. 2000, 355: 103-6. 10.1016/S0140-6736(99)02171-6.View ArticlePubMedGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2474/3/20/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article: verbatim copying and redistribution of this article are permitted in all media for any purpose, provided this notice is preserved along with the article's original URL.