A comparison of outcome measures used to report clubfoot treatment with the Ponseti method: results from a cohort in Harare, Zimbabwe

Study design and population

This study was conducted and reported according to established STARD (Standards for Reporting of Diagnostic Accuracy Studies) guidelines [8] (Additional file 1). A cohort study of 218 children with idiopathic clubfoot was conducted in 2016. The children were managed with manipulation and casting at Parirenyatwa Hospital, Harare and the results are published elsewhere [9]. All children with a diagnosis of unilateral or bilateral idiopathic clubfoot who started treatment with the Ponseti method at the study hospital between 22nd March 2011 and 23rd April 2013 (25 months) were included in the cohort. The only exclusion criterion was foot conditions other than idiopathic clubfoot, for example clubfoot associated with neural-tube defects such as spina-bifida.

Sampling technique

The phone numbers of all carers of the cohort children were extracted from the clinic records in January 2017 and contact with them was attempted at least three times. Caregivers and their children were invited to attend the study. The children were between 3.5 and 5 years from initial casting.

Ethics, consent and permissions

Ethical approval for this study was granted by the Medical Research Council of Zimbabwe (MRCZ) and the London School of Hygiene & Tropical Medicine (LSHTM) (ref:11132 /RR/4725). All children and their caregivers were read an information sheet about the study and given an opportunity to ask questions. If they agreed to participate, written consent was taken from the caregiver who remained present throughout the assessment as per national requirements. Transport costs were reimbursed and referral services available in Harare were mapped pre-emptively to ensure appropriate onward referral for any children that required further intervention.

Data collection

Two physiotherapists who are experienced in co-ordinating national clubfoot programmes reviewed the assessment tools over three days for contextual relevance. The questionnaires were available in English and Shona and were cognitively tested. We used five outcome methods, three that give a score, and two that give a binary (success/failure) outcome. The Roye score [10] is a self-reported measurement that is used in high income settings. The Bangla clubfoot assessment tool [11] and the Assessing Clubfoot Treatment (ACT) score [12] combine physical assessment and parent reported outcome measures, and have been developed for low resource settings. The Bangla score includes a functional assessment. The two binary outcomes were assessment of a plantigrade foot [5] and the relapse pattern [13]. The study protocol was pilot tested for suitability in July 2016. Children were examined independently in January 2017 by the two physiotherapists and a decision was made if referral for further intervention (re-casting or surgical review) was required. Clinical examination composed observation, physical assessment and functional review; it included assessment of passive and active range of motion (plantiflexion, dorsiflexion, eversion, inversion of the foot, and knee extension), muscle strength tests of the calf and evertors of the foot, heel raises, squatting ability and gait analysis (walking and running).

Data management and analysis strategy

The data were entered into a Microsoft Excel 2000 (Microsoft Inc., Redmond, Washington) software package. Data were analysed using Stata 14.1 (Stata-Corp 4905, Lakeway Drive College Station, Texas 77, 845, USA). Statistical significance was set at the 95% confidence level. The inter-observer variation for the measurement of the physical assessment tools was assessed i.e. Intra-class correlation coefficient (ICC) ≥0.75 [10]. Outcomes of children who had completed casting and ≥ two years of bracing were compared to all of the children who were followed up, and to those who had only completed casting. A two-tailed paired t-test was used to assess the mean difference between the outcome measures of Roye, Bangla and ACT scores. Fisher’s exact test of independence was used to assess the difference in proportion of children with an outcome of relapse and plantigrade foot. The five measures were compared against the standard of whether referral for further intervention was required (for re-casting or surgical review) as defined by a consensus agreement of two expert physiotherapists with experience of managing clubfoot in countries in Africa. Sensitivity, specificity, positive and negative predictive values were calculated for the five measures and compared to full clinical assessment (gold standard). The threshold for diagnostic accuracy was based on previous studies and was defined prior to the study. It was set at 70% for the three scores with continuous scales [14] and positive/negative for the binary outcomes [7].

Comparison to previous studies

There are limited studies that compare measurement tools in the same patient against which to compare our findings. However, success of treatment in this cohort is similar to other studies in sub-Saharan Africa (between 63 and 98% of cases) [9]. Non-adherence and surgical intervention, often defined as failure, are reported to vary from 7 to 61% and 3–39.4% [15] respectively. Ponseti and Laaveg [16] describe a scoring system that rates functional results as satisfactory in 88.5% of feet. Further studies describe success using the Ponseti and Laaveg system as 89.3% [17]. The criteria includes the need for a goniometer and the tool was therefore not included in evaluation of this cohort.

Use of outcome measures

The ease of use and rate of incorrect classification in the tools used to measure success need to be considered when selecting an outcome measure. Single item scales for assessment of individual children require no further calculation and may be easier to use in clinics (such as plantigrade foot or evidence of relapse), however their simplicity may not allow a full assessment of success. Multi-scale items prove difficult to transform into useful statistics without technology and are unlikely to be routinely used in clinics. This study found no clear agreement between the different outcome measurements in use.

All of the assessments used in this study have limitations. The Roye score has been validated in high income settings and parents in our study reported difficulty in answering the question of “How often does your child have problems finding shoes that he or she likes?” as it was understood to be related to the availability of a variety of shoes. The Bangla score took the longest time to transform with statistical analysis. Acceptability and feasibility of the ACT score is needed to be studied in future research. The ACT score is likely easy to teach, however this is unknown as the examiners were physiotherapists; the time taken for other cadres of health workers to use the ACT tool is also unknown. With regard to the relapse score, Bhaskar et al. (2013) considered ankle dorsiflexion < 15 degrees with knee in extension as grade IA relapse. This may be a reason for the restriction in defining good outcome as an evaluation of 85 normal feet in children found that the mean ankle dorsiflexion was 12.8 degrees with knees in extension [18]. Greater than 15 degrees may therefore be difficult to achieve.

Relationship between the outcome measures and clinical assessment

The Bangla and ACT tool were most helpful in predicting the need for referral for further intervention (specialist opinion or for further manipulation and casting). The five referrals that were missed with the ACT score were children who required review of a mobile curvature of the lateral border of the foot or supination in swing phase, neither of which are assessed with the score. Despite this, the ACT tool demonstrates the best diagnostic accuracy for the need for referral for further intervention.

Strengths and limitations of study

This study reports on five measurements of success in a cohort at 3.5–5 years from initial treatment. Repeat phone calls facilitated assessments when caregivers were initially unavailable. Two independent raters reduced the likelihood of reporting bias and all outcome measures were verified by the reference standard. The threshold for diagnostic accuracy was based on previous studies and was defined prior to the study. There were also study limitations. No distinction between a clubfoot that may not have been fully corrected and a relapsed clubfoot was made, and all cases with elements of the deformity were classified with the relapse score, which may be a source of potential bias that underestimates the accuracy of the relapse score. The tools were chosen based on ease of use in low resource high volume clinics and were not all initially developed to identify need for referral for further intervention.

Implications for practice

Task shifting and task sharing between orthopaedic and non-specialised health workers in some clinics means that outcome measures are even more important as teams expand. As older children are being treated with the principles of the Ponseti method [19], expert guidance on assessment and measurement in these cases is needed. The Roye score is overly optimistic of good outcomes, the Bangla score is restrictive in identifying good outcome, and the ACT score most closely aligns to clinical examination. However, the Bangla, relapse and ACT scores closely agree on false negatives and have the least chance of missing recurrence; the Bangla score and the relapse score over-estimate referral needs compared to the ACT score.