Reliability, construct and discriminative validity of clinical testing in subjects with and without chronic neck pain

Design

The study was a reproducibility study of six clinical tests with two examiners, and a test-retest study of two physical performance tests, conducted by a third examiner. The study followed a strict three-phase reproducibility protocol, including a “training”, an “overall agreement”, and a “study” phase, as recommended for nominal and ordinal data [29]. Since the clinical tests included primarily ratio interval data, the protocol was adjusted to a two-phase study by excluding the overall agreement phase [30]. This standardized protocol included a case as well as a control group, to confirm that both groups could be tested reliably. Tests were described in detail by examiner C (Experienced Physiotherapist (PT) and Manual Therapist) during phase one. Afterwards, examiner A and B (final year bachelor PT students), and examiner C tested 10 subjects with and without neck pain in an open study, to become familiar with and to standardise and equalise the test procedure and interpretation of results. During phase two, the examiners applied all tests on included subjects. Examiners were blinded to the status of the subjects, except for examiner C, since this examiner was involved in the recruitment of cases and controls. Although examiner C was aware of the subject’s status, this examiner only performed the PPT and SWAY tests, which are two fairly objective tests, thus limiting potential bias. All examiners were mutually blinded to the results of other examiners.

Study sample

Patients were recruited at physiotherapy clinics and controls via local advertisements. Inclusion criteria for neck pain patients: adults (>18 years), neck pain >6 months, reduced neck function (Neck Disability Index; NDI, minimum 10/50), pain primarily in the neck, and ability to read and understand Danish.

Inclusion criteria for controls: no present pain in neck, shoulder, elbow or hand, no neck pain lasting more than one week during the last year, matched on gender and age (+/-3 years to one of the patients), and ability to read and understand Danish.

Exclusion criteria for both groups: neuropathies/radiculopathies (defined by positive Spurling, cervical traction and plexus brachialis tests) [31], neurological deficits, being in an unstable social and/or working situation, pregnancy, known fractures, and depression according to the Beck Depression Inventory (BDI) (score >29) [32].Overall, 31 patients with chronic neck pain were recruited, and 21 were included in the final sample (Figure 1). All 21 controls were matched on gender and age (+/-3 years). Subjects received oral and written information about the project and gave their written informed consent to participate. The Regional Scientific Ethical Committee of Southern Denmark approved the study (S-20100069). The study conformed to The Declaration of Helsinki 2008.

Questionnaires and self-reported outcomes

Subjects completed self-reported questionnaires prior to enrolment, and demographics (age, gender, height and weight, type of accident, medication, symptom development over the last two months, employment and educational status) were registered. Questionnaires included the NDI (range: 0 to 50) [33], Medical Outcomes Study Short Form 36 (SF36)(range: 0 to100), with emphasis on the Physical Component Score (SF36-PCS) [34, 35], Numeric Rating Scale (NRS) for present pain (P.P.) and average pain during the last week (Week) [36], Modified Global Perceived Effectiveness (GPE) to evaluate stability of the condition with the question: “Compared to your first visit, how would you describe your neck today?” (-5 = vastly worse, 0 = unchanged, and 5 = completely recovered) [37]. Only subjects answering 0, representing unchanged, were included for intra- examiner and test-retest reliability.

Clinical tests

Subjects were not allowed to practice the tests, except for the tests of neuromuscular control; CCFT and DCE. For the CCFT, subjects performed three practice trials and for DCE one practice trial for a maximum of 30s. Test instruction followed an instruction manual, however, the amount of instruction and feedback varied among subjects, depending on the subject’s ability to understand the procedure.

Cranio Cervical Flexion Test (CCFT) was performed, using a Pressure Biofeedback Unit (Stabilizer; Chattanooga Group, South Pacific), as described by Jull et al. [38]. The subject was asked to perform cranio-cervical flexion in five incremental stages guided by the pressure sensor. The activation score has six scoring options; 20,22,24,26,28 and 30 mmHg.

Deep Cervical Extensor (DCE) test was performed in prone with their head over the edge of the bed. A laser was fixed to the top of the subject’s head and was projected to a target. The duration of time the laser beam was kept within the centre of the target was measured in seconds (sec.).

Range of movement (ROM) was examined using a bubble inclinometer (Baseline Bubble Inclinometer, Fabrication Enterprises Inc, USA) for flexion/extension and lateral flexion, and custom-made equipment for neck rotation (Figure 2). All scores were registered to the nearest degrees, except for rotation which was registered to the nearest 5 degrees.

Joint Position Error (JPE) JPE was examined following return from active rotation, flexion, and extension movements by measuring the reposition error. A laser beam was positioned 1 meter behind the subject, and the laser was projected to a cm ruler attached to a cap which the subjects wore. Data was registered in millimetres (mm).

Gaze stability (GS) was registered during rotation, flexion and extension movements as positive/negative based on the patients report of symptoms such as nausea, dizziness, disturbed vision.

Smooth Pursuit Neck Torsion Test (SPNTT) was tested in both a neutral head position and with the trunk rotated 45 degrees and was registered as positive/negative based on the patients report of symptoms such as nausea, dizziness, disturbed vision.

Postural control was measured during one-legged stance (eyes open and eyes closed) using a Wii balance board (Nintendo, Kyoto, Japan) and quantified with the SwayWithWii software program. Data was registered in millimetres (mm).

Procedure

Questionnaires were sent to participants before their first appointment. At the first visit, participants reported NRS (P.P./Week). Examiner C then screened for in- and exclusion criteria, after which SWAY and PPT tests were conducted. Following a rest period of ~2 min., tests for intra- and inter-reliability were performed. Testing order of examiner A and B was randomized for the first test round, and test order was always CCFT, ROM, JPE, GS, SPNTT and DCE. After a rest period of ~2 min. the other examiner performed the same tests in the same order on the same subjects (Figure 3). Duration between the two test occasions was 1–7 days. At the second visit GPE was added, and test order of examiner A and B was reversed. Cases and controls followed the same procedure throughout the testing session.

Sample size

Sample size was calculated based on the JPE test [23], since larger standard deviations were expected for this test. Sample size was estimated based on the 95% confidence interval according to the recommendation from Hopkins [39]. In a two one-sided test analysis for additive equivalence of paired means with bounds -5 and +5 for the mean difference and a significance level of 0.05, assuming a mean difference of 0, a common standard deviation of 16 and correlation 0.9, a sample size of 19 pairs, was required to obtain a power of at least 0.8.

Statistical analysis

Data analysis was performed blinded. Summary statistics are based on whole group-mean scores from examiner A and examiner B at the first test occasion. For the test-retest study, whole group-means are based on the first test examination. Mean values from the three repetitions for DCE, JPE, PPT and SWAY was used for analysis of reproducibility.

For calculation of intra- and inter-examiner reproducibility for ratio interval data, ICC (2,1) and Bland and Altman’s with 95% limits of agreement (LOA) were used. Interpretation of ICC was 1.00-0.76 (good to excellent), 0.75-0.41 (fair to good), and 0.40-0.00 (poor) [40]. The minimal detectable change (MDC) that is not due to error was calculated for all parametric tests, as 1.96 * √2 * SEM [41]. The standard error of measurement (SEM) was calculated as SEM = standard deviation of the mean difference between tester A and B divided by √2 [42].

For ordinal data, Cohen’s κ statistics with 95% confidence interval were calculated, with the interpretation 1.00-0.81 (almost perfect), 0.61-0.80 (substantial), 0.41-0.60 (moderate), 0.21-0.40 (fair), 0.00-0.20 (slight) and below 0.00 (poor) [43]. Furthermore, observed agreement, prevalence and expected agreement were calculated. Prevalence of the index condition was calculated as $\frac{(a+\left(b+c\right)/2}{n}$. The prevalence-adjusted-bias-adjusted kappa (PABAK) was calculated for the SPNTT, in which the values in cells a and d from the contingency table are replaced with the mean values from these cells, and values from cells b and c are replaced with the mean values from these cells [44]. Whole group results will be displayed if there is no systematic bias for cases or controls.

Construct validity between each of the clinical tests and NDI, NRS and SF36 were calculated using Spearman’s correlation coefficient (rho), due to non-normal distributions. Correlations were interpreted as for the ICC: 1.00-0.76 (strong), 0.75-0.41 (moderate), and 0.40-0.00 (weak). Positive correlation coefficients indicate positive associations, negative indicate negative associations. Since interpretation from the spearman is known to be difficult, statistical significance testing was included. Between-groups discriminative validity of the clinical tests was evaluated by a t-test and a Mann–Whitney U-test in normally and non-normally distributed data, respectively. Calculations of construct and discriminative validity were based on mean scores from the first test occasion. STATA statistical software was used for all analyses (Stata Corp., 2000, College Station, TX).

Cranio-cervical flexion test

Bland Altman plots revealed no systematic differences depending on mean scores. The intra- and inter-reliability for the CCFT was between “fair to good” and “good to excellent” (ICC: 0.63 to 0.86), in line with previous studies [45, 46]. Studies of reliability on the CCFT in asymptomatic subjects have reported slightly higher ICC values, ranging from 0.81 to 0.98 [18, 20, 21]. Studies including both symptomatic and asymptomatic populations usually have higher within subject- and day-to-day variation. The relatively large LOA and MDC, (intra-examiner: 2.9 and 3.9 mmHg; inter-examiner: 3.1 and 4.7 mmHg) (Table 4), is a future challenge for interventions. An MDC of two target levels (4.0 mmHg) is considered to be insufficient in a test with only five target levels. However, the significant correlation between CCFT and NDI, SF36-PCS and NRS, makes the test clinically relevant. The test needs improved psychometric properties for clinical use if implemented as in the present study. However, it should be noted that scoring of the CCFT may also include a measurement of endurance, that is, the number of 10 seconds holds that the subject can do at their achieved pressure level to generate a performance index. For example if a patient can achieve the second level of the test (24 mmHg) and perform six, 10 seconds holds with the correct action of cranio-cervical flexion, their performance index is 4 × 6 = 24. Highest activation score is 10 and highest performance index 100. Different results may have been achieved with this scoring approach.

Deep cervical extensors

Bland Altman plots revealed no systematic differences depending on mean scores. ICC for the intra- and inter-examiner reliability measures ranged from 0.75 to 0.90 (“good to excellent”). This is the first study to examine the reliability of this test. Although the results are promising, the large LOA and MDC, ranging from 37 to 59 seconds, indicate higher variation than expected from the ICC. The high ICC is probably due to large between subject variability, thus disguising large test-retest differences [47]. Although testing in a cranio-cervical neutral position, as recommended for the deep neck extensors [48, 49], the validity of this test has not been confirmed. Lower scores on the test correlated with higher levels of pain (NRS) and disability (NDI), but not with SF36. The present DCE test needs improved psychometric properties for clinical use.

Range of movement

Bland Altman plots revealed no systematic differences depending on mean scores. ICC measures for the intra- and inter-examiner reliability ranged from 0.80 to 0.93 (“good to excellent”), in line with previous studies using an inclinometer [50–52]. Since, “good to excellent” reproducibility was obtained using the custom-made rotation device; future clinical use of this device seems promising. LOA and MDC were large for neck flexion and extension (13 to 21°), reflecting some variation. Significant correlation was found between ROM variables and NDI, SF36-PCS and NRS. This test has satisfactory psychometric properties and can be recommended for clinical use.

Joint position error

Bland Altman plots revealed no systematic differences depending on mean scores. ICC for the intra- and inter-examiner reliability measures ranged from 0.02 (“poor”) to 0.52 (“fair to good”). Previous studies have reported varying results, however, most studies report ICC above 0.75 [23, 46, 53–55]. In this study the laser light was positioned behind the subject whilst in most previous studies the laser light was attached to the head of the subject, which may explain such differences. Other explanations for differences in results could be the equipment used, or the current number of three repetitions performed, since six repetitions are recommended for stable estimates and higher reliability [53]. This was further supported by the fact that other studies using three repetitions have reported ICC similar/lower than the present study [24, 56]. JPE revealed large LOA and MDC ranging from approximately 7 to 10 mm, and did not reach significant correlations with any of the self-reported outcome measures. As described, this current test cannot be recommended for further use.

Gaze stability

GS κ-values ranged from 0.66 (“substantial”) to 0.92 (“almost perfect”) for intra- and inter-reliability. This is the first study to examine reproducibility of the GS test in a clinical setting without sophisticated equipment. However, the present results are in line with previous studies using advanced equipment, such as wireless 3D sensors with ICC ranging from 0.40 to 0.89 [12, 26]. GS discriminated significantly between cases and controls, in line with other studies [12, 13]. Since the GS test showed significant correlations with NDI, SF36-PCS and NRS, this test has satisfactory psychometric properties and can be recommended for clinical use.

Smooth pursuit neck torsion test

κ ranged from 0.46 (“moderate”) to 0.74 (“substantial”) for intra- and inter-reliability of the SPNTT. No other studies have evaluated the reproducibility of the SPNTT in a clinical setting. The κ-values obtained for SPNTT were lower than expected, probably due to the low prevalence of the condition, known to affect the κ-value [29]. Therefore, PABAK was used to adjust for this, and κ ranging from 0.70 to 0.78 and 0.57 to 0.72 was obtained for intra- and inter-reliability, respectively. The SPNTT test was able to discriminate significantly between cases and controls, as also shown in earlier studies of whiplash patients [57–60], while other studies reported no differences [25, 61, 62]. A positive test correlated with higher NDI and NRS scores, and lower SF36-PCS. The SPNTT has satisfactory psychometric properties and can be recommended for clinical use.

SWAY

Bland Altman plots revealed no systematic differences depending on mean scores. Highest ICC values were obtained for the Romberg eyes closed condition with 0.79 (“good to excellent”) for 95% Confidence Ellipse Area (CEA), 0.60 (“fair to good”) for Anterior/Posterior (A/P), 0.69 (“fair to good”) for Medial/Lateral (M/L) and 0.77 (“good to excellent”) for Centre of Pressure path length (COP). All COP variables were above 0.75 (“good to excellent”). These results are in line with one previous study using the Wii balance board in healthy subjects reporting ICC above 0.75 [63]. The present study found no systematic bias, and significant correlations were found between NDI/SF36-PCS/NRS and area/range of displacement in the Romberg eyes closed condition and single-leg stance, but not for COP. Overall, the SWAY test has satisfactory psychometric properties and can be recommended for clinical use.

Pressure pain threshold

Bland Altman plots revealed no systematic differences depending on mean scores. ICC was “good to excellent” for all variables (Tibialis anterior: 0.86, C3-C4: 0.89, Infraspinatus: 0.83), in concordance with previous studies on patients with acute neck pain [28, 64]. MDC was 1.90 kgf (Tibialis Anterior), 0.90 kgf (C3-C4), and 1.65 kgf (Infraspinatus), also in accordance with an earlier study using hand-held algometry [28]. Since significant correlations were found between PPT and NDI/NRS (all sites) and SF36-PCS (C3-C4), this test has satisfactory psychometric properties and can be recommended for clinical use.

Considerations

A strength of the study is that it followed a standardized protocol, including a training phase, in which examiners were able to standardise and calibrate performance and interpretation of tests. The inclusion of a thorough training phase enabled inexperienced examiners to obtain satisfactory results. Further, by including a case as well as a control group, we demonstrated that both groups could be tested in a clinical manner reliably. Since ICC did not differ between groups, except for a minor tendency in the CCFT for neck pain subjects to have lower ICC, only the pooled data set was presented. Furthermore, primarily using quantifiable variables may have reduced variability which is usually introduced by more subjective estimates e.g. presence of co-contraction, breathing etc. It is unclear whether superior results would have been obtained if more experienced examiners were involved.

A weakness of the study may have been the duration of the test procedure. The entire test procedure had a length of approximately 1.5 hours, possibly imposing a fatigue effect. However, post-hoc analysis of data from the CCFT and DCE tests, comparing mean values for each of the tests in the order they were performed for each of the examiners, revealed that a significant fatigue effect was not present. Additionally, no significant learning effect was evident. The GPE was included in order to control for within-subject change between test occasions. Nevertheless, it cannot be entirely excluded that residual fatigue, not altering the subject’s perception, biased the results on the second test occasion.

Most patients were receiving treatment, and may have been familiar with the tests. Since one purpose of a reliability study is to examine all aspects of a clinical test, including information, instruction and position of subjects, this might have biased the data, possibly resulting in higher reliability estimates, since some of the patients may have been familiar with the tests. However, this may have decreased the possible difference between cases and controls, and thus resulting in lower estimates for discriminative validity.

Generally, interpretation of the data on discriminative validity must be performed with caution, since the sample size was not powered to estimate this. The general finding of the present study was that although significant differences were found for most variables, they were all within the MDC. From a clinical perspective, this naturally complicates the interpretation of tests, since “positive” findings may thus be attributed to measurement error. Sufficiently powered future studies are therefore needed, especially on discriminative and predictive validity, in addition to responsiveness, and establishment of relevant cut-off points for abnormality by investigating the normal variation in a healthy population.