Psychometric properties of a new treatment expectation scale in rheumatoid arthritis: an application of item response theory

Abstract

Background

Patient-generated health outcome measures are important in the assessment of long-term treatment goals for Rheumatoid Arthritis (RA), but few psychometrically sound measures are available. The MAPLe-RA (Measuring Actual Patient-Led expectations in RA) is a new questionnaire and its psychometric properties are not investigated. This study aims to examine these properties for each of the items using Item Response Theory (IRT) .

Methods

Participants were included if they completed the scale (MAPLe-RA). A one parameter (Rasch) model and a two parameter logistic (2PL) model were applied to these data using M-plus software.

Results

One hundred thirty-eight patients with RA were included in the analysis. MAPLe-RA scale comprised of 21 items, the mean score was 71 (20.28) ranging from 0 to 105. Most items operated in the high expectations part of the items characteristics curves (ICC). Item discrimination varied widely, items with the highest discrimination capacity from the three domains were: pain (physical domain); control of my RA (self-management) and maintaining social role (psycho-social domain); feeling better overall and involvement in treatment decision making (impact of new treatment domain).

Conclusion

RA patients’ expectations of treatment are higher in the physical and psycho-social domains and less so in the impact of new treatment domain.

Background

Patient- generated health outcome measures play an important role in the assessment of long-term treatment goals, for people experiencing Rheumatoid arthritis (RA) and are therefore well positioned to be utilised in assessing new treatments. In RA, there is no gold standard measure for the assessment of patient’s expectations at time of diagnosis or before commencing treatment. To fill this gap we developed a new patient-generated expectancy measure, called Measuring Actual Patient-Led expectations in RA (MAPLe-RA) scale [1]. Item response theory (IRT) is an approach that emphasizes the influence of the individual’s qualities as well as the items qualities, in a test, or in a questionnaire. The method was originated in education where individual qualities may reflect abilities, was then extended to other applications, with well-known examples in medicine and psychology [2–4]. In this study the underlying construct is patients’ expectations, and the method was used to understand the psychometric properties of the individual items [5, 6]. Although IRT method has been applied in several long-term conditions to assess the properties of outcome measures and questionnaires [7], it is rarely used in patient-generated measures in RA [8, 9]. For the development of new scales, traditionally, a factor extraction method based on Eigenvalues is used [10, 11] to explore the number of domains, and the strength of association of items within domains. IRT in addition provides important details, on psychometric properties of each item, including, the difficulty and the discrimination of these properties. MAPLe-RA is a new questionnaire and its psychometric properties have not yet been investigated. This study aims to examine these properties for each of the items using IRT.

Methods

The development stages of MAPLe-RA were published elsewhere [1]. In brief, stage one of the study: three repeated focus groups and two expert panels with RA patients were conducted by a patient researcher. Stage two: a feasibility study of the draft scale with 22 consecutive outpatient attendees over 1 week was conducted and stage three was the psychometric analysis, and that the results are presented here. MAPLe-RA scale, comprised of 21 items, and the response options were given in a Likert scale from 5 to 0. High scores refer to better treatment expectations. The scale is intended to measure expectations of treatment in three domains: physical, psychosocial and impact of treatment. Table 1 shows how the MAPLe-RA questionnaire is scored. MAPLe-RA was approved by the National Research Ethics Committee London-Central (REC reference number 10/HO718/82). All participants provided informed consent.

Table 1 MAPLe-RA questionnaire

Data analysis

The demographic information of participants are described in Table 2, using proportions, means and standard deviations (SD) as appropriate. The item responses for all the 21 items were skewed towards ‘better’ and ‘much better’. For the factor analysis we used the original scales of the items. However, for the purpose of the IRT model, we dichotomised the items by collapsing the (i) ‘better’ and ‘much better’ responses together and coded as 1; and (ii) ‘worst’, ‘much worst’, ‘same’ and ‘non-applicable’ responses coded as 0. We fitted a one parameter (Rasch) model which, assumes that the items are equally discriminating but with varying difficulty, and a two parameter logistic (2PL) model that assumes the items have a varying ability to discriminate among patients with different levels of the underlying construct [13, 14]. Uni-dimensionality was assumed as a priori and was further assessed using maximum likelihood method as well as principal-component factor methods. We present the results obtained from the 2PL model, as these provide more desired information, including items’ difficulty and discrimination [15, 16]. The model fits the data well as assessed by the Akaike information criterion (AIC) and Bayesian information criterion (BIC). We employed the Item-characteristic Curve (ICC) to evaluate the profile of each item within the scale and to assess the relationship between the predicted patients’ response to an individual item and the underlying construct (expectations). For all the analyses we used M-Plus statistical software.

Table 2 Demographic information of participants in the study

The two-parameter logistic model suits binary responses and may be described as:

$$ p\left({x}_j=1\left|\theta \right.\right)=\frac{1}{1+ \exp \left\{-{a}_j\left(\theta -{\beta}_j\right)\right\}} $$

(1)

Where x _j is the observed response to item j, α _j is the slope parameter, β _j is the difficulty (location) of item j, and θ is the underlying construct being measured (expectations).

Results

Study participants

A total of 160 outpatient attendees were invited to take part in phase 3 of the study, 138 (86 %) consented and completed the MAPLe-RA questionnaire. The mean age was 54 (SD = 14.30) years; 101 (74 %) were women, 73 (53 %) were of white ethnicity and 48 (38 %) reported being registered disabled (Table 2).

MAPLe-RA scale properties

In stage one and two of the scale development, patients identified 21 dimensions of new treatment expectations, grouped into (i) physical (ii) psycho-social and (iii) expectations relating to impact of treatment. This resulted in a draft questionnaire assessed in the feasibility study and subsequent stage three analysis.

The overall mean score of MAPLe-RA 21 items, was 71 (SD: 20.28; range 0 to 105). The means for the 3 domains, separately were: physical (7 items), mean 24.40 (SD: 7.21), psycho-social (6 items), mean 17.51 (SD: 8.17) and impact of new treatment domain (8 items) mean 30.76 (SD: 7.02).

Exploratory factor analysis identified that all items had strong positive associations with the first factor, weak associations in most items with the second factor, and negative associations with items 13–21 (Additional file 1: Table S1). The Eigen value for the first factor was 6.25, proportion of variance explained was 77 %, supporting the uni-dimensionality of the items.

Most items had high rates of “yes” or positive responses, in IRT context, these have low difficulty parameters, most patients would pick, and these seem to describe the majority of patients’ expectations. Item discrimination on the other hand reflects the strength of the association of an item with the underlying construct, items with high discrimination are better at differentiating respondents at the location point; small changes in the underlying construct (expectation) leads to large changes in the probability of endorsing the item (response = yes), and vice versa for items with low discrimination. These responses varied widely, and the most powerful two items from the three domains were: “swelling of the joints” and “pain” in the physical domains, discriminations: 1.38 (0.58) and 1.74 (0.82).

In the psycho-social domains, “to maintain social role” and “emotional wellbeing” were the two items with highest discrimination: 1.87 (0.63) and 1.77 (0.51), respectively. The two items with the highest discrimination in impact of new treatment domain were “feeling better overall” and “involvement in treatment decision making”, 1.47 (0.38) and 1.41 (0.52) respectively, and other items had lower discrimination. Details for all items are presented in (Table 3). Within the physical domains, two items had a difficulty that was not different from zero, namely “visible signs of RA” and “joint damage”, and the two items also had very low discrimination. The model fit was high in both the BIC (2807.48) and AIC (2743.08), which indicates a good fit of the two-parameter logistic model.

Table 3 Item characteristics (difficulties and discrimination) for MAPLe-RA scale

Figure 1 shows the Item Characteristic Curves (ICC) that represents the respondents’ expectations (underlying construct) in relation to the probability of endorsing an item and is presented graphically for two items with the highest discrimination from each domain.

Fig. 1

Two Parameter Logistic Model (2PLM) item characteristics ccurves (ICC), for two items from each domain of the MAPLe-RA scale

Discussion

MAPLe-RA is a new questionnaire that was not yet validated and its items were not examined. In this study, we used factor analysis to assess the uni-dimentionality and IRT to describe the properties of the items. The study has shown that RA patients have high expectations from their treatment. These are particularly high in the physical and psycho-social domains rather than in the impact of new treatment domain. For the latter, most items were unlikely to be endorsed by patients with less than average expectations in this RA study cohort.

Strength and limitations of the study

All domains have shown items with strong association with the underlying construct (expectation), two to three items from each domain, may be considered as good candidates that differentiate between patients’ responses. Several items however, appeared to be redundant (e.g. visible signs of RA; not to have to change medication), as they did not show strong association with the underlying construct.

The IRT method is superior to the traditionally factor extraction methods based on Eigenvalues [10, 11]. It is also a suitable way to employ when an instrument includes response categories that have several levels. In this study, the method determines whether the categories perform as they were envisioned and/or whether to collapse the responses into fewer categories [17]. The advantage of using IRT is that of an underlying construct, that gives items different weights, depending on the response pattern and the frequency of response to each item, and values instead of sum scores [12, 18]. This technique has been successfully applied in the development or the evaluation of new measures in patient-reported outcomes [19, 20]. To our knowledge the IRT method has not been applied in many Rheumatology related scale studies [21].

While the results of the new instrument appeared to have a very good reliability, it is important to interpret the findings with caution. This analysis was an exploratory phase of the scale development stages. The sample size was under powered for IRT 2PL model and the population studied was homogeneous from one RA clinic only. Although, there were some redundant items, we chose to keep these in the analysis to avoid making inappropriate decisions and or conclusions at this early stage.

Other studies have found similar results in that new measure of patient’ expectations in general need validation in larger multi-centre studies [22]. We acknowledge that further analysis is necessary, thus MAPLe-RA is currently included in a national longitudinal observational study of patients with early Rheumatoid Arthritis with a diversity of socio-demographic characteristics and a long term follow up of 18 months, to be completed in 2015. This large multi-centre study will allow us to conduct a confirmatory analysis of the new measure as well as to assess if patients expectations change over time.

Conclusions

This study extends the evidence on the value of IRT models in the assessment of health outcomes and patient-generated measures. The result highlights that RA patients’ treatment expectations are higher in the physical and psycho-social domains and less so in the impact of new treatment domain. RA patients expect high degree of involvement in their care from health care providers, and that they rate highly, controlling their pain and emotional well-being.