Does the use of telephone reminders to increase survey response rates affect outcome estimates? An ancillary analysis of a prospective cohort study of patients with low back pain
BMC Musculoskeletal Disorders volume 22, Article number: 893 (2021)
Research is often undertaken using patient-reported outcomes from questionnaires. Achieving a high response rate demands expensive and time-consuming methods like telephone reminders. However, it is unknown whether telephone reminders change outcome estimates or only affect the response rate in research of populations with low back pain (LBP). The aim is to compare baseline characteristics and the change in outcome between patients responding before and after receiving a telephone reminder.
This is an ancillary analysis of data from a prospective cohort study employing questionnaires from 812 adults with LBP lasting more than 3 months. Patients not responding to the 52-week questionnaire were sent reminder emails after two and 3 weeks and delivered postal reminders after 4 weeks. Patients still not responding were contacted by telephone, with a maximum of two attempts. Patients were categorised into three groups: 1) patients responding before a telephone reminder was performed; 2) patients responding after the telephone reminder and 3) patients not responding at all. A positive outcome was defined as a 30% improvement on the Roland Morris Disability Questionnaire after 52 weeks.
A total of 695 patients (85.2%) responded. Of these, 643 patients were classified in Group 1 and 52 patients were classified in Group 2. One hundred seventeen were classified in Group 3. No differences in outcome or baseline characteristics was found. In Group 1, 41.3% had a positive outcome, and in Group 2 48.9% had a positive outcome (P = 0.297). In group 3, non-respondents were younger, more often unemployed, more often smokers, more often reported co-morbidity, and reported higher depression scores than respondents.
Using a telephone reminder had no consequence on outcome estimates nor were there any differences in baseline characteristics between patients who responded before or after the telephone reminder.
Low back pain (LBP) is a major global challenge, and the burden of back-related disability is expanding with the increasing ageing population, placing LBP as the leading worldwide cause of years lived with disability . LBP-related costs associated with work disability and attributed to health care is massive, with the societal costs for back pain being estimated to 1 to 2% of the gross national product [2,3,4]. The majority of these costs (80–90%) is caused by productivity loss due to work-disability . It has recently been emphasised that intensified research efforts and global initiatives are needed to address the burden of LBP as a public health problem . Research within public health is often undertaken using patient-reported outcomes from questionnaires to explore a potential association between a given exposure and an outcome. However, the strength of data derived from questionnaires depends on the level of response rates . Consequently, the response rate is considered a central indicator of data quality, with regard to internal validity. Furthermore, poor response rates reduce the statistical power of the study . Non-response may introduce bias if the responders are different from non-responders in terms of baseline characteristics . Consequently, achieving adequate response rates is assumed to reduce the risk of bias and increase internal validity .
Reducing questionnaire length, providing incentives or using reminders have been suggested for improving response rates [10,11,12,13]. However, achieving a high response rate by employing multiple reminders is often expensive and time-consuming [14, 15]. Previous studies have found that prevalence estimates and exposure-outcome relationships may not be influenced by an increase in response rate [16,17,18,19]. Telephone reminders might be a stronger method to increase the response rate. However, compared to postal or e-mail reminders, telephone reminders are more time-consuming for researchers . To our knowledge, the consequences of employing telephone reminders in survey research with patients with LBP has not previously been investigated. However, we hypothesised that patients who responded after telephone reminders had worse symptoms at baseline and were less likely to have a positive outcome. The aim is to compare baseline characteristics and the change in outcome between patients responding before and after receiving a telephone reminder.
This study is reported according to the STROBE guidelines for observational studies in epidemiology .
This is an ancillary analysis of data from a prospective cohort study with follow-up of 1 year, using questionnaire data retrieved through an electronic survey .
Participants and setting
The data derive from a consecutive series of adults with LBP, referred from general practice to the regional Spine Centre at Silkeborg Regional Hospital in Denmark. There was a follow-up time of 52 weeks . To be referred, patients had to report LBP of at least 3 months’ duration, been offered conservative treatment and have received an MRI prior to the consultation. To be eligible to the study patients had to fulfil the following criteria.
≥18 years of age
LBP was the primary cause of the referral to the Spine Centre
Known spinal fractures, inflammatory disease or infection
Missing information about the number of reminders used (n = 4)
Patients referred to the Spine Centre routinely receive a digital letter with a link to a standard, online questionnaire regarding their symptoms. For all patients fulfilling the inclusion criteria and not the exclusion criteria, an extra page became visible at the end of the questionnaire, informing them about the study. Consenting patients were then requested to reply to extra questions in addition to the standard questionnaire and to complete a follow-up questionnaire after 52 weeks. Both baseline and follow-up questionnaires were electronic questionnaires to be completed online, using a link in the invitation letter. Patients not responding to the 52-week questionnaire were sent reminder emails after two and 3 weeks and delivered postal reminders after 4 weeks. Patients who still did not respond were contacted by telephone, with a maximum of two attempts. When patients did not pick up, a message was left on the answering machine if this was possible. Text messaging was not used in this study. Patient participation and data collection is illustrated in Fig. 1. Patients were considered respondents if they returned the questionnaire, regardless of completeness.
Patients were categorised into two groups according to their response pattern. In addition, a third group was defined, consisting of patients who did not respond to any of the follow-up procedures (Group 3). This allows for comparison between baseline variables of all respondents (Group 1 + 2) and patients who did not respond despite the reminder procedures:
Group 1: ‘non-telephone reminder respondents’ consisted of patients who initially responded after an automatically generated e-mail was sent approximately 52 weeks after their first visit to the Spine Centre, and patients who responded to either e-mail reminders or postal reminder. .
Group 2: ‘telephone reminder respondents’ consisted of patients who responded to a telephone call (following non-response to the previous reminders).
Group 3: Patients who did not respond to any reminders were classified as ‘non-respondents’.
The Roland Morris Disability Questionnaire (RMDQ) was the outcome measure. The outcome was dichotomised by considering patients reporting a 30% or higher improvement after 52 weeks a success. The RMDQ is a widely used patient-reported outcome questionnaire designed to measure self-rated disability due to back or leg pain [22, 23]. The version used in this study consists of 23 items with ‘yes’ or ‘no’ response options and greater levels of disability are reflected by higher numbers on a 23-point scale . The RMDQ is validated in Danish and its use is recommended in clinical practice as well as in research . A 30% improvement in the RMDQ from baseline values has previously been proposed as a minimal important change [26, 27].
Pilot testing of questionnaires
The questionnaire has been routinely administered at the Spine Centre since January 2016. However, since additional questions were included the complete questionnaire was pilot tested for face validity on 10 patients referred to the Spine Centre prior to this study . This testing process involved the patients completing the questionnaire with one of the researchers (NR) present to observe and offer verbal feedback. This feedback led to minor question modifications. The RMDQ and other validated questionnaires remained unchanged .
The complete cohort (N = 816) from the original study was used . However, patients with missing data on the administered reminder system were excluded from this study.
Descriptive statistics are presented as total numbers (N) and proportions (%), as well as mean and standard deviation (SD) if the values were normally distributed. Otherwise, median and interquartile range (IQR), i.e. the 25th–75th percentile, is used.
The study aims to investigate whether changes in outcome estimates and baseline characteristics of patients with low back pain differ between patients responding to email reminders (group 1) and patients requiring telephone-reminders (group 2), and furthermore, to assess to what extent responders (group 1 + 2) differ from patients who do not respond at all (group 3). Fisher’s exact test was used to investigate differences in the proportion of patients who had achieved a clinically relevant improvement from baseline to follow-up (RMDQ improvement > 30%) between Group 1 and 2.
Fisher’s exact test was also used to investigate between-group differences in sex, level of education, employment status, sick leave, smoking status, history of lower back surgery, co-morbidity and pain status. A 2-sided t-test was conducted to investigate whether normally distributed continuous variables (age, RMDQ and LBPRS) differed between groups. A Wilcoxon rank-sum test was used to investigate whether non-parametric continuous variables (Major Depression Inventory score (MDI-10)) differed between Group 1 and 2.
Statistical analysis was performed using Stata/SE v.16.1 (StataCorp) and results were considered significant when P ≤ 0.05.
Between 1 April and 22 December 2017, 1789 patients with LBP were invited to participate in the primary study and 828 consented, of which 812 fulfilled the criteria for this study and were included. The average follow-up time was 52 weeks. A total of 643 patients returned the questionnaire without needing a telephone reminder and were classified in Group 1 as ‘non-telephone reminder respondents’. Fifty-two patients returned the questionnaire after they were reminded by telephone and were classified in Group 2 as ‘telephone reminder respondents’. The remaining 117 patients did not reply to the follow-up questionnaire at all and were classified in Group 3 as ‘non-respondents’. The telephone reminder increased the response rate from 79.2 to 82.5% (Fig. 1).
Mean age at baseline was 53 years (sd 13.7) with 453 (55.8%) being women, and the mean RMDQ score at baseline was 14.0 (sd 4.9), indicating a moderate level of disability. There was no statistically significant difference in baseline characteristics between Group 1 and 2 (Table 1).
In terms of the proportion of patients achieving a clinically relevant improvement in functional ability (i.e. minimum 30% improvement on the RMDQ over 1 year), 257 (41.3%) had a clinically relevant improvement in RMDQ in Group 1 and 24 (48.9%) had a clinically relevant improvement in RMDQ in Group 2, P = 0.297. In group 1, 21 observations were missing and in Group 2, three observations were missing.
In comparison with patients responding to the follow-up questionnaire (Group 1 + 2), non-respondents (Group 3) were significantly younger (P < 0.001), more often unemployed (P = 0.009), more often smokers (P = 0.005), more frequently reported co-morbidity (P = 0.038) and had higher depression scores (P < 0.001) (Table 1).
In the present study, employing a telephone reminder showed that patients who responded before or after the telephone reminder had no difference in baseline characteristics or outcome estimates. Patients who did not respond to either written or telephone reminders (Group 3) were significantly younger, more often unemployed, more often smokers, reported more co-morbidity and had higher depression scores than patients in the two response groups (Group 1 + 2).
The increase in response rate after employing a telephone reminder is in line with findings from a systematic review from 2009, where response rates increased when repeated postal reminders, e-mails or telephone calls were used . Our findings are also consistent with previous findings showing lower response rates for each round of a reminding [16, 28, 29]. Further, our finding are in line with a previous study by Breen et al. which concluded that even though reminder phone calls increased response rates, this was not justified by the resources required . If the purpose is to increase the population of patients with follow-up data, increasing the initial sample size and allowing a higher drop-out rate can be a feasible alternative to telephone reminders, as this is less time-consuming for researchers.Several studies have investigated the use of different types of reminders and incentives to complete surveys, with emails or text reminders being considered the most effective in terms of time consumption for researchers (as compared to telephone reminders). Still, a recent review of strategies to improve response rates concluded,that “research studies are needed to explore whether the different strategies used by researchers with the intent to improve response rates are acceptable to potential participants and to evaluate the potential synergistic effect of combinations of several strategies identified in this review.” .
Baseline characteristics and outcomes
Reminder procedures are justified by the assumption that patients who do not respond to initial requests have different baseline characteristics and show different outcomes compared to patients who do respond to initial requests. To avoid biased estimates and increase internal validity, these multiple reminder procedures therefore seem justified [6, 7]. However, in this study, patients who responded after a telephone reminder did not differ from patients who responded without the telephone reminder in their baseline characteristics. The telephone reminder thereby only resulted in an increased response rate, while patients differing in baseline characteristics continued to be non-respondents. In other words, applying telephone reminders did not increase internal validity and, thereby, did not reduce the risk of selection bias . A supplementary analysis was performed to investigate whether the initial respondents, i.e. those responding to the questionnaire without any reminders at all (N = 439), differed from patients responding following reminders. This was not the case, neither in terms of baseline characteristics nor changes in outcome estimates (data not shown).
Further, no differences were found in the outcome estimate between the two reminder groups, which is in line with findings from health survey studies on the general population [16, 19]. A study by Lall et al. used telephone interviews to collect follow-up information from non-responders in a cohort of patients with subacute and chronic LBP and found that non-responders had less favourable improvement in outcomes regarding back pain, disability and general health after 12 months . In previous research, higher depression scores were associated with a less favourable functional improvement outcome in patients with LBP , and unemployment has been identified as the main predictor of disabling pain in patients with LBP . While patients who respond to written or telephone reminders show similar outcomes, it seems reasonable to believe that the outcomes were different among patients who, regardless of reminder procedures, remain non-responders.
Combined with ethical concerns to not disturb research participants more than necessary, the degree to which reminders are employed must be thoughtfully considered, and our findings do not justify the use of telephone reminders in the present population, considering the outcome estimate .
Strengths and weaknesses
A strength of our study is our use of validated questionnaires, increasing the possibilities of comparing our findings with results from other studies. Another strength of this study is the stringent reminder procedure, where the same researcher (AR) performed all the telephone reminder calls, to ensure a consistent approach in the reminder procedure.
Patients were recruited from a spine clinic at a public hospital with an uptake area covering the Central Denmark Region, corresponding to a population of 1.3 million Danes. Access to public hospitals is equal for all and free of charge in Denmark, and we therefore believe that the findings are generalisable to the Danish and European populations with chronic LBP. This is an ancillary trial using data from a prospective study, which is a practical and ethical approach; however, it is a limitation of the design. Collecting outcomes at more time-points could provide more information. It is possible that changes in outcomes were present after three or 6 months, however we lack the information to ascertain this. This study is examining response to online questionnaires and results may differ for paper-based questionnaires. The two different methods may appeal to different populations of potential respondents with different sociodemographic, behavioral, and health characteristics [36, 37]. Generalizability of this study may therefore be limited to studies using online questionnaires only. However, studies show, that patient-reported outcomes collected through electronic questionnaires are comparable to those collected via paper questionnaires. In accordance with research in this field, the electronic versions of the questionnaires were designed to be visually comparable to the original paper versions. For these reasons, we believe that results were not affected by the fact that a small proportion of patients (N = 29) replied paper versions of the questionnaire [38, 39].
Implications for practice
Based on our findings, researchers should consider how time and resources in studies using patient-reported questionnaires is best spent. By far the most time consuming part of our data collection was the time spent on reaching non-responders by phone, and as seen in the results, using telephone-responders to increase response-rates did not change the findings on the patient reported outcomes or resulted in reaching participants with different baseline characteristics. Instead, it was evident; that patients not responding at all were systematically different from responders, indicating that missing was not at random. This further leads to the consideration, that imputation of missing data in surveys like these could lead to bias of findings, and therefore should not be performed . Rather, researchers using patient-reported outcomes issued through surveys should focus on developing data collection methods to include the target groups that are typically hard to reach.
Using a telephone reminder had no consequence on outcome estimates nor were there any differences in baseline characteristics between patients who responded before or after the telephone reminder.
Availability of data and materials
The datasets generated and/or analysed during the current study are not publicly available due to limitations of ethical approval involving the patient data and anonymity but are available from the corresponding author on reasonable request.
Low Back Pain
Major Depression Inventory
Magnetic Resonance Imaging
Roland Morris Disability Questionnaire
Foster NE, Anema JR, Cherkin D, Chou R, Cohen SP, Gross DP, et al. Prevention and treatment of low back pain: evidence, challenges, and promising directions. Lancet. 2018;391(10137):2368–83.
Hansson EK, Hansson TH. The costs for persons sick-listed more than one month because of low back or neck problems. A two-year prospective study of Swedish patients. Eur Spine J. 2005;14(4):337–45.
Lambeek LC, van Tulder MW, Swinkels IC, Koppes LL, Anema JR, van Mechelen W. The trend in total cost of back pain in the Netherlands in the period 2002 to 2007. Spine (Phila Pa 1976). 2011;36(13):1050–8.
Norlund AI, Waddel G, Nachemson AL, Jonsson E. Cost of back pain in some OECD countries. Neck And back pain: The Scientific Evidence of Causes, Diagnosis Treatment. Philadelphia: Lippincott Williams, Wilkins; 2000. p. 421–5.
Hartvigsen J, Hancock MJ, Kongsted A, Louw Q, Ferreira ML, Genevay S, et al. What low back pain is and why we need to pay attention. Lancet. 2018;391(10137):2356–67.
Koitsalu M, Eklund M, Adolfsson J, Grönberg H, Brandberg Y. Effects of pre-notification, invitation length, questionnaire length and reminder on participation rate: a quasi-randomised controlled trial. BMC Med Res Methodol. 2018;18(1):3.
Armstrong BK. In: White E, Saracci R, editors. Principles of exposure measurement in epidemiology. Paperback ed. Oxford: Oxford University Press; 1995.
Gluud LL. Bias in clinical intervention research. Am J Epidemiol. 2006;163(6):493–501.
Rothman KGS, Lash TL. Modern epidemiology. 3rd ed. Philadelphia: Lippincott Williams & Wilkins; 2008.
Edwards PJ, Roberts I, Clarke MJ, Diguiseppi C, Wentz R, Kwan I, et al. Methods to increase response to postal and electronic questionnaires. Cochrane Database Syst Rev. 2009;3:Mr000008.
Brtnikova M, Crane LA, Allison MA, Hurley LP, Beaty BL, Kempe A. A method for achieving high response rates in national surveys of U.S. primary care physicians. PLoS One. 2018;13(8):e0202755.
Funkhouser E, Fellows JL, Gordan VV, Rindal DB, Foy PJ, Gilbert GH. Supplementing online surveys with a mailed option to reduce bias and improve response rate: the National Dental Practice-Based Research Network. J Public Health Dent. 2014;74(4):276–82.
Sahlqvist S, Song Y, Bull F, Adams E, Preston J, Ogilvie D, et al. Effect of questionnaire length, personalisation and reminder type on response rate to a complex postal survey: randomised controlled trial. BMC Med Res Methodol. 2011;11:62.
Groves R, Peytcheva E. The impact of nonresponse rates on nonresponse Bias: a Meta-analysis. Public Opin Q. 2008;72:167–89.
Lee S, Brown ER, Grant D, Belin TR, Brick JM. Exploring nonresponse bias in a health survey using neighborhood characteristics. Am J Public Health. 2009;99(10):1811–7.
Brogger J, Bakke P, Eide GE, Gulsvik A. Contribution of follow-up of nonresponders to prevalence and risk estimates: a Norwegian respiratory health survey. Am J Epidemiol. 2003;157(6):558–66.
Mealing NM, Banks E, Jorm LR, Steel DG, Clements MS, Rogers KD. Investigation of relative risk estimates from studies of the same population with contrasting response rates and designs. BMC Med Res Methodol. 2010;10:26.
Sogaard AJ, Selmer R, Bjertness E, Thelle D. The Oslo health study: the impact of self-selection in a large, population-based survey. Int J Equity Health. 2004;3(1):3.
Christensen AI, Ekholm O, Kristensen PL, Larsen FB, Vinding AL, Glumer C, et al. The effect of multiple reminders on response patterns in a Danish health survey. Eur J Pub Health. 2015;25(1):156–61.
von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP. The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. Int J Surg. 2014;12(12):1495–9.
Riis A, Karran EL, Thomsen JL, Jørgensen A, Holst S, Rolving N. The association between believing staying active is beneficial and achieving a clinically relevant functional improvement after 52 weeks: a prospective cohort study of patients with chronic low back pain in secondary care. BMC Musculoskelet Disord. 2020;21(1):47.
Roland M, Morris R. A study of the natural history of back pain. Part I: development of a reliable and sensitive measure of disability in low-back pain. Spine. 1983;8(2):141–4.
Roland M, Morris R. A study of the natural history of low-back pain. Part II: development of guidelines for trials of treatment in primary care. Spine. 1983;8(2):145–50.
Patrick DL, Deyo RA, Atlas SJ, Singer DE, Chapin A, Keller RB. Assessing health-related quality of life in patients with sciatica. Spine. 1995;20(17):1899–908 discussion 909.
Albert HB, Jensen AM, Dahl D, Rasmussen MN. Criteria validation of the Roland Morris questionnaire. A Danish translation of the international scale for the assessment of functional level in patients with low back pain and sciatica. Ugeskr Laeger. 2003;165(18):1875–80.
Ostelo RW, Deyo RA, Stratford P, Waddell G, Croft P, Von Korff M, et al. Interpreting change scores for pain and functional status in low back pain: towards international consensus regarding minimal important change. Spine. 2008;33(1):90–4.
Jordan K, Dunn KM, Lewis M, Croft P. A minimal clinically important difference was derived for the Roland-Morris disability questionnaire for low back pain. J Clin Epidemiol. 2006;59(1):45–52.
Elliott AM, Hannaford PC. Third mailings in epidemiological studies: are they really necessary? Fam Pract. 2003;20(5):592–4.
Wensing M, Schattenberg G. Initial nonresponders had an increased response rate after repeated questionnaire mailings. J Clin Epidemiol. 2005;58(9):959–61.
Breen CL, Shakeshaft AP, Doran CM, Sanson-Fisher RW, Mattick RP. Cost-effectiveness of follow-up contact for a postal survey: a randomised controlled trial. Aust N Z J Public Health. 2010;34(5):508–12.
Sammut DR, Griscti DO, Norman PIJ. Strategies to improve response rates to web surveys: a literature review. Int J Nurs Stud. 2021;123:104058. https://doi.org/10.1016/j.ijnurstu.2021.104058.
Schirmer J. Ethical issues in the use of multiple survey reminders. J Acad Ethics. 2009;7(1):125–39.
Lall R, Mistry D, Bridle C, Lamb SE. Telephone interviews can be used to collect follow-up data subsequent to no response to postal questionnaires in clinical trials. J Clin Epidemiol. 2012;65(1):90–9.
Campbell P, Hope K, Dunn KM. The pain, depression, disability pathway in those with low back pain: a moderation analysis of health locus of control. J Pain Res. 2017;10:2331–9.
Dunn KM, Jordan KP, Croft PR. Contributions of prognostic factors for poor outcome in primary care low back pain patients. Eur J Pain. 2011;15(3):313–9.
Fitzgerald D, Hockey R, Jones M, Mishra G, Waller M, Dobson A. Use of Online or Paper Surveys by Australian Women: Longitudinal Study of Users, Devices, and Cohort Retention. J Med Internet Res. 2019;21(3):e10672–e.
Ward P, Clark T, Zabriskie R, Morris T. Paper/pencil versus online data collection. J Leis Res. 2014;46(1):84–105.
Coons SJ, Gwaltney CJ, Hays RD, Lundy JJ, Sloan JA, Revicki DA, et al. Recommendations on evidence needed to support measurement equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO good research practices task force report. Value Health. 2009;12(4):419–29. https://doi.org/10.1111/j.1524-4733.2008.00470.x.
Griffiths-Jones W, Norton MR, Fern ED, Williams DH. The equivalence of remote electronic and paper patient reported outcome (PRO) collection. J Arthroplast. 2014;29(11):2136–9. https://doi.org/10.1016/j.arth.2014.07.003.
Sterne JA, White IR, Carlin JB, Spratt M, Royston P, Kenward MG, et al. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ. 2009;338:b2393. https://doi.org/10.1136/bmj.b2393.
This study was supported by a grant from the Novo Nordisk Foundation (NNF17OC0024422). They had no role in the study design, writing of the manuscript or the decision to submit the manuscript for publication.
Ethics approval and consent to participate
All patients gave written informed consent online and were informed that their consent could be withdrawn at any time. This is an observational study, and all patients received treatment as usual. In accordance with the Danish Scientific Ethical Committees Act, survey studies do not require approval by an ethics committee (§14, part 2) . All experimental protocols were approved by the executive management of the Spine Centre at Silkeborg Regional Hospital. The original trial is registered with the Danish Data Protection Agency (Central Denmark Region, journal no. 1–16–02-23-17).
Consent for publication
AR is member of the editorial board at BMC Musculoskeletal Disorders. The other authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Udby, C.L., Riis, A., Thomsen, J.L. et al. Does the use of telephone reminders to increase survey response rates affect outcome estimates? An ancillary analysis of a prospective cohort study of patients with low back pain. BMC Musculoskelet Disord 22, 893 (2021). https://doi.org/10.1186/s12891-021-04787-4
- Research design
- Reminder systems
- Patient outcome assessment
- Low Back pain