About This Journal

korean management review - Vol. 50 , No. 5
[Paper List] [Go to Volume List]

[ Article ]
korean management review - Vol. 50, No. 5, pp. 1401-1434
Abbreviation: kmr
ISSN: 1226-1874 (Print)
Print publication date 31 Oct 2021
Received 26 Feb 2021 Revised 18 Jun 2021 Accepted 05 Jul 2021
DOI: https://doi.org/10.17287/kmr.2021.50.5.1401

Causes of Careless Responding in Surveys and Preventions in the Survey Design
Won-Woo Park ; Yoowoo Lee ; Sunghyuck Mah ; Jayoung Kim ; Suhyun Bae ; Seonyoung Ji
(First Author) Professor, College of Business Administration, Seoul National University (wwpark@snu.ac.kr)
(Corresponding Author) Seoul National University (youfmc@snu.ac.kr)
(Co-Author) Seoul National University (ryanshmah@snu.ac.kr)
(Co-Author) Seoul National University (marsb612@snu.ac.kr)
(Co-Author) Seoul National University (top3043@snu.ac.kr)
(Co-Author) Seoul National University (ji179@purdue.edu)

설문조사에서 불성실 응답의 원인과 조사설계 단계에서의 예방

Copyright 2011 THE KOREAN ACADEMIC SOCIETY OF BUSINESS ADMINISTRATION
This is an open access article distributed under the terms of the Creative Commons Attribution License 4.0, which permits unrestricted, distribution, and reproduction in any medium, provided the original work is properly cited.
Funding Information ▼
Abstract

This study investigates the level of Korean researchers' awareness of careless responding (CR) in surveys, the causes of CR, as well as the potential remedies available with regards to survey design. We first reviewed 827 survey studies published in seven major journals in the field of business administration and analyzed the level of recognition of Korean researchers regarding CR. We also investigated the causes of CR through a literature review that particularly focused on the motives and burdens of respondents. We then conducted in-depth interviews with academic and practical experts to investigate the perceptions and practices regarding CR. Finally, based on the literature review and the in-depth interviews, we drew up measures to prevent CR. We suggest that there need to be academic standards that prompt researchers to report specific CR prevention or detection methods used for their research in a more detailed manner. This study, as a cornerstone, contributes to further discussions and the development of preventive measures of CR in the Korean scholarly community, particularly in the field of business administration.

Keywords: careless responding, causes of careless responding, prevention of careless responding, survey design
Acknowledgments

This research is supported by the Institute of Industrial Relations of Seoul National University Research Fund (in 2021).

References
1. Akbulut, Y.(2015), “Predictors of Inconsistent Responding in Web Surveys,” Internet Research, 25(1), pp.131-147.
2. Anduiza, E. and C. Galais(2017), “Answering Without Reading: IMCs and Strong Satisficing in Online Surveys,” International Journal of Public Opinion Research, 29(3), pp.497-519.
3. Bäckström, M. and F. Björklund(2019), “Is Reliability Compromised Towards the End of Long Personality Inventories?,” European Journal of Psychological Assessment, 35 (1), pp.14-23.
4. Beach, D. A.(1989), “Identifying the Random Responder,” The Journal of Psychology, 123 (1), pp.101-103.
5. Berry, D. T., M. W. Wetter, R. A. Baer, L. Larsen, C. Clark, and K. Monroe(1992), “MMPI-2 Random Responding Indices: Validation Using a Self-Report Methodology,” Psychological Assessment, 4(3), pp.340-345.
6. Blair, E., S. Sudman, N. M. Bradburn, and C. Stocking(1977), “How to Ask Questions about Drinking and Sex: Response Effects in Measuring Consumer Behavior,” Journal of Marketing Research, 14(3), pp.316-321.
7. Bowling, N. A., J. L. Huang, C. B. Bragg, S. Khazon, M. Liu, and C. E. Blackmore(2016), “Who Cares and Who Is Careless? Insufficient Effort Responding as a Reflection of Respondent Personality,” Journal of Personality and Social Psychology, 111(2), pp.218-229.
8. Breitsohl, H. and C. Steidelmüller(2018), “The Impact of Insufficient Effort Responding Detection Methods on Substantive Responses: Results from an Experiment Testing Parameter Invariance,” Applied Psychology, 67(2), pp. 284-308.
9. Brower, C. K.(2018), Too Long and Too Boring: The Effects of Survey Length and Interest on Careless Responding [Master’s thesis, Wright State University], CORE Scholar. https://corescholar.libraries.wright.edu/etd_all/1918(retrieved February 2021).
10. Carrier, L. M., N. A. Cheever, L. D. Rosen, S. Benitez, and J. Chang(2009), “Multitasking Across Generations: Multitasking Choices and Difficulty Ratings in Three Generations of Americans,” Computers in Human Behavior, 25(2), pp.483-489.
11. Camus, K. A.(2015), Once Careless, Always Careless? Temporal and Situational Stability of Insufficient Effort Responding (IER) [Master’s thesis, Wright State University], CORE Scholar. https://corescholar.libraries.wright.edu/etd_all/1617(retrieved February 2021).
12. Cantor, D., B. C. O’Hare, and K. S. O’Connor (2008), “The Use of Monetary Incentives to Reduce Nonresponse in Random Digit Dial Telephone Surveys,” Advances in Telephone Survey Methodology, pp.471-498.
13. Chmielewski, M. and S. C. Kucker(2020), “An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results,” Social Psychological and Personality Science, 11(4), pp.464-473.
14. Couper, M. P.(2008), Designing Effective Web Surveys, New York, Cambridge University Press.
15. Couper, M. P., M. W. Traugott, and M. J. Lamias (2001), “Web Survey Design and Administration,” Public Opinion Quarterly, 65(2), pp.230-253.
16. Curran, P. G.(2016), “Methods for the Detection of Carelessly Invalid Responses in Survey Data,” Journal of Experimental Social Psychology, 66, pp.4-19.
17. DeSimone, J. A., H. K. Davison, J. L. Schoen, and M. N. Bing(2020), “Insufficient Effort Responding as a Partial Function of Implicit Aggression,” Organizational Research Methods, 23(1), pp.154-180.
18. DeSimone, J. A., A. J. DeSimone, P. D. Harms, and D. Wood(2018), “The Differential Impacts of Two Forms of Insufficient Effort Responding,” Applied Psychology, 67(2), pp.309-338.
19. Deutskens, E., K. De Ruyter, M. Wetzels, and P. Oosterveld(2004), “Response Rate and Response Quality of Internet-Based Surveys: An Experimental Study,” Marketing Letters, 15(1), pp.21-36.
20. Dickinson, T. L. and P. M. Zellinger(1980), “A Comparison of the Behaviorally Anchored Rating and Mixed Standard Scale Formats,” Journal of Applied Psychology, 65(2), pp. 147-154.
21. Diefendorff, J. M. and M. M. Chandler(2011), “Motivating Employees,” In S. Zedeck (Ed.), APA Handbook of Industrial and Organizational Psychology, Vol 3: Maintaining, Expanding, and Contracting the Organization, Washington, DC, American Psychological Association, pp.65-135.
22. Dunn, A. M., E. D. Heggestad, L. R. Shanock, and N. Theilgard(2018), “Intra-Individual Response Variability as an Indicator of Insufficient Effort Responding: Comparison to Other Indicators and Relationships with Individual Differences,” Journal of Business and Psychology, 33(1), pp.105-121.
23. Fan, J., D. Gao, S. A. Carroll, F. J. Lopez, T. S. Tian, and H. Meng(2012), “Testing the Efficacy of a New Procedure for Reducing Faking on Personality Tests Within Selection Contexts,” Journal of Applied Psychology, 97(4), pp.866-880.
24. Fleischer, A., Mead, A. D., and J. Huang(2015), “Inattentive responding in MTurk and other online samples,” Industrial and Organizational Psychology, 8(2), pp.196-202.
25. Fleming, A. C., Hlebasko, H., Adams, S. C., Roach, K. N., and N. D. Christiansen(2020), “Effects of sexism and job–applicant match on leadership candidate evaluations,” Social Behavior and Personality: An International Journal, 48(9), pp.1-8.
26. Galesic, M.(2006), “Dropouts on the Web: Effects of Interest and Burden Experienced During an Online Survey,” Journal of Official Statistics, 22(2), pp.313-328.
27. Galesic, M., and M. Bosnjak(2009), “Effects of Questionnaire Length on Participation and Indicators of Response Quality in a Web Survey,” Public Opinion Quarterly, 73(2), pp.349-360.
28. Gehlbach, H. and S. Barge(2012), “Anchoring and Adjusting in Questionnaire Responses,” Basic and Applied Social Psychology, 34(5), pp.417-433.
29. Gibson, A.(2016), Stop With the Questions Already! The Effects of Questionnaire Length and Monetary Incentives on Insufficient Effort Responding [Master's thesis, Wright State University], CORE Scholar, https://corescholar.libraries.wright.edu/etd_all/1656(retrieved February 2021).
30. Gibson, A.(2019), Stop What You’re Doing, Right Now! Effects of Interactive Messages on Careless Responding [Doctoral dissertation, Wright State University], CORE Scholar, https://corescholar.libraries.wright.edu/etd_all/2119/(retrieved February 2021).
31. Gibson, A. M. and N. A. Bowling(2020), “The Effects of Questionnaire Length and Behavioral Consequences on Careless Responding,” European Journal of Psychological Assessment, 36(2), pp.410-420.
32. Gideon, L.(2012), Handbook of Survey Methodology for the Social Sciences, New York, Springer.
33. Givon, M. M. and Z. Shapira(1984), “Response to Rating Scales: A Theoretical Model and Its Application to the Number of Categories Problem,” Journal of Marketing Research, 21(4), pp.410-419.
34. Goldammer, P., Annen, H., Stöckli, P. L., and K. Jonas(2020), “Careless responding in questionnaire measures: Detection, impact, and remedies,” The Leadership Quarterly, 31(4), 101384.
35. Göritz, A. S.(2004), “The Impact of Material Incentives on Response Quantity, Response Quality, Sample Composition, Survey Outcome, and Cost in Online Access Panels,” International Journal of Market Research, 46 (3), pp.327-345.
36. Grau, I., C. Ebbeler, and R. Banse(2019), “Cultural Differences in Careless Responding,” Journal of Cross-Cultural Psychology, 50(3), pp. 336-357.
37. Groves, R. M.(1989), Survey Errors and Survey Costs. New York, John Wiley and Sons.
38. Huang, J. L., N. A. Bowling, M. Liu, and Y. Li (2015), “Detecting Insufficient Effort Responding with an Infrequency Scale: Evaluating Validity and Participant Reactions,” Journal of Business and Psychology, 30(2), pp.299-311.
39. Huang, J. L., P. G. Curran, J. Keeney, E. M. Poposki, and R. P. DeShon(2012), “Detecting and Deterring Insufficient Effort Responding to Surveys,” Journal of Business and Psychology, 27(1), pp.99-114.
40. Huang, J. L., M. Liu, and N. A. Bowling(2015), “Insufficient Effort Responding: Examining an Insidious Confound in Survey Data,” Journal of Applied Psychology, 100(3), pp. 828-845.
41. Huttenlocher, J., L. V. Hedges, and N. M. Bradburn (1990), “Reports of Elapsed Time: Bounding and Rounding Processes in Estimation,” Journal of Experimental Psychology: Learning, Memory, and Cognition, 16(2), pp.196-213.
42. Johnson, J. A.(2005), “Ascertaining the Validity of Individual Protocols from Web-Based Personality Inventories,” Journal of Research in Personality, 39(1), pp.103-129.
43. Joshi, A., S. Kale, S. Chandel, and D. K. Pal(2015), “Likert Scale: Explored and Explained,” Current Journal of Applied Science and Technology, 7(4), pp.396-403.
44. Jun, J., T. Lee, S. Chung, and C. Park(2017), “A Study on Determinants of Intention to Use Car Sharing: Moderating Effect of Consumer Innovativeness,” Journal of Marketing Management Research, 22(2), pp.49-66.
45. Kanfer, R., M. Frese, and R. E. Johnson(2017), “Motivation Related to Work: A Century of Progress,” Journal of Applied Psychology, 102(3), pp.338-355.
46. Kang, C. W., S. B. Choi, and M. S. Kwon(2020), “False Responses Detection of Survey System using Machine Learning,” Journal of The Korean Data Analysis Society, 22(3), pp. 1013-1021.
47. Kang, W., C. Yoo, and D. Kim(2015), “The Effects of Negative Publicity on Brand Evaluation: Moderating Roles of Brand Strategy and Prior Attitude,” Journal of Marketing Management Research, 20(3), pp.65-99.
48. Keith, M. G., Tay, L., and P. D. Harms(2017), “Systems perspective of Amazon Mechanical Turk for organizational research: Review and recommendations,” Frontiers in Psychology, 8, 1359.
49. Kim, Y., J. Dykema, J. Stevenson, P. Black, and D. P. Moberg(2019), “Straightlining: Overview of Measurement, Comparison of Indicators, and Effects in Mail–Web Mixed-Mode Surveys,” Social Science Computer Review, 37(2), pp. 214-233.
50. Krosnick, J. A.(1991), “Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys,” Applied Cognitive Psychology, 5(3), pp.213-236.
51. Krosnick, J. A. and M. K. Berent(1993), “Comparisons of Party Identification and Policy Preferences: The Impact of Survey Question Format,” American Journal of Political Science, 37 (3), pp.941-964.
52. Kühne, S. and M. Kroh(2018), “Personalized Feedback in Web Surveys: Does it Affect Respondents’ Motivation and Data Quality?,” Social Science Computer Review, 36(6), pp.744-755.
53. Laurent, A.(1972), “Effects of Question Length on Reporting Behavior in the Survey Interview,” Journal of the American Statistical Association, 67(338), pp.298-305.
54. Lee, Y. S., J. Y. Lee, and K. T. Lee(2008), “Amounts of Responding Times and Unreliable Responses at Online Surveys,” Survey Research, 9(2), pp.51-83.
55. Liu, M., N. A. Bowling, J. L. Huang, and T. A. Kent(2013), “Insufficient Effort Responding to Surveys as a Threat to Validity: The Perceptions and Practices of SIOP Members,” The Industrial-Organizational Psychologist, 51(1), pp.32-38.
56. Maniaci, M. R., and R. D. Rogge(2014), “Caring about Carelessness: Participant Inattention and Its Effects on Research,” Journal of Research in Personality, 48(1), pp.61-83.
57. Marshall, A.(2019), Caring More About Careless Responding: Applying the Theory of Planned Behavior to Reduce Careless Responding on Online Surveys [Doctoral dissertation, Colorado State University], Mountain Scholar, https://hdl.handle.net/10217/195385(retrieved February 2021).
58. McKay, A. S., D. M. Garcia, J. P. Clapper, and K. S. Shultz(2018), “The Attentive and the Careless: Examining the Relationship Between Benevolent and Malevolent Personality Traits with Careless Responding in Online Surveys,” Computers in Human Behavior, 84, pp.295-303.
59. Meade, A. W. and S. B. Craig(2012), “Identifying Careless Responses in Survey Data,” Psychological Methods, 17(3), pp.437-455.
60. Mondak, J. J.(2001), “Developing Valid Knowledge Scales,” American Journal of Political Science, 45(1), pp.224-238.
61. Nederhof, A. J.(1985), “Methods of coping with social desirability bias: A review,” European Journal of Social Psychology, 15(3), pp. 263-280.
62. Nguyen, H. L. T.(2017), Tired of Survey Fatigue? Insufficient Effort Responding Due to Survey Fatigue [Master’s thesis, Middle Tennessee State University], JEWLScholar@MTSU, http://jewlscholar.mtsu.edu/xmlui/handle/mtsu/5302(retrieved February 2021).
63. Nichols, A. L. and J. E. Edlund(2020), “Why Don't We Care More About Carelessness? Understanding the Causes and Consequences of Careless Participants,” International Journal of Social Research Methodology, 23(6), pp. 625-638.
64. O'Muircheartaigh, C. and P. Campanelli(1999), “A Multilevel Exploration of the Role of Interviewers in Survey Non‐response,” Journal of the Royal Statistical Society: Series A (Statistics in Society), 162(3), pp.437-446.
65. Palaniappan, K. and I. Y. S. Kum(2019), “Underlying Causes Behind Research Study Participants’ Careless and Biased Responses in the Field of Sciences,” Current Psychology, 38(6), pp. 1737-1747.
66. Park, H-J. and S. H. Lee(2017), Scientific Survey Method to Change the World, Seoul, Republic of Korea, Chang-ji Co.
67. Park, W-W., M. S. Kim, S. M. Jeong, and K. M. Huh(2007), “Cause and Remedies of Common Method Bias,” Korean Journal of Management, 15, pp.89-133.
68. Park, W-W., S. H. Mah, S. H. Bae, S. Y. Ji, Y. W. Lee, and J. Y. Kim(2020), “Careless Responding in Surveys: Detection Methods and the Impact of Screening on Research,” Korean Management Review, 49(2), pp.331-364.
69. Podsakoff, P. M., S. B. MacKenzie, J. Y. Lee, and N. P. Podsakoff(2003), “Common Method Biases in Behavioral Research: A Critical Review of the Literature and Recommended Remedies. Journal of Applied Psychology, 88(5), pp.879-903.
70. Price, J. H., Daek, J. A., Murnan, J., Dimmig, J., and S. Akpanudo(2005), “Power analysis in survey research: Importance and use for health educators,” American Journal of Health Education, 36(4), pp.202-209.
71. Reymond, P., H. Weber, M. Damond, and E. E. Farmer(2000), “Differential Gene Expression in Response to Mechanical Wounding and Insect Feeding in Arabidopsis,” The Plant Cell, 12(5), pp.707-719.
72. Rogelberg, S. G. and J. M. Stanton(2007), “Introduction: Understanding and Dealing with Organizational Survey Nonresponse,” Organizational Research Methods, 10(2), pp.195-209.
73. Saris, W. E. and I. N. Gallhofer(2014), Design, Evaluation, and Analysis of Questionnaires for Survey Research, New York, John Wiley and Sons.
74. Schmitt, N. and D. M. Stults(1985), “Factors Defined by Negatively Keyed Items: The Result of Careless Respondents?,” Applied Psychological Measurement, 9, pp.367-373.
75. Seo, R. and J. Lee(2019), “How to Manage the Green-Eyed Monster: Money Priming May Modify the Effects of Jealousy on the Preferences for Attention-Grabbing and Luxury Products,” Korean Management Review, 48 (5), pp.1221-1252.
76. Shamon, H. and C. C. Berning(2020), “Attention Check Items and Instructions in Online Surveys with Incentivized and Non-Incentivized Samples: Boon or Bane for Data Quality?,” Survey Research Methods, 14(1), pp.55-77.
77. Shuman, H. and S. Presser(1981), Questions and Answers in Attitude Surveys Experiments on Questions Forms and Context. Thousand Oaks, CA, Sage.
78. Singer, E.(2002), “The Use of Incentives to Reduce Nonresponse in Household Surveys,” In R. M. Groves, D. A. Dillman, J. L. Eltinge, and R. J. A. Little (Eds.), Survey Nonresponse, New York, Wiley, pp.163-177.
79. Singer, E. and C. Ye(2013), “The Use and Effects of Incentives in Surveys,” The Annals of the American Academy of Political and Social Science, 645(1), pp.112-141.
80. Song, M. R., W. Chu, and M. Im(2018), “Psychological Characteristics and Usage Patterns of Electric Vehicle Owners in South Korea,” Korean Management Review, 47(2), pp.271-306.
81. Swan, M.(2013), “The Quantified Self: Fundamental Disruption in Big Data Science and Biological Discovery,” Big Data, 1(2), pp.85-99.
82. Tourangeau, R., M. P. Couper, and F. Conrad(2004), “Spacing, Position, and Order: Interpretive Heuristics for Visual Features of Survey Questions,” Public Opinion Quarterly, 68(3), pp.368-393.
83. Tourangeau, R., E. Singer, and S. Presser(2003), “Context Effects in Attitude Surveys: Effects on Remote Items and Impact on Predictive Validity,” Sociological Methods and Research, 31(4), pp.486-513.
84. Vigil-Colet, A., D. Navarro-Gonzalez, and F. Morales-Vives(2020), “To Reverse or to not Reverse Likert-Type Items: That is the Question,” Psicothema, 32(1), pp.108-114.
85. Ward, M. K. and A. W. Meade(2018), “Applying Social Psychology to Prevent Careless Responding During Online Surveys,” Applied Psychology, 67(2), pp.231-263.
86. Ward, M. K., A. W. Meade, C. M. Allred, G. Pappalardo, and J. W. Stoughton(2017), “Careless Response and Attrition as Sources of Bias in Online Survey Assessments of Personality Traits and Performance,” Computers in Human Behavior, 76, pp.417-430.
87. Ward, M. K. and S. B. Pond Ⅲ(2015), “Using Virtual Presence and Survey Instructions to Minimize Careless Responding on Internet-Based Surveys,” Computers in Human Behavior, 48, pp.554-568.
88. Weijters, B., A. De Beuckelaer, and H. Baumgartner (2014), “Discriminant Validity Where There Should Be None: Positioning Same-Scale Items in Separated Blocks of a Questionnaire,” Applied Psychological Measurement, 38(6), pp.450-463.
89. Wenz, A., A. Jackle, J. Burton, and M. P. Couper (2020), “The Effects of Personalized Feedback on Participation and Reporting in Mobile App Data Collection,” Social Science Computer Review, Advance online publication.
90. Woodside, A. G. and C. Dubelaar(2003), “Increasing Quality in Measuring Advertising Effectiveness: A Meta-Analysis of Question Framing in Conversion Studies,” Journal of Advertising Research, 43(1), pp.78-85.
91. Zwarun, L. and A. Hall(2014), “What’s Going On? Age, Distraction, and Multitasking During Online Survey Taking,” Computers in Human Behavior, 41, pp.236-244.

∙ The author Won-Woo Park received Bachelor and Master of Business Administration from Seoul National University, and Ph.D. (1989) from the University of Pittsburgh. He worked as a faculty member of some private universities of the USA and of South Korea before joining Seoul National University in 1998. He published about 130 research articles and 16 books, received teaching awards from both the college (Business School) and the university (SNU). He was the recipient of the ‘2020 Emerald Literati Award’ from Emerald Publishing.

∙ The author Yoowoo Lee is a Ph.D. majoring in Vocational Education & Workforce Development at Seoul National University (SNU). She received her MA in majoring in organizational behaviors and human resource management in the School of Business Administration at Chungang University (CAU). Her research interests include age management, mid-career change, and survey research methodology.

∙ The author Sunghyuck Mah is a Ph.D. candidate majoring in organizational behaviors and human resource management at the School of Business Administration, Seoul National University (SNU). He earned his Bachelor and Master of Business Administration from Yonsei University and SNU, respectively. His research interests include empowering leadership, overqualification among young generations, and survey research methodology.

∙ The author Jayoung Kim is a Ph.D. candidate majoring in strategy and international management at the Graduate School of Business, Seoul National University. She received her BS in Housing & Interior Design from Yonsei University and MS in Business Administration from Seoul National University. Her current research interests include growth strategies of the platform business, entrepreneurship, and research methodology.

∙ The author Suhyun Bae is a Ph.D. majoring in Vocational Education & Workforce Development at Seoul National University (SNU). She received her BA in Psychology from the University at Albany - State University of New York, and MA in Education from Seoul National University. Major research areas include HRD, generational conflict, organizational culture, vocational education, and research methodology.

∙ The author Seonyoung Ji is a student of Organizational Behavior & Human Resources Ph.D. program at Purdue University, West Lafayette. She received her B.B.A. degree from Sungkyunkwan University and M.S. in Business Administration from Seoul National University. Her main research interests include meaningful work, positive organizational psychology, and research methodology.

 
Mail : editor_kmr@daum.net
COPYRIGHT ⓒ 2011 Korean Academic Society Of Business Administration. All rights reserved