About This Journal

korean management review - Vol. 49 , No. 2
[Paper List] [Go to Volume List]

[ Article ]
korean management review - Vol. 49, No. 2, pp. 331-364
Abbreviation: kmr
ISSN: 1226-1874 (Print)
Print publication date 30 Apr 2020
Received 21 Aug 2019 Revised 27 Dec 2019 Accepted 23 Jan 2020
DOI: https://doi.org/10.17287/kmr.2020.49.2.331

Careless Responding in Surveys: Detection Methods and the Impact of Screening on Research
Won-woo Park* ; Sung Hyuck Mah** ; Su hyun Bae*** ; Seon Young Ji**** ; Yoo Woo Lee***** ; Ja Young Kim******
*Seoul National University, First Author
**Seoul National University, Corresponding Author
***Seoul National University, Co-Author
****Seoul National University, Co-Author
*****Seoul National University, Co-Author
******Seoul National University, Co-Author

설문조사에서 불성실 응답의 탐지방법과 제거의 효과
박원우* ; 마성혁** ; 배수현*** ; 지선영**** ; 이유우***** ; 김자영******
*(주저자) 서울대학교 (wwpark@snu.ac.kr)
**(교신저자) 서울대학교 (ryanshmah@snu.ac.kr)
***(공저자) 서울대학교 (top3043@snu.ac.kr)
****(공저자) 서울대학교 (seonyoungji@gmail.com)
*****(공저자) 서울대학교 (youfmc@snu.ac.kr)
******(공저자) 서울대학교 (marsb612@snu.ac.kr)
Funding Information ▼
Abstract

This study investigates the phenomenon of ‘careless responding’ that prevails in surveys. Specifically, we focus on methods of detection and the effects of screening careless responding. First, this study discusses the diverse definitions and types of careless responding. Methods of detection are introduced in the following triangular categorization: direct, unobtrusive, and statistical. The theoretical review portion provides a comparative summary of detection methods and their advantages and disadvantages to help future researchers apply a suitable one to their own study designs. Furthermore, this study conducts an empirical study to identify the impact of screening careless responding on statistical results, analyzing survey data from 3,030 employees working in a major conglomerate in Korea. The major findings are as follows: (1) According to the applied method, the target and the proportion of screened responses varied, ranging from 0.5% to 14% at maximum. Only a small proportion was detected coincidentally by two or more methods. (2) The screening of careless responding only had a slight impact on statistical figures of factor analysis, correlations, and regression. Such less than moderate impacts were in line with earlier findings and less threatening to the validation of research models. (3) Lastly, we demonstrated a multi-hurdling method that adopted two methods sequentially. To conclude, the study discusses possible applications of detection methods and avenues for future research.

초록

본 연구는 설문조사에서 발생하는 불성실 응답(careless responding)을 탐지하는 방법과 이를 제거함으로써 얻을 수 있는 효과에 대해 논의한다. 우선, 본 연구는 설문조사에서 발생하는 다양한 불성실 응답의 개념과 유형을 정리하여 설명하였다. 이어서, 최근 활발히 연구가 진행되고 있는 불성실 응답의 다양한 탐지방법들을 직접적, 비개입적, 통계적 방법으로 나누어 소개하였다. 특히, 향후 연구자들이 자신의 연구 맥락에 맞추어 이를 활용할 수 있도록 각 방법이 지닌 의미와 장단점, 그리고 탐지 기준을 비교하여 안내하였다. 나아가 본 연구에서는 소개된 다수의 불성실 응답 탐지방법 및 제거의 효과를 확인하기 위해, 국내 대기업 구성원 3,030명을 대상으로 한 설문조사 결과를 활용하여 별도의 실증분석을 실시하였다. 여기에는 총 다섯 가지 불성실 응답 탐지방법이 적용되었으며, 각 방법에 의하여 탐지된 불성실 응답을 제거하기 전․후의 통계적 결과를 비교하는 방식으로 효과를 검증하였다. 실증연구를 통해 확인한 주요 결론은 다음과 같다. 첫째, 탐지에 활용된 방법에 따라, 불성실 응답으로 분류되는 응답자의 숫자와 비율이 달라졌다. 개별 탐지방법에 따라 약 0.5%~14%의 답변이 불성실 응답으로 분류되는 것으로 나타났다. 둘째, 불성실 응답의 제거가 통계적 추론 결과에 미치는 영향을 확인한 결과, 불성실 응답의 존재가 상관관계, 요인분석, 회귀분석 결과에 미치는 영향은 기존 문헌의 결과와 유사한 수준으로, 연구 가설의 결과를 왜곡시킬 정도로 크지 않은 것으로 나타났다. 셋째, 2개 이상의 탐지방법을 조합하는 다중 허들방식(multi-hurdle)을 적용하여 불성실 응답을 탐지한 예시를 보여주었다. 마지막으로 본 연구에서는 문헌고찰과 실증분석 결과를 바탕으로, 연구자들이 설문조사에서 활용할 수 있는 불성실 응답 탐지방법과 향후 관련 연구방향을 제시하였다.

Keywords: Research Methodology, Surveys, Careless Responding
키워드: 연구방법론, 설문조사, 불성실 응답
Acknowledgments

본 연구는 서울대학교 노사관계연구소 연구비 지원에 의하여 작성되었습니다.

References
1. 박원우·김미숙·정상명·허규만(2007), “동일방법편의(Common Method Bias)의 원인과 해결방안,” 인사조직연구, 15(1), pp.89-133.
2. 백영민·김은미·이준웅(2012), “자기응답방법에서 나타나는 인터넷이용시간 과도응답과 그 원인,” 한국언론학보, 56(2), pp.121-142.
3. 이윤석·이지영·이경택(2008), “온라인조사의 응답오차에 대한 연구: 설문 응답 시간과 응답 성실성의 관계,” 조사연구, 9(2), pp.51-83.
4. Akbulut, Y.(2015), “Predictors of inconsistent responding in web surveys,” Internet Research, 25(1), pp.131-147.
5. Bachman, J. G., and P. M. O’Malley(1984), “Yeasaying, nay-saying, and going to extremes: Black-white differences in response style,” Public Opinion Quarterly, 48(2), pp.491-509.
6. Baer, R. A., J. Ballenger, D. T. R. Berry, and M. W. Wetter(1997), “Detection of random responding on the MMPI-A,” Journal of Personality Assessment, 68(1), pp.139-151.
7. Bagby, R. M., J. R. Gillis, and R. Rogers(1991), “Effectiveness of the Millon Clinical Multi Axial Inventory validity index in the detection of random responding,” Psychological Assessment, 3(2), pp.285-287.
8. Bagozzi, R., Y. Yi, and L. Phillips(1991), “Assessing construct validity in organizational research,” Administrative Science Quarterly, 36(3), pp.421-458.
9. Bardo, J. W., S. J. Yeager, and M. J. Klingsporn (1982), “Preliminary assessment of formatspecific central tendency and leniency error in summated rating scales,” Perceptual and Motor Skills, 54(1), pp.227-234.
10. Barge, S., and H. Gehlbach(2012), “Using the theory of satisficing to evaluate the quality of survey data,” Research in Higher Education, 53(2), pp.182-200.
11. Beach, D. A.(1989), “Identifying the random responder,” Journal of Psychology: Interdisciplinary and Applied, 123(1), pp.101-103.
12. Behrend, T. S., D. J. Sharek, A. W. Meade, and E. N. Wiebe(2011), “The viability of crowd sourcing for survey research,” Behavior Research Methods, 43(3), pp.800-813.
13. Berry, D. T., M. W. Wetter, R. A. Baer, L. Larsen, C. Clark, and K. Monroe(1992), “MMPI-2 random responding indices: Validation using a self-report methodology,” Psychological Assessment, 4(3), pp.340-345.
14. Breitsohl, H., and C. Steidelmüller(2018), “The impact of insufficient effort responding detection methods on substantive responses: Results from an experiment testing parameter invariance,” Applied Psychology, 67 (2), pp.284-308.
15. Brief, A. P., and S. J. Motowidlo(1986), “Prosocial organizational behaviors,” Academy of Management Review, 11(4), pp.710-725.
16. Bowling, N. A., and J. L. Huang(2018), “Your attention please! Toward a better understanding of research participant carelessness,” Applied Psychology, 67(2), pp.227-230.
17. Bowling, N. A., J. L. Huang, C. B. Bragg, S. Khazon, M. Liu, and C. E. Blackmore(2016), “Who cares and who is careless? Insufficient effort responding as a reflection of respondent personality,” Journal of Personality and Social Psychology, 111(2), pp.218-229.
18. Calsyn, R. J., and J. P. Winter(1999), “Understanding and controlling response bias in needs assessment studies,” Evaluation Review, 23 (4), pp.399-417.
19. Cannell, C. F., P. V. Miller, and L. Oksenberg(1981), “Research on interviewing techniques,” Sociological Methodology, 12, pp.389-437.
20. Chen, G., S. M. Gully, and D. Eden(2001). “Validation of a new general self-efficacy scale,” Organizational Research Methods, 4 (1), pp.62-83.
21. Costa, P. T., and R. R. McCrae(2008), “The revised NEO personality inventory (NEO-PI-R),” In G. J. Boyle, G. Matthews, and D. H. Saklofske (Eds.), The SAGE Handbook of Personality Theory and Assessment, London, England, SAGE, pp.179-198.
22. Credé, M.(2010), “Random responding as a threat to the validity of effect size estimates in correlational research,” Educational and Psychological Measurement, 70(4), pp.596-612.
23. Curran, P. G.(2016), “Methods for the detection of carelessly invalid responses in survey data,” Journal of Experimental Social Psychology, 66(5), pp.4-19.
24. Curran, P. G., L. Kotrba, and D. Denison(2010, April), “Careless responding in surveys: Applying traditional techniques to organizational settings,” In 25th Annual Conference of Society for Industrial and Organizational Psychology, Atlanta, GA.
25. DeSimone, J. A., A. J. DeSimone, P. D. Harms, and D. Wood(2018), “The differential impacts of two forms of insufficient effort responding,” Applied Psychology, 67(2), pp.309-338.
26. DeSimone, J. A., and P. D. Harms(2018), “Dirty data: The effects of screening respondents who provide low-quality data in survey research,” Journal of Business and Psychology, 33(5), pp.559-577.
27. DeSimone, J. A., P. D. Harms, and A. J. DeSimone (2015), “Best practice recommendations for data screening,” Journal of Organizational Behavior, 36(2), pp.171-181.
28. Deutskens, E., K. De Ruyter, M. Wetzels, and P. Oosterveld(2004), “Response rate and response quality of internet-based surveys: An experimental study,” Marketing Letters, 15 (1), pp.21-36.
29. Dunn, A. M., E. D. Heggestad, L. R. Shanock, and N. Theilgard(2018), “Intra-individual response variability as an indicator of insufficient effort responding: Comparison to other indicators and relationships with individual differences,” Journal of Business and Psychology, 33(1), pp.105-121.
30. Gallen, R. T., and D. T. Berry(1996), “Detection of random responding in MMPI-2 protocols,” Assessment, 3(2), pp.171-178.
31. Goldberg, L. R., and J. M. Kilkowski(1985), “The prediction of semantic consistency in selfdescriptions: Characteristics of persons and of terms that affect the consistency of responses to synonym and antonym pairs,” Journal of Personality and Social Psychology, 48 (1), pp.82-98.
32. Green, S. B., and T. Stutzman(1986), “An evaluation of methods to select respondents to structured job‐analysis questionnaires,” Personnel Psychology, 39(3), pp.543-564.
33. Graen, G. B., and M. Uhl-Bien(1995), “Relationshipbased approach to leadership: Development of leader-member exchange (LMX) theory of leadership over 25 years: Applying a multi-level multi-domain perspective,” The Leadership Quarterly, 6(2), pp.219-247.
34. Guenole, N., J. Ferrar, and S. Feinzig(2017), The Power of People: How Successful Organizations Use Workforce Analytics to Improve Business Performance, New York, NY, FT Press.
35. Hough, L. M., N. K. Eaton, M. D. Dunnette, J. D. Kamp, and R. A. McCloy(1990), “Criterionrelated validities of personality constructs and the effect of response distortion on those validities,” Journal of Applied Psychology, 75(5), pp.581-595.
36. Huang, J. L., N. A. Bowling, M. Liu, and Y. Li(2015), “Detecting insufficient effort responding with an infrequency scale: Evaluating validity and participant reactions,” Journal of Business and Psychology, 30(2), pp.299-311.
37. Huang, J. L., P. G. Curran, J. Keeney, E. M. Poposki, and R. P. DeShon(2012), “Detecting and deterring insufficient effort responding to surveys,” Journal of Business and Psychology, 27(1), pp.99-114.
38. Huang, J. L., M. Liu, and N. A. Bowling(2015), “Insufficient effort responding: Examining an insidious confound in survey data,” Journal of Applied Psychology, 100(3), pp.828-845.
39. Jackson, D. N.(1977), Jackson Vocational Interest Survey Manual, London, Canada, Research Psychologists.
40. Johnson, J. A.(2005), “Ascertaining the validity of individual protocols from Web-based personality inventories,” Journal of Research in Personality, 39(1), pp.103-129.
41. Kam, C. C. S., and J. P. Meyer(2015), “How careless responding and acquiescence response bias can influence construct dimensionality: The case of job satisfaction,” Organizational Research Methods, 18(3), pp.512-541.
42. Kim, D. S., C. J. McCabe, B. L. Yamasaki, K. A. Louie, and K. M. King(2018), “Detecting random responders with infrequency scales using an error-balancing threshold,” Behavior Research Methods, 50(5), pp.1960-1970.
43. Kline, R. B.(2015), Principles and Practice of Structural Equation Modeling, New York, NY, Guilford.
44. Krosnick, J. A.(1991), “Response strategies for coping with the cognitive demands of attitude measures in surveys,” Applied Cognitive Psychology, 5(3), pp.213-236.
45. LaRose, R., and H. Y. S. Tsai(2014), “Completion rates and non-response error in online surveys: Comparing sweepstakes and pre-paid cash incentives in studies of online behavior,” Computers in Human Behavior, 34, pp.110-119.
46. Little, T. D., W. A. Cunningham, G. Shahar, and K. F. Widaman(2002), “To parcel or not to parcel: Exploring the question, weighing the merits,” Structural Equation Modeling, 9(2), pp.151-173.
47. Liu, M., N. A. Bowling, J. L. Huang, and T. A. Kent(2013), “Insufficient effort responding to surveys as a threat to validity: The perceptions and practices of SIOP members,” The Industrial-Organizational Psychologist, 51(1), pp.32-38.
48. Mahalanobis, P. C.(1936), “On the generalized distance in statistics,” Proceedings of National Institute of Science of India, 12, pp.49-55.
49. Maniaci, M. R., and R. D. Rogge(2014), “Caring about carelessness: Participant inattention and its effects on research,” Journal of Research in Personality, 48, pp.61-83.
50. Maslach, C., and S. E. Jackson(1981), “The measurement of experienced burnout,” Journal of Occupational Behavior, 2(2), pp.99-113.
51. McGrath, R. E., M. Mitchell, B. H. Kim, and L. Hough(2010), “Evidence for response bias as a source of error variance in applied assessment,” Psychological Bulletin, 136(3), pp.450-470.
52. Meade, A. W., and S. B. Craig(2012), “Identifying careless responses in survey data,” Psychological Methods, 17(3), pp.437-455.
53. Niessen, A. S. M., R. R. Meijer, and J. N. Tendeiro (2016), “Detecting careless respondents in web-based questionnaires: Which method to use?,” Journal of Research in Personality, 63, pp.1-11.
54. Oppenheimer, D. M., T. Meyvis, and N. Davidenko (2009), “Instructional manipulation checks: Detecting satisficing to increase statistical power,” Journal of Experimental Social Psychology, 45(4), pp.867-872.
55. R Core Team(2018), R: A Language and Environment for Statistical Computing (Version 3.5.2) [Computer Software], R Foundation for Statistical Computing, Vienna, Austria, Available at http://www.R-project.org/.
56. Roivainen, E., J. Veijola, and J. Miettunen(2016), “Careless responses in survey data and the validity of a screening instrument,” Nordic Psychology, 68(2), pp.114-123.
57. Scandura, T. A., and G. B. Graen(1984), “Moderating effects of initial leader–member exchange status on the effects of a leadership intervention,” Journal of Applied Psychology, 69(3), pp.428-436.
58. Schmitt, N., and D. M. Stults(1985), “Factors defined by negatively keyed items: The result of careless respondents?,” Applied Psychological Measurement, 9(4), pp.367-373.
59. Schneider, S., M. May, and A. A. Stone(2018), “Careless responding in internet-based quality of life assessments,” Quality of Life Research, 27(4), pp.1077-1088.
60. Schonla, M., and V. Toepoel(2015), “Straight lining in Web survey panels over time,” Survey Research Methods, 9(2), pp.125-137.
61. Ward, M. K., and A. W. Meade(2018), “Applying social psychology to prevent careless responding during online surveys,” Applied Psychology, 67(2), pp.231-263.
62. Ward, M. K., A. W. Meade, C. M. Allred, G. Pappalardo, and J. W. Stoughton(2017), “Careless response and attrition as sources of bias in online survey assessments of personality traits and performance,” Computers in Human Behavior, 76, pp.417-430.
63. Ward, M. K., and S. B. Pond, Ⅲ.(2015), “Using virtual presence and survey instructions to minimize careless responding on Internetbased surveys,” Computers in Human Behavior, 48, pp.554-568.
64. Warwick, C., J. Rimmer, A. Blandford, J. Gow, and G. Buchanan(2009), “Cognitive economy and satisficing in information seeking: A longitudinal study of undergraduate information behavior,” Journal of the American Society for Information Science and Technology, 60 (12), pp.2402-2415.
65. Williams, L. J., and S. E. Anderson(1991), “Job satisfaction and organizational commitment as predictors of organizational citizenship and in-role behaviors,” Journal of Management, 17(3), pp.601-617.
66. Wood, R., and A. Bandura(1989), “Social cognitive theory of organizational management,” Academy of Management Review, 14(3), pp. 361-384.
67. Woods, C. M.(2006), “Careless responding to reverseworded items: Implications for confirmatory factor analysis,” Journal of Psychopathology and Behavioral Assessment, 28(3), pp.189-194.
68. Yentes, R. D., and F. Wilhelm(2018), careless: Procedures for Computing Indices of Careless Responding, R packages version 1.1.1, Available at https://github.com/ryentes/careless.

• 저자 박원우는 서울대 경영대학에서 학사 및 석사학위를, 그리고 미 Pittsburgh대학에서 경영학(인사조직) 박사학위(1989년)를 취득하였다. Pittsburgh대학에서 조교수로 근무한 후 귀국하여 중앙대와 경희대 교수를 거쳐 1998년부터 서울대에 재직 중이다. 학계에선 한국경영학회 부회장, 한국인사조직학회 부회장, 한국윤리경영학회 회장 등으로 봉사하였으며, 주요 연구분야는 groupthink, empowerment, trust, efficacy, goal orientation, culture change, 및 happiness인데, 그간 130여 편의 국내외 학술논문과 16편의 단행본 도서를 출간하였고, 서울대학교 경영대학의 우수강의상을 수차례, 2018년엔 서울대학교 교육상을 수상하였다.

• 저자 마성혁은 현재 서울대학교 경영대학 인사조직전공 박사과정에 재학 중이다. 연세대학교에서 경영학 학사학위를, 서울대학교에서 경영학 석사학위를 취득하였다. 주요 연구분야는 리더십, 직업과 소명의식, 일과 삶의 균형, 연구방법론 등이다.

• 저자 배수현은 현재 서울대학교 교육학 박사과정을 수료하였다. 뉴욕주립대학교 알바니에서 심리학과를 졸업하였으며, 서울대학교에서 교육학 석사학위를 취득하였다. 주요 연구분야는 직업교육, HRD 등이 있다.

• 저자 지선영은 현재 서울대학교 경영대학 석사과정에 재학 중이며 성균관대학교에서 경영학 학사학위를 취득하였다. 주요 관심분야는 의미있는 일(meaningful work), 조직문화, 연구방법론 등이다.

• 저자 이유우는 현재 서울대학교 농․산업교육과 박사과정을 수료하였으며 중앙대학교 대학원 경영학과 인사조직전공에서 석사학위를 취득하였다. 주요 연구분야는 연령관리, 경력 전환, 연구방법론 등이다.

• 저자 김자영은 현재 서울대학교 경영대학 전략․국제경영전공 박사과정에 재학 중이다. 연세대학교 실내건축학과를 졸업하였으며, 서울대학교 경영학 석사학위를 취득하였다. 주요 연구분야는 국제합작투자의 설계, 벤처기업의 성장 및 국제화 전략, 팀 및 기업의 창의성 등이다.

 
Mail : editor_kmr@daum.net
COPYRIGHT ⓒ 2011 Korean Academic Society Of Business Administration. All rights reserved