BazEkon - Biblioteka Główna Uniwersytetu Ekonomicznego w Krakowie

BazEkon home page

Meny główne

Autor
Wanat Tomasz (Poznań University of Economics and Business)
Tytuł
Wpływ eliminacji odpowiedzi nieuważnych respondentów na replikację wyników badań w naukach społecznych
The impact of eliminating careless responses and outliers on the replication of research findings in social sciences
Źródło
Ruch Prawniczy, Ekonomiczny i Socjologiczny, 2024, nr 4, s. 251-271, bibliogr. 63 poz.
Słowa kluczowe
Badania naukowe, Metody badawcze, Nauki społeczne
Scientific research, Research methods, Social sciences
Uwagi
streszcz., summ.
Abstrakt
Znaczna część badań naukowych jest trudna lub nawet niemożliwa do replikowania lub odtworzenia, co określane jest mianem kryzysu replikacji. Jednym z czynników przyczyniających się do tego kryzysu jest niska jakość danych wykorzystywanych w badaniach. Często można to przypisać nieuważnym lub nietypowym respondentom. Eliminacja danych z tych grup może poprawić jakość danych badawczych i potencjalnie zwiększyć prawdopodobieństwo udanej replikacji. Eliminacja takich danych może czasami mieć skutek odwrotny. Metody wykrywania i usuwania nieuważnych i nietypowych respondentów różnią się znacznie, dlatego też dają różne wyniki i mogą być stosowane na wiele sposobów, dodając kolejny poziom złożoności w kontekście replikacji. Głównym celem artykułu jest wskazanie na zagrożenie tkwiące w posługiwaniu się różnymi metodami wykrywania nieuważnych i nietypowych odpowiedzi dla możliwości odtworzenia wyników badania. Artykuł podzielony jest na dwie części. W pierwszej omówiono zagadnienia związane ze źródłami kryzysu replikacji w naukach społecznych i potencjalnego wpływu metod wykrywania nieuważnych odpowiedzi respondentów na możliwości replikowania badań. W drugiej części, na podstawie analizy przypadku jednego z badań zamieszczonych w systemie Open Science Framework (OSF), pokazano, jak subtelny, a zarazem znaczący może być wpływ zastosowanych metod wykrywania i usuwania nieuważnych i nietypowych respondentów na powodzenie replikacji badań. W końcowej części artykułu wskazano na kroki mające na celu ograniczenie problemu z replikacją związaną z wykorzystaniem metod wykrywania nieuważnych i nietypowych respondentów. (abstrakt oryginalny)

Much of scientific research is difficult or even impossible to replicate or reproduce, a phenomenon known as the replication crisis. One contributing factor to this crisis is the poor quality of the data used in research. This can often be attributed to inattentive or atypical respondents. By eliminating data from these groups, the quality of the research data might improve, potentially increasing the likelihood of successful replication. However, this approach can also contribute to the replication crisis. The methods for detecting and removing inattentive and atypical respondents vary significantly, produce different outcomes, and can be applied in numerous ways - adding another layer of complexity to the replication challenge. The main purpose of the article is to point out the risks inherent in using different methods for detecting inattentive and atypical responses in relation to the replicability of survey results. The article is divided into two parts. The first discusses issues related to the sources of the replication crisis in the social sciences and the potential impact of methods for detecting inattentive responses on research replicability. In the second part, based on a case study of one of the surveys posted on Open Science Framework (OSF), the article demonstrates how subtle yet significant the impact of the methods used to detect and remove inattentive and atypical respondents can be on the success of survey replication. The final section identifies steps to reduce the replication problem associated with the use of methods to detect inattentive and atypical responses. (original abstract)
Dostępne w
Biblioteka Główna Uniwersytetu Ekonomicznego w Katowicach
Biblioteka Główna Uniwersytetu Ekonomicznego w Poznaniu
Pełny tekst
Pokaż
Bibliografia
Pokaż
  1. Agnoli, F., Wicherts, J. M., Veldkamp, C. L., Albiero, P., i Cubelli, R. (2017). Questionable research practices among Italian research psychologists. PloS ONE, 12(3), e0172792. DOI: https://doi.org/10.1371/journal.pone.0172792
  2. Aguinis, H., Ramani, R. S., i Alabduljader, N. (2018). What you see is what you get? Enhancing methodological transparency in management research. Academy of Management Annals, 12(1), 83-110. DOI: https://doi.org/10.5465/annals.2016.0011
  3. Amrhein, V., Trafimow, D., i Greenland, S. (2019). Inferential statistics as descriptive statistics: There is no replication crisis if we don't expect replication. The American Statistician, 73(sup1), 262-270. DOI: https://doi.org/10.1080/00031305.2018.1543137
  4. Andrade, C. (2021). HARKing, cherry-picking, p-hacking, fishing expeditions, and data dredging and mining as questionable research practices. The Journal of Clinical Psychiatry, 82(1), e1-e3. DOI: https://doi.org/10.4088/JCP.20f13804
  5. Baillie, M., Le Cessie, S., Schmidt, C. O., Lusa, L., Huebner, M., for the Topic Group "Initial Data Analysis" of the STRATOS Initiative. (2022). Ten simple rules for initial data analysis. PLoS Computational Biology, 18(2), e1009819. DOI: https://doi.org/10.1371/journal.pcbi.1009819
  6. Bakker, B. N., Jaidka, K., Dörr, T., Fasching, N., i Lelkes, Y. (2021). Questionable and open research practices: Attitudes and perceptions among quantitative communication researchers. Journal of Communication, 71(5), 715-738. DOI: https://doi.org/10.1093/joc/jqab031
  7. Banks, G. C., O'Boyle, E. H., Pollack, J. M., White, C. D., Batchelor, J. H., Whelpley, C. E., Abston, K. A., Bennett, A. A., i Adkins, C. L. (2016). Questions about questionable research practices in the field of management: A guest commentary. Journal of Management, 42(1), 5-20. DOI: https://doi.org/10.1177/0149206315619011
  8. Benjamini, Y. (2020). Selective inference: The silent killer of replicability. Harvard Data Science Review, 2(4). DOI: https://doi.org/10.1162/99608f92.fc62b261
  9. Białek, A., i Wolski, P. (2023). Dwa głosy o kryzysie wiarygodności w psychologii. Przegląd Psychologiczny, 66(1), 9-26. DOI: https://doi.org/10.31648/przegldpsychologiczny.9455
  10. Bridges, A. J. (2022). Hypothesizing after results are known: HARKing. W W. O'Donohue, A. Masuda i S. Lilienfeld (red.), Avoiding questionable research practices in applied psychology (s. 175-190). Springer International Publishing. DOI: https://doi.org/10.1007/978-3-031-04968-2_8
  11. Brzeziński, J. M. (2023). Czy kryzys wiarygodności w psychologii? Przegląd Psychologiczny, 66(1), 27-47. DOI: https://doi.org/10.31648/przegldpsychologiczny.9456
  12. Bybee, S., Cloyes, K., Baucom, B., Supiano, K., Mooney, K., i Ellington, L. (2022). Bots and nots: Safeguarding online survey research with underrepresented and diverse populations. Psychology Sexuality, 13(4), 901-911. DOI: https://doi.org/10.1080/19419899.2021.1936617
  13. Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., Altmejd, A., Buttrick, N., Chan, T., Chen, Y., Forsell, E., Gampa, A., Heikensten, E., Hummer, L., Imai, T., ... Wu, H. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2(9), 637-644. DOI: https://doi.org/10.1038/s41562-018-0399-z
  14. Cockburn, A., Dragicevic, P., Besançon, L., i Gutwin, C. (2020). Threats of a replication crisis in empirical computer science. Communications of the ACM, 63(8), 70-79. DOI: https://doi.org/10.1145/3360311
  15. Coiera, E., Ammenwerth, E., Georgiou, A., i Magrabi, F. (2018). Does health informatics have a replication crisis? Journal of the American Medical Informatics Association, 25(8), 963-968. DOI: https://doi.org/10.1093/jamia/ocy028
  16. Cornell, D., Klein, J., Konold, T., i Huang, F. (2012). Effects of validity screening items on adolescent survey data. Psychological Assessment, 24(1), 21-35. DOI: https://doi.org/10.1037/a0024824
  17. Credé, M. (2010). Random responding as a threat to the validity of effect size estimates in correlational research. Educational and Psychological Measurement, 70(4), 596-612. DOI: https://doi.org/10.1177/0013164410366686
  18. Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4-19. DOI: https://doi.org/10.1016/j.jesp.2015.07.006
  19. DeSimone, J. A., i Harms, P. D. (2018). Dirty data: The effects of screening respondents who provide low-quality data in survey research. Journal of Business and Psychology, 33(5), 559-577. DOI: https://doi.org/10.1007/s10869-017-9514-9
  20. Dunn, A. M., Heggestad, E. D., Shanock, L. R., i Theilgard, N. (2018). Intra-individual response variability as an indicator of insufficient effort responding: Comparison to other indicators and relationships with individual differences. Journal of Business and Psychology, 33(1), 105-121. DOI: https://doi.org/10.1007/s10869-016-9479-0
  21. Duvendack, M., Palmer-Jones, R., i Reed, W. R. (2017). What is meant by "replication" and why does it encounter resistance in economics? American Economic Review, 107(5), 46-51. DOI: https://doi.org/10.1257/aer.p20171031
  22. Ellis, R. J. (2022). Questionable research practices, low statistical power, and other obstacles to replicability: Why preclinical neuroscience research would benefit from registered reports. Eneuro, 9(4), ENEURO.0017-22.2022. DOI: https://doi.org/10.1523/ENEURO.0017-22.2022
  23. Fox, C. R., i Tversky, A. (1995). Ambiguity aversion and comparative ignorance. The Quarterly Journal of Economics, 110(3), 585-603. DOI: https://doi.org/10.2307/2946693
  24. Franco, A., Malhotra, N., i Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502-1505. DOI: https://doi.org/10.1126/science.1255484
  25. Franco, A., Malhotra, N., i Simonovits, G. (2016). Underreporting in psychology experiments: Evidence from a study registry. Social Psychological and Personality Science, 7(1), 8-12. DOI: https://doi.org/10.1177/1948550615598377
  26. Freese, J., i Peterson, D. (2017). Replication in social science. Annual Review of Sociology, 43(1), 147-165. DOI: https://doi.org/10.1146/annurev-soc-060116-053450
  27. Hayes, A. F. (2018). Introduction to mediation: A regression-based approach. Guilford Press.
  28. He, J., Van de Vijver, F. J., Espinosa, A. D., i Mui, P. H. (2014). Toward a unification of acquiescent, extreme, and midpoint response styles: A multilevel study. International Journal of Cross Cultural Management, 14(3), 306-322. DOI: https://doi.org/10.1177/1470595814541424
  29. Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., i DeShon, R. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27(1), 99-114. DOI: https://doi.org/10.1007/s10869-011-9231-8
  30. Hudson, R. (2023). Explicating exact versus conceptual replication. Erkenntnis, 88(6), 2493-2514. DOI: https://doi.org/10.1007/s10670-021-00464-z
  31. Ioannidis, J. P. (2008). Why most discovered true associations are inflated. Epidemiology, 19, 640-648. DOI: https://doi.org/10.1097/EDE.0b013e31818131e7
  32. Kam, C. C. S., Meyer, J. P. (2015). How careless responding and acquiescence response bias can influence construct dimensionality: The case of job satisfaction. Organizational Research Methods, 18(3), 512-541. DOI: https://doi.org/10.1177/1094428115571894
  33. Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196-217. DOI: https://doi.org/10.1207/s15327957pspr0203_4
  34. Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5(3), 213-236. DOI: https://doi.org/10.1002/acp.2350050305
  35. Lakens, D. (2015). On the challenges of drawing conclusions from p-values just below 0.05. PeerJ, 3, e1142. https://peerj.com/articles/1142/ DOI: https://doi.org/10.7717/peerj.1142
  36. Lakens, D., i Evers, E. R. K. (2014). Sailing from the seas of chaos into the corridor of stability: Practical recommendations to increase the informational value of studies. Perspectives on Psychological Science, 9(3), 278-292. DOI: https://doi.org/10.1177/1745691614528520
  37. Leichtmann, B., Nitsch, V., i Mara, M. (2022). Crisis ahead? Why human-robot interaction user studies may have replicability problems and directions for improvement. Frontiers in Robotics and AI, 9, 838116. DOI: https://doi.org/10.3389/frobt.2022.838116
  38. Leys, C., Delacre, M., Mora, Y. L., Lakens, D., i Ley, C. (2019). How to classify, detect, and manage univariate and multivariate outliers, with emphasis on pre-registration. International Review of Social Psychology, 32(1, Article 5. DOI: https://doi.org/10.5334/irsp.289
  39. Lynch Jr, J. G., Bradlow, E. T., Huber, J. C., i Lehmann, D. R. (2015). Reflections on the replication corner: In praise of conceptual replications. International Journal of Research in Marketing, 32(4), 333-342. DOI: https://doi.org/10.1016/j.ijresmar.2015.09.006
  40. Maniaci, M. R., i Rogge, R. D. (2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48, 61-83. Maxwell, S. E., Lau, M. Y., i Howard, G. S. (2015). Is psychology suffering from a replication crisis? What does "failure to replicate" really mean? American Psychologist, 70(6), 487-498. DOI: https://doi.org/10.1037/a0039400
  41. Meade, A. W., i Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437-455. DOI: https://doi.org/10.1037/a0028085
  42. Muradchanian, J., Hoekstra, R., Kiers, H., i van Ravenzwaaij, D. (2023). The role of results in deciding to publish: A direct comparison across authors, reviewers, and editors based on an online survey. PloS ONE, 18(10), e0292279. DOI: https://doi.org/10.1371/journal.pone.0292279
  43. Muthukrishna, M., Bell, A. V., Henrich, J., Curtin, C. M., Gedranovich, A., McInerney, J., i Thue, B. (2020). Beyond Western, educated, industrial, rich, and democratic (WEIRD) psychology: Measuring and mapping scales of cultural and psychological distance. Psychological Science, 31(6), 678-701. DOI: https://doi.org/10.1177/0956797620916782
  44. Niessen, A. S. M., Meijer, R. R., i Tendeiro, J. N. (2016). Detecting careless respondents in webbased questionnaires: Which method to use? Journal of Research in Personality, 63, 1-11. DOI: https://doi.org/10.1016/j.jrp.2016.04.010
  45. Nosek, B. A., i Errington, T. M. (2020). What is replication? PLoS Biology, 18(3), e3000691. DOI: https://doi.org/10.1371/journal.pbio.3000691
  46. Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., Rohrer, J. M., Romero, F., Scheel, A. M., Scherer, L. D., Schönbrodt, F. D., i Vazire, S. (2022). Replicability, robustness, and reproducibility in psychological science. Annual Review of Psychology, 73(1), 719-748. DOI: https://doi.org/10.1146/annurev-psych-020821-114157
  47. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. DOI: https://doi.org/10.1126/science.aac4716
  48. Oppenheimer, D. M., Meyvis, T., i Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867-872. DOI: https://doi.org/10.1016/j.jesp.2009.03.009
  49. Pagell, M. (2021). Replication without repeating ourselves: Addressing the replication crisis in operations and supply chain management research. Journal of Operations Management, 67(1), 105-115. DOI: https://doi.org/10.1002/joom.1120
  50. Peels, R., i Bouter, L. (2018). The possibility and desirability of replication in the humanities. Palgrave Communications, 4(1), 95. DOI: https://doi.org/10.1057/s41599-018-0149-x
  51. Pfundmair, M, i Lermer, E. (2023) Examining the link between social exclusion and social-risk taking: A correlational and experimental investigation [version 1; peer review: 1 approved with reservations], Routledge Open Research, 2023, 2:4 Last updated: 28 Mar 2023. https://osf.io/umrht DOI: https://doi.org/10.12688/routledgeopenres.17729.1
  52. Ravn, T., i Sørensen, M. P. (2021). Exploring the gray area: Similarities and differences in questionable research practices (QRPs) across main areas of research. Science and Engineering Ethics, 27(4), 40. DOI: https://doi.org/10.1007/s11948-021-00310-z
  53. Richard, F. D., Bond, C. F., i Stokes-Zoota, J. J. (2003). One hundred years of social psychology quantitatively described. Review of General Psychology, 7(4), 331-363. Saad, D. (2021). Nowe narzędzia i techniki zwiększające trafność badań internetowych. Com.Press, 4(1), 106-121. DOI: https://doi.org/10.1037/1089-2680.7.4.331
  54. Scheel, A. M., Schijen, M. R. M. J., i Lakens, D. (2021). An excess of positive results: Comparing the standard psychology literature with registered reports. Advances in Methods and Practices in Psychological Science, 4(2), 1-12. DOI: https://doi.org/10.1177/25152459211007467
  55. Stodden, V. (2015). Reproducing Statistical Results. Annual Review of Statistics and Its Application, 2(1), 1-19. DOI: https://doi.org/10.1146/annurev-statistics-010814-020127
  56. Świątkowski, W., i Dompnier, B. (2017). Replicability crisis in social psychology: Looking at the past to find new pathways for the future. International Review of Social Psychology, 30(1), 111-124. DOI: https://doi.org/10.5334/irsp.66
  57. Tincani, M., i Travers, J. (2019). Replication research, publication bias, and applied behavior analysis. Perspectives on Behavior Science, 42(1), 59-75. DOI: https://doi.org/10.1007/s40614-019-00191-5
  58. Vecchio, R., Caso, G., Cembalo, L., i Borrello, M. (2020). Is respondents' inattention in online surveys a major issue for research? Economia Agro-Alimentare, 1, 1-18. DOI: https://doi.org/10.3280/ecag1-2020oa10069
  59. Vow els, M. J. (2021). Misspecification and unreliable interpretations in psychology and social science. Psychological Methods, 28, 507-526. DOI: https://doi.org/10.1037/met0000429
  60. Wanat, T. (2010). Niedoskonalości w formulowaniu hipotez badawczych w pracach doktorskich. Zagadnienia Naukoznawstwa, 46(1), 27-41.
  61. Ward, M., i Meade, A. W. (2023). Dealing with careless responding in survey data: Prevention, identification, and recommended best practices. Annual Review of Psychology, 74, 577-596. DOI: https://doi.org/10.1146/annurev-psych-040422-045007
  62. Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., Van Aert, R. C. M., i Van Assen, M. A. L. M. (2016). Degrees of freedom in planning, running, analyzing, and reporting psychological studies: A checklist to avoid p-hacking. Frontiers in Psychology, 7, 1832. DOI: https://doi.org/10.3389/fpsyg.2016.01832
  63. Yang, S., Wang, L., Ding, P. (2019). Causal inference with confounders missing not at random. Biometrika, 106(4), 875-888. DOI: https://doi.org/10.1093/biomet/asz048
Cytowane przez
Pokaż
ISSN
0035-9629
Język
pol
URI / DOI
https://doi.org/10.14746/rpeis.2024.86.4.14
Udostępnij na Facebooku Udostępnij na Twitterze Udostępnij na Google+ Udostępnij na Pinterest Udostępnij na LinkedIn Wyślij znajomemu