class: center, middle, inverse, title-slide .title[ # Open Science practices and reproducible research in the classroom:A case study ] .author[ ### Joseph V. Casillas ] .institute[ ### Rutgers University ] .date[ ### LAGB 2022, Ulster University
Special session: Advances and challenges in teaching linguistics at university ] --- class: middle background-image: url(./docs/libs/img/flip.gif) background-size: contain background-position: 100% 50% # A lot of studies don't replicate ??? In this talk I am going to recount an experience I had teaching a graduate level methods course, but before I go into detail about the course and what I did, Id like to talk for just a second about the motivation for what I did. Over the past 10 years several fields have looked at the replicability of major findings This was a big deal in Psychology, and I'm sure that by now you have probably heard about it In short, they tested whether 100 influential findings could be replicated Take a second and think about that. How many findings would need to be replicated for you to have confidence in their merit? I work in SLA/bilingualism, but what about in your field? --- class: center middle .pull-left[ <img src="index_files/figure-html/donut-psych-1.png" width="1600" style="display: block; margin: auto;" /> ] .pull-right[ <br><br><br><br> # Replication crisis ### .Large[53%<br>did not replicate] ] .left[.footnote[OSC (2015)]] ??? 100 prominent papers analyzed, only 53% did not replicate --- class: middle <img src="index_files/figure-html/waffle-rest-1.png" width="4000" style="display: block; margin: auto;" /> ??? Not just psych --- class: middle background-image: url(https://raw.githubusercontent.com/jvcasillas/media/master/teaching/img/think.png) background-size: 500px background-position: 97% 50% .pull-left[ .Large[ - .Large[Small sample sizes] - .Large[QRPs] - .Large[p-hacking] - .Large[Harking] - .Large[Poor theory] - .Large[Lack of transparency] ] ] ??? These things are obviously all bad, but its unlikely they alone account for low replicability --- background-image: url(./docs/libs/img/osrrl.png) background-size: 375px background-position: 50% 90% # About the class ### Open Science and Reproducible Research in Linguistics ??? We department offers a standard research methods course and I was given the opportunity to teach in during my 3rd year in the program The course was a pretty standard methods course, based on my personal experience, which covered different methods, techniques, and designs in many of the subfields of linguistics I was given the green light to offer the course as I saw fit, and I decided I wanted to make the primary focus on learning open science methods Im going to briefly talk about some important details of the course - goals - length - levels - material --- background-image: url(./docs/libs/img/osf_dark.png) background-size: 170px background-position: 15% 30% # Goals ??? I wanted the course to be practical for students w/ limited programming exp. so they would finish feeling like more independent researchers and also be beneficial in the sense that they would walk away w/ something tangible, i.e. conf pres teach open science practices -- count: false background-image: url(./docs/libs/img/osf_dark.png), url(./docs/libs/img/tradition.png) background-size: 170px, 220px background-position: 15% 30%, 50% 30% ??? teach research methods in linguistics -- count: false background-image: url(./docs/libs/img/osf_dark.png), url(./docs/libs/img/tradition.png), url(./docs/libs/img/labtocat.png) background-size: 170px, 220px, 210px background-position: 15% 30%, 50% 30%, 85% 30% ??? teach project management from start to finish -- count: false background-image: url(./docs/libs/img/osf_dark.png), url(./docs/libs/img/tradition.png), url(./docs/libs/img/labtocat.png), url(./docs/libs/img/collaboration.png) background-size: 170px, 220px, 210px, 210px background-position: 15% 30%, 50% 30%, 85% 30%, 13% 86% ??? teach collaborative research/writing -- count: false background-image: url(./docs/libs/img/osf_dark.png), url(./docs/libs/img/tradition.png), url(./docs/libs/img/labtocat.png), url(./docs/libs/img/collaboration.png), url(./docs/libs/img/reviewer2.png) background-size: 170px, 220px, 210px, 210px, 220px background-position: 15% 30%, 50% 30%, 85% 30%, 13% 86%, 50% 90% ??? dealing with journals/editors/reviewers -- count: false background-image: url(./docs/libs/img/osf_dark.png), url(./docs/libs/img/tradition.png), url(./docs/libs/img/labtocat.png), url(./docs/libs/img/collaboration.png), url(./docs/libs/img/reviewer2.png), url(./docs/libs/img/elsevier.png) background-size: 170px, 220px, 210px, 210px, 220px, 200px background-position: 15% 30%, 50% 30%, 85% 30%, 13% 86%, 50% 90%, 85% 90% ??? submitting abstracts to journals --- background-image: url(./docs/libs/img/timo.png) background-size: contain ??? I believe replication studies are particularly beneficial for pedagogical purposes, a position advocated by Roettger and Baer-Henney (2019b) --- background-image: url(./docs/libs/img/gonzales_1.png), url(./docs/libs/img/gonzales_2.png) background-size: 650px, 650px background-position: 2% 5%, 2% 90% class: middle, right -- .pull-right[ # Conceptual replication with adult second language learners of Spanish ] ??? Before the semester began I decided to do a conceptual replication of a study in the journal Cognition, Gonzales, Byers-Heinlein, and Lotto (2019), extending it to a new pop. (adult L2 learners) --- # Material .pull-left[ ### OS .Large[ - Git/github - R/rstudio - Tidyverse - Psychopy - Prolific - Rmarkdown, papaja ] ] -- .pull-right[ ### Methods .Large[ - Production (delayed rep., picture naming, ultrasound) - Perception (2AFC, AXB) - Psycholinguistics (self-paced reading, eye-tracking) - Online experiments ] ] --- background-image: url(./docs/libs/img/schedule.png) background-size: contain ??? 16 week, standard semester gant chart with timeline of goals gant chart with actual realization of goals --- class: left # The students .Large[ | | | | :------------ | :---------------------------------- | | First year | .huge[👩🏽🎓👨🏽🎓👨🏽🎓👩🏽🎓] | | Second year | .huge[👩🏽🎓👩🎓] | | Third year\* | .huge[👩🏽🎓] | | Fourth year\* | .huge[🧑🏼🎓] | ] .footnote[\*Auditors, highly motivated] ??? - n = 8 - first year (n = 4) - second year (n = 2) - third year (n = 1)* - fourth year (n = 1)* --- class: title-slide-section-grey, middle # What happened? --- count: false background-image: url(./docs/libs/img/gonzales_1.png) background-size: 200px background-position: 5% 10% ??? We contacted the authors of the study we planned to replicate (ht Kalim Gonzales and @Krista_BH) and were able to get the original stimuli used in their experiment --- count: false background-image: url(./docs/libs/img/gonzales_1.png), url(./docs/libs/img/prereg.png), url(./docs/libs/img/osf_dark.png), url(./docs/libs/img/labtocat.png) background-size: 200px, 100px, 100px, 100px background-position: 5% 10%, 40% 10%, 50% 10%, 60% 10% ??? Together we went through the previous lit and then drafted a pre-registration using #github and #OSF. It was an ideal opportunity to learn the basics of #Git as well as some ins and outs of scientific collaboration. 🤗 Students learned about the preparation needed in the experimental design stages in order to complete a #prereg. We thought about our statistical analyses, power/sample size, stopping rules, etc. --- count: false background-image: url(./docs/libs/img/gonzales_1.png), url(./docs/libs/img/prereg.png), url(./docs/libs/img/osf_dark.png), url(./docs/libs/img/labtocat.png), url(./docs/libs/img/psychopy.png), url(./docs/libs/img/pavlovia.png) background-size: 200px, 100px, 100px, 100px, 230px, 130px background-position: 5% 10%, 40% 10%, 50% 10%, 60% 10%, 88% 10%, 97% 10% ??? We built the experiment using open source software, #PsychoPy3, and ran it online via #pavlovia and @gitlab --- count: false background-image: url(./docs/libs/img/gonzales_1.png), url(./docs/libs/img/prereg.png), url(./docs/libs/img/osf_dark.png), url(./docs/libs/img/labtocat.png), url(./docs/libs/img/psychopy.png), url(./docs/libs/img/pavlovia.png), url(./docs/libs/img/prolific.png) background-size: 200px, 100px, 100px, 100px, 230px, 130px, 200px background-position: 5% 10%, 40% 10%, 50% 10%, 60% 10%, 88% 10%, 97% 10%, 5% 50% ??? We collected a huge sample using the online platform prolific to recruit participants --- count: false background-image: url(./docs/libs/img/gonzales_1.png), url(./docs/libs/img/prereg.png), url(./docs/libs/img/osf_dark.png), url(./docs/libs/img/labtocat.png), url(./docs/libs/img/psychopy.png), url(./docs/libs/img/pavlovia.png), url(./docs/libs/img/prolific.png), url(./docs/libs/img/r.png), url(./docs/libs/img/rmd.png) background-size: 200px, 100px, 100px, 100px, 230px, 130px, 200px, 125px, 125px background-position: 5% 10%, 40% 10%, 50% 10%, 60% 10%, 88% 10%, 97% 10%, 5% 50%, 43% 50%, 57% 50% ??? We finished collecting the data halfway through the semester and carried out our statistical analysis. --- count: false background-image: url(./docs/libs/img/gonzales_1.png), url(./docs/libs/img/prereg.png), url(./docs/libs/img/osf_dark.png), url(./docs/libs/img/labtocat.png), url(./docs/libs/img/psychopy.png), url(./docs/libs/img/pavlovia.png), url(./docs/libs/img/prolific.png), url(./docs/libs/img/r.png), url(./docs/libs/img/rmd.png), url(./docs/libs/img/papaja.png) background-size: 200px, 100px, 100px, 100px, 230px, 130px, 200px, 125px, 125px, 125px background-position: 5% 10%, 40% 10%, 50% 10%, 60% 10%, 88% 10%, 97% 10%, 5% 50%, 43% 50%, 57% 50%, 88% 50% ??? We then began writing up a manuscript using papaja in Rmarkdown (ht @FrederikAust) --- count: false background-image: url(./docs/libs/img/gonzales_1.png), url(./docs/libs/img/prereg.png), url(./docs/libs/img/osf_dark.png), url(./docs/libs/img/labtocat.png), url(./docs/libs/img/psychopy.png), url(./docs/libs/img/pavlovia.png), url(./docs/libs/img/prolific.png), url(./docs/libs/img/r.png), url(./docs/libs/img/rmd.png), url(./docs/libs/img/papaja.png), url(./docs/libs/img/present.png) background-size: 200px, 100px, 100px, 100px, 230px, 130px, 200px, 125px, 125px, 125px, 225px background-position: 5% 10%, 40% 10%, 50% 10%, 60% 10%, 88% 10%, 97% 10%, 5% 50%, 43% 50%, 57% 50%, 88% 50%, 25% 90% ??? We were able to present preliminary results and got great feedback from @kleinschmidt and some of the members of his lab 🤜🏽🤛🏼 --- count: false background-image: url(./docs/libs/img/gonzales_1.png), url(./docs/libs/img/prereg.png), url(./docs/libs/img/osf_dark.png), url(./docs/libs/img/labtocat.png), url(./docs/libs/img/psychopy.png), url(./docs/libs/img/pavlovia.png), url(./docs/libs/img/prolific.png), url(./docs/libs/img/r.png), url(./docs/libs/img/rmd.png), url(./docs/libs/img/papaja.png), url(./docs/libs/img/present.png), url(./docs/libs/img/ssla.png), url(./docs/libs/img/psyarxiv.png) background-size: 200px, 100px, 100px, 100px, 230px, 130px, 200px, 125px, 125px, 125px, 225px, 85px, 170px background-position: 5% 10%, 40% 10%, 50% 10%, 60% 10%, 88% 10%, 97% 10%, 5% 50%, 43% 50%, 57% 50%, 88% 50%, 25% 90%, 69% 90%, 81% 90% ??? It was a bit tight, but we finished the manuscript by the end of the semester and submitted it for publication on the last day of class 😅 We also submitted abstracts to two conferences. The paper was under review for a little over a year 🤬, but we also uploaded a pre-print to @PsyArXiv and got some helpful feedback at a later stage (ht @TimoRoettger). In the meantime we presented at the two national conferences. For some of my co-authors this was the first conference presentation for their CVs. --- background-image: url(./docs/libs/img/ssla_pub.png), url(./docs/libs/img/elaine.gif) background-size: contain, 400px background-position: 50% 0%, 50% 90% ??? in June (2020) the paper was accepted for publication! --- class: title-slide-section-grey, middle # My assessment --- # Assessment ### The good .Large[ - Promote open science practices from early stage - Promote importance of replication studies - 2 national-level conference presentations - Peer-reviewed journal publication ] --- # Assessment ### More good #### Experience .large[ - collaborating - contacting external researchers - designing experiment (limited) - troubleshooting method - pre-registering study - dealing with data (acquiring, cleaning, analyzing, communicating) - drafting manuscript with literate programming - writing/submitting abstracts - submitting to journal - dealing with editors/reviewers - revising draft ] --- # Assessment ### Problems and lessons learned .Large[ - IRB - Teaching tough concepts (no way around this) - Getting first year students to contribute - Ambition ] --- # Materials .Large[ - [Course website](https://www.osrrl.jvcasillas.com/) - [OSF](https://osf.io/cp9bs/) - [Pre-registration](https://osf.io/qvjzy) - [Pre-print](https://psyarxiv.com/adkn8/) - [HLS 2019 presentation](https://www.rap-group.jvcasillas.com/dpbe_l2_replication/docs/slides/hls_2019/) - [CASPSLAP 2020 presntation](https://www.rap-group.jvcasillas.com/dpbe_l2_replication/docs/slides/caspslap_2020/) - [Published manuscript (2020)](https://doi.org/10.1017/S0272263120000273) ] --- count: false class: inverse, middle <blockquote align='center' class="twitter-tweet" data-lang="de"> <a href="https://twitter.com/jvcasill/status/1291061282158903303?ref_src=twsrc%5Etfw"></a> </blockquote> --- # The students (my co-authors) .Large[ - [Cristina Lozano-Argüelles](https://crislozano.me) - [Laura Fernández Arroyo](https://de.linkedin.com/in/laura-fernandez-arroyo) - [Nicole Rodríguez]() - [Ezequiel Durand López](https://edurandlopez.wordpress.com) - [Juan José Garrido Pozú](https://juanjgarridop.github.io) - [Jennifer Markovits](https://span-port.rutgers.edu/people/ph-d-students-tas/graduate-students/129-current-students-phd-in-bilingualismsla/973-jennifer-markovits) - [Jessica Varela](https://www.researchgate.net/profile/Jessica-Varela-2) - [Núria de Rocafiguera](https://www.researchgate.net/profile/Nuria-De-Rocafiguera) ] --- class: inverse # New, relevant tools ## ".RUred[The tools that we used in class helped<br>me to complete my dissertation study.]" -- ## ".RUred[The material can be applied to <br>academic life even after the semester<br> ends.]" -- background-image: url(https://media.giphy.com/media/3o7aTwT5vjBVfsYEj6/giphy.gif) background-size: 350px background-position: 95% 50% ## ".RUred[We are left the class with a few more <br>tools in our toolbox.]" --- count: false # First contact with open-source software </br></br></br> .content-box-blue[ # .grey[I had no idea there were free, user-friendly, accessible programs like PsychoPy that could run experiments that I had been previously run on paid (and expensive) programs.] ] --- count: false # Assessing lit .pull-left[ .Large[ .content-box-green[ The course was also important in making me think critically about the articles I read now that I am aware of the reproducibility crisis - it makes me look for the pre-registered reports and the availability of code/data/etc. ] ] ] .pull-right[ .Large[ .content-box-red[ Many common assumptions in fields other than linguistics (e.g., psych) were false / the experiments behind them were plagued with type I/II errors. ] ] ] --- count: false class: inverse # .RUred[Personal growth, independence] .pull-left[ ### "[We] grew a bit as researchers and will be able to use 21st century tools to facilitate our future investigations." ] -- <br><br><br><br><br><br> .pull-right[ ### "I got scared that my experiments would not have sufficient statistical power, and that I would be one of those poor guys creating a whole theory based on type 1 / 2 errors. But that fear soon turned into active work to prevent that from happening (sort of)." ] --- count: false class: title-slide-section-red # .black[What was lacking] ### "Cover one online method every class." -- ### "I would add one or two classes/workshops about the real importance<br> of meta-analysis and what info is needed to perform them <br> (maybe how to do them too)." -- ### "I would include one or two whole classes in which we study a particular<br> theory that was based entirely on false assumptions or type 1 or 2 errors.<br> Why that happened, how the authors got to that point, whether they<br> withdrew the published papers, how it would be best to proceed, etc. It can<br> be from other fields to prevent people from thinking that you're against<br> those authors." --- count: false # What (else) was lacking .pull-left[ ### Give a research skills/experience survey at the beginning of the course and then create groups with different levels based on the responses. [...] Students with more experience can tutor students with less experience. Thus, higher-level students will develop skills to teach how to conduct research. ] -- .pull-right[ .content-box-blue[ ### The experience was conducted too fast ] .content-box-blue[ ### It is necessary to improve the pedagogic strategies (of the professor) ] ] --- # Final thoughts and suggestions .Large[ - Plan ahead (irb, journal) - Have a plan b (c, d, and e) - Make important tasks group work - Pair less experienced students with more experienced students - Have platform for backup, questions, problems, change log - Make expectations very clear - Communicate risk ] -- .Large[ - DO IT! ] --- exclude: true Starns, Cataldo, Rotello, Annis, Aschenbrenner, Bröder, Cox, Criss, Curl, Dobbins, and others (2019) Camerer, Dreber, Holzmeister, Ho, Huber, Johannesson, Kirchler, Nave, Nosek, Pfeiffer, and others (2018) Errington, Mathur, Soderberg, Denis, Perfito, Iorns, and Nosek (2021) --- count: false class: title-slide-final background-image: url(https://github.com/jvcasillas/ru_xaringan/raw/master/img/logo/ru_shield.png), url(./docs/libs/img/qr.png), url(./docs/libs/img/osrrl.png) background-size: 120px, 160px, 170px background-position: 20% 45%, 50% 45%, 80% 45% <br> # Thank you <br><br><br><br><br><br><br><br><br><br> .Large[ | | | | ----------------------------------: | :-------------------------------------- | |
| .lightgrey[joseph.casillas@rutgers.edu] | |
| .lightgrey[@jvcasill] | |
| .lightgrey[www.jvcasillas.com] | ] --- count: false # References Camerer, C. F., A. Dreber, F. Holzmeister, et al. (2018). "Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015". In: _Nature Human Behaviour_ 2.9, pp. 637-644. DOI: [10.1038/s41562-018-0399-z](https://doi.org/10.1038%2Fs41562-018-0399-z). Errington, T. M., M. Mathur, C. K. Soderberg, et al. (2021). "Investigating the replicability of preclinical cancer biology". In: _eLife_ 10. Ed. by R. Pasqualini and E. Franco, p. e71601. ISSN: 2050-084X. DOI: [10.7554/eLife.71601](https://doi.org/10.7554%2FeLife.71601). URL: [https://doi.org/10.7554/eLife.71601](https://doi.org/10.7554/eLife.71601). Gonzales, K., K. Byers-Heinlein, and A. J. Lotto (2019). "How Bilinguals Perceive Speech Depends on Which Language They Think They’re Hearing". In: _Cognition_ 182, pp. 318-330. DOI: [10.1016/j.cognition.2018.08.021](https://doi.org/10.1016%2Fj.cognition.2018.08.021). Roettger, T. B. and D. Baer-Henney (2019b). "Toward a replication culture: Speech production research in the classroom". In: _Phonological Data and Analysis_ 1.4, pp. 1-23. Starns, J. J., A. M. Cataldo, C. M. Rotello, et al. (2019). "Assessing theoretical conclusions with blinded inference to investigate a potential inference crisis". In: _Advances in Methods and Practices in Psychological Science_ 2.4, pp. 335-349. <style type="text/css"> .title-slide { background-image: url(https://github.com/jvcasillas/ru_xaringan/raw/master/img/logo/ru_shield.png), url(./docs/libs/img/lagb.png); background-position: 9% 15%, 91% 17%; background-size: 55px, 200px; background-color: #fff; padding-left: 100px; } /* H1 fonts */ .title-slide h1 { color: #cc0033; padding-top: 250px; font-weight: normal; font-size: 45px; text-align: left; text-shadow: none; padding-bottom: 18px; margin-bottom: 18px; } .remark-slide table { margin: auto; border: 0px none; border-collapse: collapse; align: left; } .remark-slide table thead th { border: 0px none; background-color: none; } th, td { padding: 5px; border: 0px none; background-color: none; } .remark-slide thead, .remark-slide tfoot, .remark-slide tr:nth-child(even) { border: 0px none; background: none; } .big-text p { font-size: 4em; line-height: 0.3; } .h-center { margin: 0 auto; } .h-right { float: right; } .w-45 { width: 45%; } .w-75 { width: 75%; } .w-100 { width: 100%; } .huge { font-size: 2.8em; } </style>