How to Teach Reproducibility in the Classroom

class: center, middle, inverse, title-slide

# How to Teach Reproducibility in the Classroom
## 2021 Conference on Teaching & Research in Economic Education
### Fernando Hoces de la Guardia, BITSS
### June 2021 | <a href="https://bitss.github.io/CTREE_slides/">slides</a>

---

background-image: url("Images/BITSSlogo.png"), url(Images/cega.png)
background-size: contain, 200px
background-position: 50% 100% , 100% 10% 
count:true

#[BITSS](https://bitss.org)

<style>

.center2 {
  margin: 0;
  position: absolute;
  top: 50%;
  left: 50%;
  -ms-transform: translate(-50%, -50%);
  transform: translate(-50%, -50%);
}

pre.sourceCode {
    max-height: 200px;
    overflow-y: auto;
}

/*
.remark-slide-number {
  position: inherit;
}

.remark-slide-number .progress-bar-container {
  position: absolute;
  bottom: 0;
  height: 4px;
  display: block;
  left: 0;
  right: 0;
}

.remark-slide-number .progress-bar {
 height: 100%;
 background-color: blue;
}
*/
</style>

</style>

.pull-left[
The Berkeley Initiative for Transparency in the Social Sciences works to improve the credibility of science by 
advancing transparency, reproducibility, rigor, and ethics in research and policy analysis.

.font130[**Core ACRe**]

Aleks Bogdanoski  
 
 Albert Chae

Fernando Hoces  de la Guardia
 
 Katie Hoeberling

Edward Miguel
 
Lars Vilhuber
 
]
.pull-right[ .right[
We are part of the Center for Effective Global Action ([CEGA](https://cega.berkeley.edu/)).

.font130[**Many Others**]

CEGA staff

Undergrad and Graduate RAs

Catalysts

Outside Collaborators   
(Researchers, Programmers)
  ]]

---

# Table of Contents

.font160[
1. [Motivation](#motivation)

2. [**A**ccelerating **C**omputational **Re**producibility: Framework](#acre)

3. [ACRe Resources for Teaching Reproducibility in the Classroom: Demo](#demo)

]

---
count:false
name:motivation
# Table of Contents

.font160[
1. [**Motivation**](#motivation)

2. [**A**ccelerating **C**omputational **Re**producibility: Framework](#acre)

3. [ACRe Resources for Teaching Reproducibility in the Classroom: Demo](#demo)

]

---

# .font100[Motivation 1: Low Replicability and Reproducibility]

.font180[
| Replication in Social Sciences (same method, different sample) 	| Reproduction in Economics (same data and methods) 	|
|-------------------------------------------------------------------	|------------------------------------------------------	|
| OSC ([2015](https://docs.google.com/document/d/1mm_4HZnEz_2Bh8XuiS2tpqCH08MFPyqUhi1baKPqR8Y/edit#heading=h.7vqf2cziid7z)): 30%-60% 	 | Chang & Li ([2015](https://www.nowpublishers.com/article/Details/CFR-0053)): 43% 	|
| Camerer et. al. ([2016](https://science.sciencemag.org/content/351/6280/1433)): ~60% 	| Gertler et. al. ([2017](https://www.nature.com/articles/d41586-018-02108-9)): 14% 	|
| Nosek & Camerer et. al. ([2018](https://www.nature.com/articles/s41562-018-0399-z)): ~60% 	| Kingi et. al. ([2018](https://hautahi.com/static/docs/Replication_aejae.pdf)): 43% 	|
| Klein et. al. ([2018](https://journals.sagepub.com/doi/10.1177/2515245918810225)): 50% 	| Wood et. al. ([2018](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0209416#abstract0)): 25% 	|
]

---

# .font90[Motivation 2: More Inclusive Concept Scholarly Output]     
.font120[
In addition to reproducibility problems, there are two additional motivations for this projects:   
]

.font120[
 <blockquote>
 Clarebout Principle:
 
 “An article about computational science in a scientific publication is not the scholarship itself, it’s merely scholarship advertisement. The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures.”
 
.right[ <cite> Buckheit and D.L. Donoho (1995, [2009](https://academiccommons.columbia.edu/doi/10.7916/D8QZ2MDS/download))</cite> ]
 </blockquote>
]

---

# .font90[Motivation 2: More Inclusive Concept Scholarly Output]

.font120[
In addition to reproducibility problems, there are two additional motivations for this projects:   
]

.font120[
 <blockquote>
 Clarebout Principle:
 
 “An article about computational science in a scientific publication is not the scholarship itself, it’s merely scholarship advertisement. The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures.”
 
.right[ <cite> Buckheit and D.L. Donoho (1995, [2009](https://academiccommons.columbia.edu/doi/10.7916/D8QZ2MDS/download))</cite> ]
 </blockquote>
]

.font120[
  .pull-left[
  
  Well discussed potential positive effects on:
  - Pedagogy  
  - Incremental generation of knowledge 
  ]
]
--
.font120[
  .pull-right[
  Under discussed:
  
  - Possible positive effect on diversity, equity and inclusion: no connections or language skills ("appropriate politeness") required to obtain materials 
  ]
]
???
An under discussed benefit of this principle is that increases access to knowledge for students who are outside of elite schools (no connections or "appropriate politeness" required to obtain materials).

---