Transparencia en Investigación: Problemas y Soluciones

class: center, middle, inverse, title-slide

# Transparencia en Investigación: Problemas y Soluciones
## .font80[Ciclo de Seminarios Económicos, Universidad de Talca]
### Fernando Hoces, Berkeley Initiative for Transparency in the Social Sciences
### 15 10 2020 | <a href="https://github.com/BITSS/slides-utalca-2020">diapositivas</a>

---

count: false
<style>

.center2 {
  margin: 0;
  position: absolute;
  top: 50%;
  left: 50%;
  -ms-transform: translate(-50%, -50%);
  transform: translate(-50%, -50%);
}

pre.sourceCode {
    max-height: 200px;
    overflow-y: auto;
}

/*
.remark-slide-number {
  position: inherit;
}

.remark-slide-number .progress-bar-container {
  position: absolute;
  bottom: 0;
  height: 4px;
  display: block;
  left: 0;
  right: 0;
}

.remark-slide-number .progress-bar {
 height: 100%;
 background-color: blue;
}
*/
</style>

</style>

# Contenidos

.font130[

1. [BITSS](#about-bitss)

2. [Transparencia en la Investigación Científica](#transparencia)

3. [Problemas y Soluciones](#ADD)

4. [Aplicación al Análisis de Políticas Públicas](#ADD)
]

---
count: false
# Contenidos

.font130[

1. [**BITSS**](#about-bitss)

2. [Transparencia en la Investigación Científica](#ADD)

3. [Problemas y Soluciones](#ADD)

4. [Aplicación al Análisis de Políticas Públicas](#ADD)
]

---
background-image: url("Images/BITSSlogo.png"), url(Images/cega.png)
background-size: contain, 200px
background-position: 50% 100% , 0% 100% 
name: about-bitss
# Sobre Nosotros
  
  
### [BITSS](https://bitss.org)

.pull-left[
Berkeley Initiative for Transparency in the Social Sciences trabaja para mejorar la credibilidad de las ciencias al promover transparencia, reproducibilidad, rigor, y ética en la investigación

.font150[ ACRE

OPA

]

]
.pull-right[ .right[
Somos parte del Center for Effective Global Action ([CEGA](https://cega.berkeley.edu/)).

.font150[   
Talleres y   
Conferencias

Meta-investigación
]

]]

---
count: false
# Contenidos

.font130[

1. [BITSS](#about-bitss)

2. [**Transparencia en la Investigación Científica**](#transparencia)

3. [Problemas y Soluciones](#ADD)

4. [Aplicación al Análisis de Políticas Públicas](#ADD)
]

---
name: transparencia

# Ética en la Investigación Científica

.font120[
- Transparencia es un elemento central de la ética del investigador.

- Valores científicos acuñados por Robert Merton (Merton 1942):

- **Universalismo**: cualquier persona puede presentar un argumento, independiente de su estatus.  
  - **Comunismo/Comunalismo**: el conocimiento es compartido de manera abierta.  
  - **Desinterés**: la verdad como motivación, y no los beneficios monetarios.  
  - **Escepticismo Organizado**: revisión a través de pares (peer review), replicación.
]
---
background-image: url(Images/AMdV2007_1.PNG)
background-size: contain

# En la Practica [(Anderson et al 2007)](http://www.jstor.org/stable/pdf/10.1525/jer.2007.2.4.3.pdf)

---
background-image: url(Images/AMdV2007_2.PNG)
background-size: contain
count: false

# En la Practica [(Anderson et al 2007)](http://www.jstor.org/stable/pdf/10.1525/jer.2007.2.4.3.pdf)  
---
background-image: url(Images/AMdV2007.PNG)
background-size: contain
count: false

# En la Practica [(Anderson et al 2007)](http://www.jstor.org/stable/pdf/10.1525/jer.2007.2.4.3.pdf)

---
count: false
# Contenidos

.font130[

1. [BITSS](#about-bitss)

2. [Transparencia en la Investigación Científica](#transparencia)

3. [**Problemas y Soluciones**](#problemas)

4. [Aplicación al Análisis de Políticas Públicas](#ADD)
]

---
name: problemas

# Problema #1: Sesgo de Publicación

El sesgo de publicacion ocurre cuando los estudios publicados en revistas cientificas estan sobrerepresentados por estudios que obtienen un particular tipo de restultados (eg. rechazan la hipotesis nula).

Evidencia que sugiere la existencia de sesgo de publicacion:

- El tamaño de los efectos disminuye con el tamaño muestral ([Gerber et al  2001](http://pan.oxfordjournals.org/content/9/4/385.short)).

- La publicación de efectos nulos esta desapareciendo en el tiempo, en todas las disciplinas ([Fanelli 2011](http://link.springer.com/article/10.1007/s11192-011-0494-7)).

Evidencia que mide la magnitud del sesgo de publicación:

- Estudio que  siguió a experimentos completados muestra que aquellos experimentos con fuertes resultados son 40pp más probable de ser publicados, y 60pp más probable de ser escritos. Alto "file drawer problem". ([Franco et al 2014](http://science.sciencemag.org/content/345/6203/1502))

- En economía [Andrews and Kasy (2019)](https://www.aeaweb.org/articles?id=10.1257/aer.20180310) estiman que, para algunas literaturas, los estudios que rechazan la nula son entre 3 y 30 (!) veces mas probables de ser publicados en journals top.   
  
---
background-image: url(Images/Tess.PNG)
background-size: contain

# Sesgo de Publicacion en TESS (NSF)   
---

# Problema #2: P-Hacking  
.font120[
- Definición: flexibilidad en el análisis de datos permite presentar *casi cualquier resultado* bajo un umbral arbitrario; significancia estadística pierde sentido.

- Otros nombres: "specification searching" ([Leamer 1983](http://www.econ.ucla.edu/workingpapers/wp239.pdf)), "data-fishing", grados de libertad del investigador, o "data-mining".

- No implica intencionalidad. Puede ser subconsciente, o simplemente una practica estándar del análisis estadístico ([Gelman and Loken 2013](http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf)).

- Evidencia: comportamiento anomalo de test estadisticos entorno a umbrales arbitrarios.
]
---
background-image: url(Images/GerberSoc.PNG), url(Images/GerberPS.PNG)
background-size: 400px, 500px
background-position: 0% 50%, 100% 50%

# Evidencia: Sociologia y Ciencias Politicas
.pull-left[
Sociología [(Gerber and Malhotra 2008a)](http://smr.sagepub.com/content/37/1/3.short)
]

.pull-right[
Ciencias Políticas [(Gerber and Malhotra 2008b)](http://nowpublishers.com/article/Details/QJPS-8024)
]
---
background-image: url(Images/Brodeur.PNG), url(Images/Brodeur_2.PNG)
background-size: 400px, 500px
background-position: 0% 50%, 100% 50%

# Evidencia: Economía

.pull-left[
AER, QJE, JPE [(Brodeur et al 2016)](http://ftp.iza.org/dp7268.pdf)
]

.pull-right[
Top-5,  Variables Instrumentales [(Brodeur et al 2020)](https://www.aeaweb.org/content/file?id=12747)
]

---
count:false

# Soluciones Para Problemas 1 y 2

.font150[
- Registros (o pre-pregistros) 
- Planes de pre-analysis 
- Reportes registrados 
]

---

# Registros (o pre-registros)

Registro público de los elementos centrales de una investigación (idealment ex-ante): hipótesis, variables de interest (dep. e ind.), y población de interes

Adopción casi universal en RCTs en medicina. Journals top (ICMJE) no publican estudios si no están registrados. <http://clinicaltrials.gov>
 
En ciencias sociales: 
 - Registro de AEA, actualmente solo para RCTs. <http://socialscienceregistry.org> 
 - Registro de EGAP, para ciencias politicas y estudios de governancia <http://egap.org/design-registration> 
 - Registro de 3ie, para evaluaciones en países en desarrollo. <http://ridie.3ieimpact.org>
 - Open Science Framework, multiples formatos <http://osf.io> 
 - As Predicted, formato simple: <http://aspredicted.org> 
 
---
background-image: url(Images/AEANewRegistrations_1.png)
background-size: 600px
 
# .font90[Rapida Adopción de Registros en Economía]
[Christensen et al 2020](https://doi.org/10.7910/DVN/FUO7FC)

---
background-image: url(Images/plosone.PNG)
background-size: 500px

# .font90[Registros en Estudios Nutricionales]
[Kaplan and Irvin 2015](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0132382)

---

# Planes de Pre-analysis

Un plan de pre-análisis (PAPs) es una descripción detallada de los análisis a ejecutar.

- Cómo se van a medir los principales variables de resultado,

- Dentro de las variables de resultado: cúales son primarias, cúales secundarias?

- Explicación detallada de todo el conjunto de tests a llevar a cabo.

- Ajustes por tests de hipótesis multiples

- Los subgrupos a ser analizados,

- Dirección esperada de los impactos si se quiere usar test de una cola, y

- La especificación primaria a ser utilizada en el análisis.

Extensa discussion de PAPs economía. Evidencia reciente apunta a falta de uniformidad, pero satisfacción por parte de autores (Ofosu and Posner 2020a, 2020b)

---
background-image: url(Images/GoBifo1.PNG)
background-size: 700px
# .font90[Utilidad de PAPs en Practica: [Casey et al 2012](https://academic.oup.com/qje/article-abstract/127/4/1755/1841616?redirectedFrom=fulltext)]

---
background-image: url(Images/results_blind_review.png)
background-size:400px
background-position:85% 90%
# Reportes registrados

Publicacion basada en calidad del diseño y de la pregunta de investigacion.

Cambia momento de peer review hacia antes del la recolección de datos, análisis y resultados. 
.pull-left[
1. Diseñar un estudio
2. Enviar a un journal
3. Revisión basada en la importancia de la pregunta y calidad del diseño
4. Obtener aceptación en principio
5. Ejecutar el estudio, y publicar incluso con resultados nulos
]

En BITSS trabajamos con el Journal of Development Economics para instaurar este tipo de publicaciones en economía ([Bogdanoski et al 2018](https://osf.io/preprints/metaarxiv/v7pxe/))

---

# .font70[Problema #3: Baja Replicabilidad y Reproducibilidad]

.font120[
| Replicabilidad en las Ciencias Sociales (mismo método, diferente muestra) 	| Reproducibilidad en Economía (mismo método y datos) 	|
|-------------------------------------------------------------------	|------------------------------------------------------	|
| OSC ([2015](https://docs.google.com/document/d/1mm_4HZnEz_2Bh8XuiS2tpqCH08MFPyqUhi1baKPqR8Y/edit#heading=h.7vqf2cziid7z)): 30%-60% 	 | Chang & Li ([2015](https://www.nowpublishers.com/article/Details/CFR-0053)): 43% 	|
| Camerer et. al. ([2016](https://science.sciencemag.org/content/351/6280/1433)): ~60% 	| Gertler et. al. ([2017](https://www.nature.com/articles/d41586-018-02108-9)): 14% 	|
| Nosek & Camerer et. al. ([2018](https://www.nature.com/articles/s41562-018-0399-z)): ~60% 	| Kingi et. al. ([2018](https://hautahi.com/static/docs/Replication_aejae.pdf)): 43% 	|
| Klein et. al. ([2018](https://journals.sagepub.com/doi/10.1177/2515245918810225)): 50% 	| Wood et. al. ([2018](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0209416#abstract0)): 25% 	|
]

---
# Soluciones

Soluciones Generales  
 - Condicionar fondos de investigación, y decisiones en journals, a seguimiento de altos estándares ([Guías TOP](https://www.cos.io/initiatives/top-guidelines)).   
 - Seguir buenas prácticas: organización de archivos, uso de control de versiones, software abierto, etc.   
 - Compartir código y datos.

Solucion especifica para Economía referente a **reproducibilidad computacional**: 
 - AEA en 2019 [cambió fuertemente](https://www.aeaweb.org/journals/policies/data-code/) sus politicas de publicacion de datos y codigo
 - Antes del 2019 no debemos esperar reproducibilidad del 100%. Pero podemos mejorarla. 
 - En BITSS hemos desarrollado estandares y una plataforma (en construcción) para guiar reproducciones en economía: **Accelerating Computational Reproducibility in Economics (ACRE)**

---
count: true  
background-image: url(Images/paper-claims.svg)
background-size: 610px
background-position: 100% 0%
  
# ACRE: Marco Conceptual

.pull-left[
.font100[
Cada **ejercicio de reproducción** 
está centrado entorno a una 
**afirmación científica**

Un artículo puede contener 
varias afirmaciones.

Cada afirmacion se basara 
en **objetos de resultados**: 
tablas, figuras y resultados 
en texto.

Cada ejercicio de reproducción 
es a nivel de afirmaciones, y 
los reproductores deben 
documentar sus 
**especificaciones** de interes
]
 
 
 DI: Display Item 
 S: Specificaiton
]
.pull-right[ 
]

---
background-image: url(Images/stages.svg)
background-size: contain
names: Stages

# Etapas

---
count: false
background-image: url(Images/assess.svg)
background-size: contain

# Evaluación

---
background-image: url(Images/assess.svg)
background-size: 300px
background-position: 0% 100%

# Identificar Componentes

# Asignar un Nivel   
.font90[
```md
                            Levels of Computational Reproducibility
                           (P denotes "partial", C denotes "complete")
    
    
                                       | Availability of materials, and reproducibility |
                                       |------------------------------------------------|
                                       |Analysis| Analysis|     | Cleaning| Raw   |     |
                                       |Code    | Data    | CRA | Code    | Data  | CRR |
                                       | P | C  | P  | C  |     | P  |  C | P | C |     |
                                       ---------|---------|-----|---------|-------|-----|
      L1: No materials.................| -   -  | -    -  |  -  |  -    - | -   - |  -  |
      ---------------------------------|--------|---------|-----|---------|-------|-----|
      L2: Only code ...................| ✔   ✔  | -    -  |  -  |  -    - | -   - |  -  |
      L3: Partial analysis data & code.| ✔   ✔  | ✔    -  |  -  |  -    - | -   - |  -  |
      L4: All analysis data & code.....| ✔   ✔  | ✔    ✔  |  -  |  -    - | -   - |  -  |
      L5: Reproducible from analysis...| ✔   ✔  | ✔    ✔  |  ✔  |  -    - | -   - |  -  |
      ---------------------------------|--------|---------|-----|---------|-------|-----|
      L6: Some cleaning code...........| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    - | -   - |  -  |
      L7: All cleaning code............| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | -   - |  -  |
      L8: Some raw data................| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | ✔   - |  -  |
      L9: All raw data.................| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | ✔   ✔ |  -  |
      L10:Reproducible from raw data...| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | ✔   ✔ |  ✔  |
```
]
---
count:false
background-image: url(Images/assess.svg)
background-size: 300px
background-position: 0% 100%

# Niveles

.font90[
```md
                            Levels of Computational Reproducibility
                           (P denotes "partial", C denotes "complete")
    
    
*                                      | Availability of materials, and reproducibility |
*                                      |------------------------------------------------|
*                                      |Analysis| Analysis|     | Cleaning| Raw   |     |
*                                      |Code    | Data    | CRA | Code    | Data  | CRR |
*                                      | P | C  | P  | C  |     | P  |  C | P | C |     |
                                       ---------|---------|-----|---------|-------|-----|
      L1: No materials.................| -   -  | -    -  |  -  |  -    - | -   - |  -  |
      ---------------------------------|--------|---------|-----|---------|-------|-----|
      L2: Only code ...................| ✔   ✔  | -    -  |  -  |  -    - | -   - |  -  |
      L3: Partial analysis data & code.| ✔   ✔  | ✔    -  |  -  |  -    - | -   - |  -  |
      L4: All analysis data & code.....| ✔   ✔  | ✔    ✔  |  -  |  -    - | -   - |  -  |
      L5: Reproducible from analysis...| ✔   ✔  | ✔    ✔  |  ✔  |  -    - | -   - |  -  |
      ---------------------------------|--------|---------|-----|---------|-------|-----|
      L6: Some cleaning code...........| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    - | -   - |  -  |
      L7: All cleaning code............| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | -   - |  -  |
      L8: Some raw data................| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | ✔   - |  -  |
      L9: All raw data.................| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | ✔   ✔ |  -  |
      L10:Reproducible from raw data...| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | ✔   ✔ |  ✔  |
```
]
---
count:false
background-image: url(Images/assess.svg)
background-size: 300px
background-position: 0% 100% 
# Niveles

.font90[
```md
                            Levels of Computational Reproducibility
                           (P denotes "partial", C denotes "complete")
    
    
                                       | Availability of materials, and reproducibility |
                                       |------------------------------------------------|
                                       |Analysis| Analysis|     | Cleaning| Raw   |     |
                                       |Code    | Data    | CRA | Code    | Data  | CRR |
                                       | P | C  | P  | C  |     | P  |  C | P | C |     |
                                       ---------|---------|-----|---------|-------|-----|
*     L1: No materials.................| -   -  | -    -  |  -  |  -    - | -   - |  -  |
      ---------------------------------|--------|---------|-----|---------|-------|-----|
      L2: Only code ...................| ✔   ✔  | -    -  |  -  |  -    - | -   - |  -  |
      L3: Partial analysis data & code.| ✔   ✔  | ✔    -  |  -  |  -    - | -   - |  -  |
      L4: All analysis data & code.....| ✔   ✔  | ✔    ✔  |  -  |  -    - | -   - |  -  |
      L5: Reproducible from analysis...| ✔   ✔  | ✔    ✔  |  ✔  |  -    - | -   - |  -  |
      ---------------------------------|--------|---------|-----|---------|-------|-----|
      L6: Some cleaning code...........| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    - | -   - |  -  |
      L7: All cleaning code............| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | -   - |  -  |
      L8: Some raw data................| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | ✔   - |  -  |
      L9: All raw data.................| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | ✔   ✔ |  -  |
      L10:Reproducible from raw data...| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | ✔   ✔ |  ✔  |
```
]
---
count:false
background-image: url(Images/assess.svg)
background-size: 300px
background-position: 0% 100% 
# Niveles

.font90[
```md
                            Levels of Computational Reproducibility
                           (P denotes "partial", C denotes "complete")
    
    
                                       | Availability of materials, and reproducibility |
                                       |------------------------------------------------|
                                       |Analysis| Analysis|     | Cleaning| Raw   |     |
                                       |Code    | Data    | CRA | Code    | Data  | CRR |
                                       | P | C  | P  | C  |     | P  |  C | P | C |     |
                                       ---------|---------|-----|---------|-------|-----|
      L1: No materials.................| -   -  | -    -  |  -  |  -    - | -   - |  -  |
      ---------------------------------|--------|---------|-----|---------|-------|-----|
*     L2: Only code ...................| ✔   ✔  | -    -  |  -  |  -    - | -   - |  -  |
      L3: Partial analysis data & code.| ✔   ✔  | ✔    -  |  -  |  -    - | -   - |  -  |
      L4: All analysis data & code.....| ✔   ✔  | ✔    ✔  |  -  |  -    - | -   - |  -  |
      L5: Reproducible from analysis...| ✔   ✔  | ✔    ✔  |  ✔  |  -    - | -   - |  -  |
      ---------------------------------|--------|---------|-----|---------|-------|-----|
      L6: Some cleaning code...........| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    - | -   - |  -  |
      L7: All cleaning code............| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | -   - |  -  |
      L8: Some raw data................| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | ✔   - |  -  |
      L9: All raw data.................| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | ✔   ✔ |  -  |
      L10:Reproducible from raw data...| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | ✔   ✔ |  ✔  |
```
]
---
count:false
background-image: url(Images/assess.svg)
background-size: 300px
background-position: 0% 100% 
# Niveles

.font90[
```md
                            Levels of Computational Reproducibility
                           (P denotes "partial", C denotes "complete")
    
    
                                       | Availability of materials, and reproducibility |
                                       |------------------------------------------------|
                                       |Analysis| Analysis|     | Cleaning| Raw   |     |
                                       |Code    | Data    | CRA | Code    | Data  | CRR |
                                       | P | C  | P  | C  |     | P  |  C | P | C |     |
                                       ---------|---------|-----|---------|-------|-----|
      L1: No materials.................| -   -  | -    -  |  -  |  -    - | -   - |  -  |
      ---------------------------------|--------|---------|-----|---------|-------|-----|
      L2: Only code ...................| ✔   ✔  | -    -  |  -  |  -    - | -   - |  -  |
*     L3: Partial analysis data & code.| ✔   ✔  | ✔    -  |  -  |  -    - | -   - |  -  |
      L4: All analysis data & code.....| ✔   ✔  | ✔    ✔  |  -  |  -    - | -   - |  -  |
      L5: Reproducible from analysis...| ✔   ✔  | ✔    ✔  |  ✔  |  -    - | -   - |  -  |
      ---------------------------------|--------|---------|-----|---------|-------|-----|
      L6: Some cleaning code...........| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    - | -   - |  -  |
      L7: All cleaning code............| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | -   - |  -  |
      L8: Some raw data................| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | ✔   - |  -  |
      L9: All raw data.................| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | ✔   ✔ |  -  |
      L10:Reproducible from raw data...| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | ✔   ✔ |  ✔  |
```
]
---
count:false
background-image: url(Images/assess.svg)
background-size: 300px
background-position: 0% 100% 
# Niveles

.font90[
```md
                            Levels of Computational Reproducibility
                           (P denotes "partial", C denotes "complete")
    
    
                                       | Availability of materials, and reproducibility |
                                       |------------------------------------------------|
                                       |Analysis| Analysis|     | Cleaning| Raw   |     |
                                       |Code    | Data    | CRA | Code    | Data  | CRR |
                                       | P | C  | P  | C  |     | P  |  C | P | C |     |
                                       ---------|---------|-----|---------|-------|-----|
      L1: No materials.................| -   -  | -    -  |  -  |  -    - | -   - |  -  |
      ---------------------------------|--------|---------|-----|---------|-------|-----|
      L2: Only code ...................| ✔   ✔  | -    -  |  -  |  -    - | -   - |  -  |
      L3: Partial analysis data & code.| ✔   ✔  | ✔    -  |  -  |  -    - | -   - |  -  |
*     L4: All analysis data & code.....| ✔   ✔  | ✔    ✔  |  -  |  -    - | -   - |  -  |
      L5: Reproducible from analysis...| ✔   ✔  | ✔    ✔  |  ✔  |  -    - | -   - |  -  |
      ---------------------------------|--------|---------|-----|---------|-------|-----|
      L6: Some cleaning code...........| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    - | -   - |  -  |
      L7: All cleaning code............| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | -   - |  -  |
      L8: Some raw data................| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | ✔   - |  -  |
      L9: All raw data.................| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | ✔   ✔ |  -  |
      L10:Reproducible from raw data...| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | ✔   ✔ |  ✔  |
```
]
---
count:false
background-image: url(Images/assess.svg)
background-size: 300px
background-position: 0% 100% 
# Niveles

.font90[
```md
                            Levels of Computational Reproducibility
                           (P denotes "partial", C denotes "complete")
    
    
                                       | Availability of materials, and reproducibility |
                                       |------------------------------------------------|
                                       |Analysis| Analysis|     | Cleaning| Raw   |     |
                                       |Code    | Data    | CRA | Code    | Data  | CRR |
                                       | P | C  | P  | C  |     | P  |  C | P | C |     |
                                       ---------|---------|-----|---------|-------|-----|
      L1: No materials.................| -   -  | -    -  |  -  |  -    - | -   - |  -  |
      ---------------------------------|--------|---------|-----|---------|-------|-----|
      L2: Only code ...................| ✔   ✔  | -    -  |  -  |  -    - | -   - |  -  |
      L3: Partial analysis data & code.| ✔   ✔  | ✔    -  |  -  |  -    - | -   - |  -  |
      L4: All analysis data & code.....| ✔   ✔  | ✔    ✔  |  -  |  -    - | -   - |  -  |
*     L5: Reproducible from analysis...| ✔   ✔  | ✔    ✔  |  ✔  |  -    - | -   - |  -  |
      ---------------------------------|--------|---------|-----|---------|-------|-----|
      L6: Some cleaning code...........| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    - | -   - |  -  |
      L7: All cleaning code............| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | -   - |  -  |
      L8: Some raw data................| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | ✔   - |  -  |
      L9: All raw data.................| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | ✔   ✔ |  -  |
      L10:Reproducible from raw data...| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | ✔   ✔ |  ✔  |
```
]
---
count:false
background-image: url(Images/assess.svg)
background-size: 300px
background-position: 0% 100% 
# Niveles

.font90[
```md
                            Levels of Computational Reproducibility
                           (P denotes "partial", C denotes "complete")
    
    
                                       | Availability of materials, and reproducibility |
                                       |------------------------------------------------|
                                       |Analysis| Analysis|     | Cleaning| Raw   |     |
                                       |Code    | Data    | CRA | Code    | Data  | CRR |
                                       | P | C  | P  | C  |     | P  |  C | P | C |     |
                                       ---------|---------|-----|---------|-------|-----|
      L1: No materials.................| -   -  | -    -  |  -  |  -    - | -   - |  -  |
      ---------------------------------|--------|---------|-----|---------|-------|-----|
      L2: Only code ...................| ✔   ✔  | -    -  |  -  |  -    - | -   - |  -  |
      L3: Partial analysis data & code.| ✔   ✔  | ✔    -  |  -  |  -    - | -   - |  -  |
      L4: All analysis data & code.....| ✔   ✔  | ✔    ✔  |  -  |  -    - | -   - |  -  |
      L5: Reproducible from analysis...| ✔   ✔  | ✔    ✔  |  ✔  |  -    - | -   - |  -  |
      ---------------------------------|--------|---------|-----|---------|-------|-----|
      L6: Some cleaning code...........| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    - | -   - |  -  |
      L7: All cleaning code............| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | -   - |  -  |
      L8: Some raw data................| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | ✔   - |  -  |
      L9: All raw data.................| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | ✔   ✔ |  -  |
*     L10:Reproducible from raw data...| ✔   ✔  | ✔    ✔  |  ✔  |  ✔    ✔ | ✔   ✔ |  ✔  |
```
]

---

#  .font90[Promoviendo una Conversación Constructiva]

Caso 3: Respondiendo ante la ausencia de algunos materiales

**Ejemplo de email:**

>**Subject:** Clarification for reproduction materials for `[“Title of the paper”]`

>Dear Dr. `[Lastname of Corresponding Author]`,
>
>Thank you for sharing the materials. They have been immensely helpful for my work.
>
>Unfortunately, I ran into a few issues as I delved into the reproduction exercise, and I think your guidance would be helpful in resolving them. `[Describe the issues and how you have tried to resolve them. Describe whatever files or parts of the data or code are missing. Refer to examples 1 and 2 below for more details]`.  
>  
>Thank you in advance for your help.
>  
>Best regards,  
>`[Reproducer]`

---
count:false
# .font90[Promoviendo una Conversación Constructiva]

Caso 3: Respondiendo ante la ausencia de algunos materiales

**Template email:**

>**Subject:** Clarification for reproduction materials for `[“Title of the paper”]`

>Dear Dr. `[Lastname of Corresponding Author]`,
>
>Thank you for sharing the materials. They have been immensely helpful for my work.
>
>Unfortunately, I ran into a few issues as I delved into the reproduction exercise, and I think your guidance would be helpful in resolving them. **`[Describe the issues and how you have tried to resolve them. Describe whatever files or parts of the data or code are missing. Refer to examples 1 and 2 below for more details]`**.  
>  
>Thank you in advance for your help.
>  
>Best regards,  
>`[Reproducer]`

---

# .font80[Ejemplo de un problema descrito en detalle]

.font80[
>Specifically, I am attempting to reproduce [`display item X (e.g., table 1, figure 3)`]. I found that the following components are required to reproduce to reproduce [`display item X `]:

```md
         display_item_X
            └───[code] formatting_table1.R
                ├───output1_part1.txt  
                |   └───[code] output_table1.do           
                |       └───[data] analysis_data01.csv
                |          └───[code] data_cleaning01.R*
                |             └───[data] UNKNOWN
                └───output1_part2.txt  
                    └───[code] output_table2.do           
                        └───[data] analysis_data02.csv
                           └───[code] data_cleaning02.R
                              └───[data] admin_01raw.csv* 
```

>I have marked with an asterisk (\*) the items that I could not find in the reproduction materials: data_cleaning01.R and admin_01raw.csv. After accessing these files, I will also be able to identify the name of the raw data set required to obtain output1_part1.txt. This is to let you know that I may need to contact you again if I cannot find this file (labeled as UNKNOWN above) in the reproduction materials.
>
>I understand that this request will require some work for you or somebody in your research group, but I want to assure you that I will add these missing files to the reproduction package for your paper on the ACRE platform. Doing this will ensure that you will not be asked twice for the same missing file.
]

---
count:false

# .font80[Ejemplo de un problema descrito en detalle]

.font80[
>Specifically, I am attempting to reproduce [`display item X (e.g., table 1, figure 3)`]. I found that the following components are required to reproduce to reproduce [`display item X `]:

```md
         display_item_X
            └───[code] formatting_table1.R
                ├───output1_part1.txt  
                |   └───[code] output_table1.do           
                |       └───[data] analysis_data01.csv
*               |          └───[code] data_cleaning01.R*
*               |             └───[data] UNKNOWN
                └───output1_part2.txt  
                    └───[code] output_table2.do           
                        └───[data] analysis_data02.csv
                           └───[code] data_cleaning02.R
*                             └───[data] admin_01raw.csv* 
```

>I have marked with an asterisk (\*) the items that I could not find in the reproduction materials: **data_cleaning01.R** and **admin_01raw.csv**. After accessing these files, I will also be able to identify the name of the raw data set required to obtain output1_part1.txt. This is to let you know that I may need to contact you again if I cannot find this file (labeled as **UNKNOWN** above) in the reproduction materials.
>
>I understand that this request will require some work for you or somebody in your research group, but I want to assure you that I will add these missing files to the reproduction package for your paper on the ACRE platform. **Doing this will ensure that you will not be asked twice for the same missing file.**
]

---
background-image: url(Images/Homepage.png)
background-size: contain 
name: platform
#  [socialsciencereproduction.org](https://www.socialsciencereproduction.org)

---

# Interesados en Participar

Pueden revisar las ACRE Guidelines, y tambien sugerir contribuciones: https://bitss.github.io/ACRE/

.center[
Una version beta de la plataforma va a estar disponible a principios de Noviembre
.font120[
[**Incripciones Aquí**](https://forms.gle/yZivWcwijCzEhrBU6) 
]
Para interesados en participar en el piloto de la plataforma. 
]

---
count: false
# Contenidos

.font130[

1. [BITSS](#about-bitss)

2. [Transparencia en la Investigación Científica](#ADD)

3. [Problemas y Soluciones](#ADD)

4. [**Aplicación al Análisis de Políticas Públicas**](#opa)
]

---
background-image: url(Images/figure_1_1.png)
background-size: contain
name: opa

# .font80[Transparencia en el Análisis de Políticas Públicas]
[Hoces, Grant and Miguel 2020](https://osf.io/preprints/metaarxiv/jnyqh/)

---
background-image: url(Images/figure_1_1_black.png)
background-size: contain 
# Un Link Entre la Investigación y la PP

---

# Crisis de Credibilidad en Análisis de PP
.font140[
 
- Incredible Certitudes (Manski, 2013) 
]
--
.font140[
- Report wars (Wesselink et al, 2013)
]
--
.font140[
- Alternative facts (“The Death of Expertise” Nichols, 2017; “The Death of Truth”, Kakutani 2018; “Post-Truth”, McIntyre 2018)
]

---

# .font80[Transparencia en Investigación (Open Science)]

.font180[

| 	| Empirical Research 	| Policy Analysis 	|
|:---------:	|:-------------------------------------------------------:	|:----------------------------------:	|
| **Problems** 	| Reproducibility Crisis 	| Credibility Crisis 	|
| **Solutions** 	| Open Science Principles, Guidelines, Applications 	| ...	|
]
---

# .font70[Transparencia en Analisis de PP (Open Policy Analysis)]

.font180[

# Principios de OPA
 
## 1 - Resultados Transparentes

## 2 - Análisis Transparentes

## 3 - Materiales Transparentes

---
background-image: url(Images/main_pe.png), url(Images/open_output1.svg)
background-size: 500px, 300px
background-position: 80% 50%, 0% 40%
count:true
# .font80[Resultados Transparentes: Un Solo Resultado]
---
background-image: url(Images/output-input.gif), url(Images/open_output2.svg)
background-size: 550px, 300px
background-position: 90% 50%, 0% 40%
count:true
# .font80[Resultados Transparentes: Claro Link Input-Output] 
---
background-image: url(Images/open_analysis.gif), url(Images/open_analysis.svg)
background-size: 550px, 300px
background-position: 100% 50%, 0% 40%
count:true
# Análisis Transparentes

---
background-image: url(Images/open_materials.gif), url(Images/open_materials.svg)
background-size: 600px, 300px
background-position: 100% 50%, 0% 40%
count:true
# Materiales Transparentes
---
background-image: url(Images/opa_framework.svg)
background-size: contain
count:true
# .font80[Marco Conceptual Para el Análisis de PP Abierto]

---
# Nuestra Agenda de OPA
 
.font140[
- Desarrollar un marco conceptual de OPA
- Apoyar la transicion/adoption de OPA, y desarrollar **proyectos de demonstración**. 
 - [Ver aqui](https://www.bitss.org/opa/projects/progressive-wealth-tax/) nuestro primer proyecto sobre el impuesto a la riqueza en USA. 
- Congregar instituciones y analistas interesados en OPA. 
- Entrenar estudiantes y analistas.
]

---
class: inverse, center, middle
name: framework
count: false

# Gracias

<html><div style='float:left'></div><hr color='#EB811B' size=1px width= %total% ></html>
.font130[
<fhoces@berkeley.edu>
]