Chapter 7 Guidance for a Constructive Exchange Between Reproducers and Original Authors

This chapter contains guidance for constructive and respectful communication between reproducers and original authors. Exchanges that contain charged or adversarial language can damage professional relationships and hamper scientific progress. Janz and Freese (2019) articulate two important steps reproducers can take to ensure that their interactions with original authors are constructive. We summarize and build on this approach below and encourage you to follow this guidance. Remember the golden rule of reproductions (and replications): treat others and their work, as you would like others to treat you and your work!

1. Carefully and transparently plan your study.
a. Clearly state that you are conducting a reproduction of their original work.
b. Explain why you have chosen this study. c. Explain how “far” your results must deviate from the original work before claiming that the study could not be reproduced. Engage deeply with the substantive literature to ensure that your interpretation of differences between the original and reproduction is thorough and acceptable to other authors in the field.

2. Use professional and sensitive language. Discuss potential discrepancies between your work and the original paper, just like you might do for your own work.
a. Avoid binary judgments and statements like “failed to reproduce.” State clearly which results reproduced and which did not (e.g., “we successfully reproduced X, but failed to reproduce Y”) unless you uncover apparent scientific misconduct (e.g., see Broockman, Kalla and Aronow, 2015). b. Talk about the study, not the author, to avoid making it personal. Make clear what the positive contribution of the original article is. Consider sending a copy of your reproduction report to the original authors.
c. Discuss what your reproduction contributes to the literature, and refrain from claiming to give the final answer to the question.
d. For papers published five or more years ago, be mindful that norms for reproducibility have evolved since then.
e. Remember, the goal is not to criticize previous work or hunt for errors, but to move the literature forward!

To help you put these recommendations into practice, we developed template language for common scenarios that reproducers and authors may encounter in their interactions. While we hope that you find these useful, note that they are only recommendations, and you are welcome to modify them based on your project’s context and needs. Feel free to contact us if you need more guidance or would like to provide feedback on these materials.

7.1 For reproducers contacting the authors of the original study

Consider the following before you contact the original author:

  1. Eventually, you will be able to search the SSRP for reproduction materials or records of past interactions with the authors of a given paper, but you can skip this step for now.

  2. Carefully read all footnotes, appendices, tables, captions, etc. to learn if, how, and where reproduction materials are provided. Follow this Data and Code Guidance to determine whether you have everything before you start. A few things to consider:

    • A Readme file, if available, would be a good place to start.
    • Check whether there are any restrictions on accessing the data or code, and whether there are instructions on how to access these files for the purpose of reproduction.

  3. If a reproduction package is not readily available in the location where the article is published (e.g., the journal website), check the authors’ websites, Dataverse profiles, or other relevant archives and/or data repositories like the ICPSR Publications Related Archive.

  4. If steps 1 and 2 don’t yield anything, contact the corresponding author (copying the co-authors, if any), consolidating your requests into as few emails as possible. In your email, make sure to include the following details:

    • Basic information about the paper to reproduce (include title, version, date, and a DOI (or just a URL));
    • Context for the reproduction (as part of a class exercise, thesis, personal project, etc.) and a note that the outcome will be recorded on the Social Science Reproduction Platform(SSRP);
    • Items from the reproduction package that are missing, as well as locations where you had (unsuccessfully) searched for them;
    • Your use plan: Will the materials be used exclusively for this project? Ask for permission to share the data publicly.
    • Right to consultation and results: Will you share the outcome of the reproduction with the original authors?
    • A deadline to respond (we suggest at least two weeks).

  5. Follow up if you don’t get a response within two weeks (or whatever deadline you set), and include any details or clarifications that were left out in your first email.

  6. Record the outcome of your interaction with the original author at the Select a Paper stage on the SSRP. You can qualify the outcome as one of the following:

    • A complete reproduction package was provided
    • An incomplete reproduction package was provided. You can also select one of the following reasons:
      • Data is of sensitive, confidential, or proprietary character and cannot be shared;
      • Data is of sensitive, confidential, or proprietary character, but access instructions were provided.
    • The author declined to share the reproduction package
    • The author did not respond (including after a reminder was sent) within four weeks after the initial request.

7.1.1 Contacting the original author(s) when there is no reproduction package

Template email:

Subject: Reproduction package for ["Title of the paper"]

Dear [Title (e.g., "Dr.") Last name of Corresponding Author],

I am contacting you to request a reproduction package for your paper titled [Title] which was published in [Journal] in [year] (vol [volume], no. [no.]), [link]. A reproduction package contains (raw and/or analytic) data, code, and other documentation that makes it possible to reproduce the paper. Would you be able to share any of these items?

I am a [graduate student/postdoc/other position] at [Institution], and I would like to reproduce the results, tables, and other figures using the reproduction materials mentioned above. I have chosen this paper because [add context for why you want to reproduce this particular paper using neutral language (e.g., "This is a seminal paper in my field"), avoiding any statements that would put the respondent on the defensive]. Unfortunately, I was not able to locate any of these materials on the journal website, Dataverse [or other data and code repositories], or your website.

I will record the result of my reproduction attempt on the Social Science Reproduction Platform (SSRP), an open-source repository for the results of verifications of computational reproducibility of published work in the social sciences. SSRP is hosted by the Berkeley Initiative for Transparency in the Social Sciences). With your permission, I will also record the materials you share with me, which would allow access for other reproducers and avoid repeated requests directed to you. Please let me know if there are any legal or ethical restrictions that apply to any of the reproduction materials so that I can take that into consideration during this exercise.

In addition to your response above, would you be available to respond to future (non-repetitive) inquiries from me or other SSRP users? Though your cooperation with my and/or future requests would be extremely helpful, please note that you are not required to respond.

Since I am required to complete this project by [date], I would appreciate your response by [deadline].

Let me know if you have any questions. Please also feel free to contact my supervisor/instructor [Name (email)] for further details on this exercise. Thank you in advance for your help!

Best regards,
[Reproducer]

7.1.1.1 Follow-up if the author only provides the appendix

If the original author asks you to reproduce the results using the appendix, you are not obligated to undertake that effort. Should you choose to do so, you can send a follow-up email as follows.

Dear [Title (e.g., "Dr.") Last name of Corresponding Author],

Thank you for your response. Our group’s goal is to improve computational reproducibility using original data and code. Your appendix was very helpful, and I have attempted to use it to reproduce your results.[Describe initial efforts to reproduce the results.].

To help me advance with the reproduction, I hope you can share the source code and provide guidance on the following. [List out any unclear steps, data sources, or otherwise missing components.].

Thank you in advance for your time. Once completed, I aim to make the reproduction package publicly available so you can use it as a reference in future requests. Please let me know if you have any questions.

Best regards,
[Reproducer]

7.1.2 Contacting the original author(s) to request specific missing items of a reproduction package

Template email:

Subject: Reproduction materials for ["Title of the paper"]

Dear [Title (e.g., "Dr.") Last name of Corresponding Author],

I am contacting you regarding reproduction materials for your paper titled [Title] which was published in [Journal] in [year] (vol [volume], no. [no.]), [link].

I am a [graduate student/postdoc/other position] at [Institution], and I’m working on reproducing this paper as part of a class exercise. [Add context for why you want to reproduce this particular paper using neutral language (e.g., "This is a seminal paper in my field"), avoiding any statements that would put the respondent on the defensive].

To help me reproduce the paper in full, I hope that you can share the following items: [list items missing from reproduction package, preferably bulleted if more than one (e.g., raw/analytic data, code, protocols for conducting the experiment, etc.)]. I have already searched [locations where you searched for items, with links provided], but I could not locate the items. I will not share any of the materials without your permission, and I will use them exclusively for the purpose of this exercise. Let me know if any legal or ethical restrictions apply to any of the reproduction materials to consider during this exercise.

Note that I will record the outcome of my reproduction on the Social Science Reproduction Platform(SSRP), an open-source repository for the results of verifications of computational reproducibility of published work in the social sciences. The SSRP was developed by the Berkeley Initiative for Transparency in the Social Sciences). Let me know if you would like me to share the outcome of my reproduction with you, and whether you are interested in providing a response.

Since I am required to complete this project by [date], I would appreciate your response by [deadline].

Let me know if you have any questions. Please also feel free to contact my supervisor/instructor [Name (email)] for further details on this exercise. Thank you in advance for your help!

Best regards,
[Reproducer]

7.1.3 Asking for additional guidance when some materials have been shared

Note: Even when a corresponding author has shared a reproduction package, you may still face challenges in interpreting or executing the materials. That shouldn’t discourage you from asking the corresponding author to provide clarifications or to share missing materials. As in the previous scenario described above, demonstrate that you’ve made an honest effort to reproduce the work using the available resources and try to consolidate your requests into as few emails as possible.

Template email:

Subject: Clarification for reproduction materials for ["Title of the paper"]

Dear [Title (e.g., "Dr.") Last name of Corresponding Author],

Thank you for sharing your materials. They have been immensely helpful for my work.

Unfortunately, I ran into a few issues as I delved into the reproduction, and I think your guidance would be helpful in resolving them. [Describe the issues and how you have tried to resolve them. Describe whatever files or parts of the data or code are missing. Refer to examples 1 and 2 below for more details].

Thank you in advance for your help.

Best regards,
[Reproducer]

1: An example of well-described issues:

Specifically, I am attempting to reproduce Display Item X (e.g., table 1, figure 3). I found that the following components are required to reproduce Display Item X:

Display Item X └───[code] formatting_table1.R ├───display_itemx_part1.txt
| └───[code] output_table1.do
| └───[data] analysis_data01.csv | └───[code] data_cleaning01.R | └───[data] UNKNOWN └───display_itemx_part2.txt
└───[code] output_table2.do
└───[data] analysis_data02.csv └───[code] data_cleaning02.R └───[data] admin_01raw.csv >I have marked with an asterisk (*) the items that I could not find in the reproduction materials: data_cleaning01.R and admin_01raw.csv. After accessing these files, I will also identify the name of the raw data set required to obtain output1_part1.txt. This is to let you know that I may need to contact you again if I cannot find this file (labeled as UNKNOWN above) in the reproduction materials. > >I understand that this request will require some work for you, but I want to assure you that I will add these missing files to the reproduction package for your paper on the Social Science Reproduction Platform. Doing this will ensure that you will not be asked more than once for the same missing file.o

2. An example of poorly described issues:

Your paper does not reproduce. I have tried for several hours now, and can’t get the DO files to run. Could you please share all the missing reproduction materials? Data and code sharing are basic principles of open science, so I am confident that you will do the right thing.

7.1.4 response when the original author has declined to share due to undisclosed reasons

Note: You can also use this template if a corresponding author has not submitted a response after two or more follow-up emails.

Template email:

Subject: Re: Reproduction materials for ["Title of the paper"]

Dear [Title (e.g., "Dr.") Last name of Corresponding Author],

Thank you for considering my request. I will try to reproduce the paper using the available materials and will record the missing items accordingly on the Social Science Reproduction Platform (SSRP). l will also post my assessment of the reproducibility of the paper in its current form based on the SSRP reproducibility scale.

Let me know if you have any questions.

Best regards,
[Reproducer]

7.1.6 Contacting the original author to share the results of your reproduction exercise

Note: Reporting the results of reproductions can be the most contentious part of the process, particularly when the reproducer is unable to fully reproduce the paper or finds significant deviations from the original work. However, if the reproduction can correctly identify the sources of such deviations, it may be viewed as an improved version of the original work.

Regardless of the outcome of the reproduction exercise, the guidance from the introduction of this chapter still stands: reproduce the work of others as you would like for others to reproduce yours, and make sure that is reflected in how you discuss any discrepancies between your and the original work.

Template email:

Subject: Reproducibility Assessment of ["Title of the paper"]

Dear [Title (e.g., "Dr.") Last name of Corresponding Author],

Thank you for your support throughout my project as I worked to verify and advance the reproducibility of [Paper]. I’m writing now to share the results of my project and to invite your feedback.

The results of each step of my reproduction include i) Assessment, ii) Improvements, iii) Robustness Checks, (and iv) Extensions, if applicable).
`[Include the following items in the body of your email:

  • Briefly describe which parts of the paper you tried to reproduce (e.g., a specific estimate, a table, etc.).
  • Within the scope of your reproduction, describe exactly which items you were able to reproduce.
  • Discuss the differences you observed between the results of your reproduction and the original work, and demonstrate that you did your due diligence in trying to reproduce each item. Remember that it is more constructive to discuss discrepancies, differences or deviations, rather than errors, mistakes, or failures, and always talk about the work—not the author!
  • Use sensitive language when presenting discrepancies, e.g., “Unfortunately, I found X, which differs from the Y result in the original paper….” Be cognizant of any potential limitations of your work, and explain how you have tried to address them—that way, you will proactively address potential criticism!
  • Describe how you tried to improve the reproducibility of the paper. If some of the improvements are based on discretionary judgment (e.g., file organization or code commenting), try to explain why you think they are an improvement over the original work. If you didn’t make improvements, point out some concrete steps that the author(s) can take to improve the reproducibility of the section you reproduced.]`

I look forward to your questions, comments, and suggestions for my work. As discussed previously, I will record the outcomes of my reproduction, along with the improvements, on the Social Science Reproduction Platform.

Best regards,
[Reproducer]

7.1.7 Responding to hostile responses from original authors

Note: Planning your study carefully and transparently, and using professional and sensitive language are the best ways to ensure that the interaction will be beneficial to both you and the original author. However, unpleasant interactions may happen despite your best efforts, and can range anywhere from dismissive comments to bullying, discrimination, and harassment. Find guidance at the end of this chapter on how to deal with instances of bullying, harassment, or discrimination.

7.1.7.1 Dismissive comments

In cases of dismissive comments, the best course of action may be to simply thank the author for their response and continue with the exercise.

Template email:

Subject: Re: Reproduction materials for ["Title of the paper"]

Dear [Title (e.g., "Dr.") Last name of Corresponding Author],

Thank you for your response. I will work to reproduce your paper using the available materials and will record my results accordingly on the Social Science Reproduction Platform. l will also post my assessment of the reproducibility of the paper in its current form based on the SSRP reproducibility scale.

Let me know if you have any questions.

Best regards,
[Reproducer]

7.2 For original authors responding to requests from reproducers

This section contains guidance for authors of papers involved in reproductions on the Social Science Reproduction Platform. We present language that may help various scenarios in which authors find themselves when interacting with reproducers. Though every interaction between authors and reproducers takes place in a distinct context and may carry its own unique challenges, the guiding principle of this chapter always applies: “Treat others and their work as you would like others to treat you and your work!” We hope that these resources will facilitate more efficient and constructive exchanges between the parties involved. Let us know if you need guidance in other scenarios!

7.2.1 Responding to a repeated request that has been addressed in an earlier interaction

Dear [Reproducer],

Thank you for your interest in my work. I have been contacted about this issue by another SSRP reproducer before and provided a response, which I suspect may be already recorded on the SSRP. I’m copying my original response below for your reference. You may find further guidance in the readme file in the reproduction package.

If there are no prior records of these issues on the SSRP, please record the enclosed response [and materials]. This will also help avoid the duplication of effort >on the part of others who may be interested in reproducing this work.

Good luck with the remainder of your project and thank you in advance for your cooperation!

Best regards, [Author]

7.2.2 Acknowledging that the author no longer has access to certain part(s) of the reproduction package

Dear [Reproducer],

Thank you for reviewing my work closely. I wish I could be of more help, but unfortunately I no longer have access to the >requested materials due to [briefly describe the circumstances that prevent you from providing the materials].

While I recognize that the current standards in the discipline suggest that work should be reproducible using materials that are readily available for this purpose note that this paper was written when different standards of reproducibility >were mandated. Please feel free to evaluate the paper as is and propose any improvements wherever possible.

I look forward to working with you to address this and improve the overall reproducibility of the paper.

Best regards, [Author]

7.2.3 Acknowledging that some material is still embargoed for future research

Dear [Reproducer],

Thank you for your interest in my work. The data/materials/program that you reference are not publicly accessible for the time >being because they are embargoed until [embargo period].

[Depending on the restrictions that apply to the reproduction package, consider alternatives to sharing the reproduction materials in full. These include: 1.Sharing the analytic version of the data (the version of the dataset that was used for analysis in the final version of your paper); 2. Providing a public description of the steps other researchers can follow to request access to the raw data or materials, including an estimate of the costs and the duration of the process. Find examples of data availability statements for proprietary or restricted-access data [here](https://social-science-data-editors.github.io/guidance/Requested_information_dcas.html); and 3. Providing access to all data and materials for which the constraints do not apply.]

I hope you find this helpful for reproducing the paper. Please feel free to contact me if you have any further questions.

Best regards, [Author]

7.2.4 Responding to incomplete/unclear requests

Dear [Reproducer],

Thank you for your interest in my work. I would be happy to assist you and other reproducers to assess and improve the reproducibility of this paper.

To help me give more concrete guidance on this issue, I’d appreciate if you could provide a more specific description >of the items that you need from me. You can find helpful information and resources in Chapter 6 of the Guide, >specifically here [based on the context, you may need to point >the reproducer to a >different scenario and/or provide further information].

Feel free to contact me if you have any further questions. Thank you for your cooperation.

Best regards, [Author]

7.3 Harassment and/or discrimination

The American Economic Association and other economic societies have strict policies against harassment and discrimination. Here are some of the behaviors that the AEA Policy on Harassment and Discrimination has listed as unacceptable and could emerge in a hostile exchange regarding a reproduction:

  • Intentionally intimidating, threatening, harassing, or abusive actions or remarks (both spoken and in other media)
    • Prejudicial actions or comments that undermine the principles of equal opportunity, fair treatment, or free academic exchange
    • Deliberate intimidation, stalking, or following
    • Real or implied threat of physical harm.

Here are a some steps you can take if you believe you have experienced bullying, discrimination or harassment:

  • ** file a complaint with the AEA Ombudsperson.** Any AEA member can file a complaint. You can also join the AEA solely for the purpose of filing a report. The person about whom you are making the complaint need not be an AEA member. A non-AEA member can also file a report if the act of harassment or discrimination was committed by an AEA member or in the context of an AEA-sponsored activity. Learn more about the process here.
    • ** file a report with your institution’s office for the prevention of harassment & discrimination.** US-based institutions have internal mechanisms that allow students and faculty to seek support in cases of discrimination and harassment on the basis of race, color, national origin, gender, age, or sexual orientation/identity, including allegations of sexual harassment and sexual violence. Formal titles of this office vary across institutions, but common names include “Office for the Prevention of Harassment and Discrimination” (in institutions that are part of the University of California system), “Office of Equity and Title IX,” etc.
    • Contact your institution’s Ombudsperson/Ombuds Office. If you believe that you have experienced academic bullying or other forms of disrespectful behavior that fall outside the scope of harassment and/or discrimination as described above, you should know that university ombuds officers offer a confidential, impartial resource to discuss your concerns and learn about potential next steps available in your case.
    • Access mental health services at your institution. While no amount of bullying, discrimination, or harassment is acceptable or the fault of the victim, these unfortunately still occur and can take a toll on victims’ mental health. Many universities offer short-term Counseling & Psychological Services (CAPS) for academic, career, and personal issues.
    • Ask for support from your academic supervisor. If you are unsure on how to proceed, consult your academic supervisor on whether continuing the reproduction is appropriate.