Chapter 8 Guidance for Constructive Communication Between Reproducers and Original Authors

This chapter contains guidance for constructive and respectful communication between reproducers and original authors. Exchanges that contain charged or adversarial language can damage professional relationships and hamper scientific progress. Janz and Freese (2019) articulate two important steps reproducers can take to ensure that their interactions with original authors are constructive. We summarize and build on this approach below and encourage you to follow this guidance. Remember the golden rule of reproductions (and replications): treat others and their work, as you would like others to treat you and your work!

1. Carefully and transparently plan your study.

1. Clearly state that you are conducting a reproduction of their original work.
   
2. Explain why you have chosen this study.
3. If you are not able to reproduce the results explain how “far” your results must deviate from the original work before claiming that the study could not be reproduced. Engage deeply with the substantive literature to ensure that your interpretation of differences between the original and reproduction is thorough and acceptable to other authors in the field.

2. Use professional and sensitive language. Discuss potential discrepancies between your work and the original paper, just like you might do for your own work.

1. Avoid binary judgments and statements like “failed to reproduce.” Instead, state clearly which results reproduced and which did not (e.g., “we successfully reproduced X, but failed to reproduce Y”) unless you uncover apparent scientific misconduct (e.g., see Broockman, Kalla and Aronow, 2015).
   
2. Talk about the study, not the author to avoid making it personal. Make clear what the positive contribution of the original article is. Consider sending a copy of your reproduction report to the original authors.
   
3. Discuss what your reproduction contributes to the literature, and refrain from claiming to give the final answer to the question.
   
4. For papers published five or more years ago, be mindful that norms for reproducibility have evolved since then.
   
5. Remember, the goal is not to criticize previous work or hunt for errors, but to move the literature forward!

To help you put these recommendations into practice, we developed template language for common scenarios that reproducers and authors may encounter in their interactions. While we hope you find these useful, note that they are only recommendations, and you are welcome to modify them based on your project’s context and needs. Feel free to contact us if you need more guidance or would like to provide feedback on these materials.

8.1 For reproducers contacting the authors of the original study

Consider the following before you contact the original authors:

1. Search the SSRP for reproduction materials or records of past interactions with the authors of a given paper. For each reproduction, consult the “Author inquiries” column to see whether authors have indicated whether they are available for further inquiries.

2. Carefully read all footnotes, appendices, tables, captions, etc., to learn if, how, and where reproduction materials are provided. Follow this Data and Code Guidance to determine whether you have everything before you start. A few things to consider:

   • A Readme file, if available, would be a good place to start.
     
   • Check whether there are any restrictions on accessing the data or code, and whether there are instructions on how to access these files for the purpose of reproduction.

3. If a reproduction package is not readily available in the location where the article is published (e.g., the journal website), check the authors’ websites, Dataverse profiles, or other relevant archives and/or data repositories like the ICPSR Publications Related Archive.

4. If steps 1 and 2 don’t yield anything, contact the corresponding author (copying the co-authors, if any), consolidating your requests into as few emails as possible. In your email, make sure to include the following details:

   • Basic information about the paper to reproduce (include title, version, date, and a DOI (or just a URL));
     
   • Context for the reproduction (as part of a class exercise, thesis, personal project, etc.) and a note that the outcome will be recorded on the Social Science Reproduction Platform(SSRP);
   • Items from the reproduction package that are missing, as well as locations where you had (unsuccessfully) searched for them;
     
   • Your use plan: Will the materials be used exclusively for this project? Ask for permission to share the data publicly.
     
   • Right to consultation and results: Will you share the outcome of the reproduction with the original authors?
     
   • A deadline to respond (we suggest at least two weeks).

5. Follow up if you don’t get a response within two weeks (or whatever deadline you set), and include any details or clarifications that were left out in your first email.

6. Record the outcome of your interaction with the original author at the Select a Paper stage on the SSRP. You can qualify the outcome as one of the following:

   • A complete reproduction package was provided;
   • An incomplete reproduction package was provided. You can also select one of the following reasons:
     • Data is of sensitive, confidential, or proprietary character and cannot be shared;
     • Data is of sensitive, confidential, or proprietary character, but access instructions were provided.
   • The author declined to share the reproduction package; or
   • The author did not respond (including after a reminder was sent) within four weeks after the initial request.

8.1.1 Contacting the original author(s) when there is no reproduction package

Template email:

Subject: Reproduction package for ["Title of the paper"]

Dear [Title (e.g., "Dr.") Last name of Corresponding Author],

I am contacting you to request a reproduction package for your paper titled [Title] which was published in [Journal] in [year] (vol [volume], no. [no.]), [link]. A reproduction package contains (raw and/or analytic) data, code, and other documentation that makes it possible to reproduce the paper. Would you be able to share any of these items?

I am a [graduate student/postdoc/other position] at [Institution], and I would like to reproduce the results, tables, and other figures using the reproduction materials mentioned above. I have chosen this paper because [add context for why you want to reproduce this particular paper using neutral language (e.g., "This is a seminal paper in my field"), avoiding any statements that would put the respondent on the defensive]. Unfortunately, I was not able to locate any of these materials on the journal website, Dataverse [or other data and code repositories], or your website.

I will record the result of my reproduction attempt on the Social Science Reproduction Platform (SSRP), an open-source platform for systematically conducting and recording reproductions. With your permission, I will also record the materials you share with me, which would allow access for other reproducers and avoid repeated requests directed to you. Please let me know if there are any legal or ethical restrictions that apply to any of the reproduction materials so that I can take that into consideration during this exercise.

In addition to your response above, would you be available to respond to future (non-repetitive) inquiries from me or other SSRP users? Though your cooperation with my and/or future requests would be extremely helpful, please note that you are not required to respond.

Since I am required to complete this project by [date], I would appreciate your response by [deadline].

Let me know if you have any questions. Please also feel free to contact my supervisor/instructor [Name (email)] for further details on this exercise. Thank you in advance for your help!

Best regards,
[Reproducer]

8.1.2 Follow-up if the author only provides the appendix

If the original author asks you to reproduce the results using the appendix, you are not obligated to undertake that effort. Should you choose to do so, you can send a follow-up email as follows:

Dear [Title (e.g., "Dr.") Last name of Corresponding Author],

Thank you for your response. The purpose of my reproduction is to assess and improve computational reproducibility using the original data and code. Your appendix was very helpful, and I have attempted to use it to reproduce your results.[Describe initial efforts to reproduce the results.].

To help me advance with the reproduction, I hope you can share the source code and provide guidance on the following: [List out any unclear steps, data sources, or otherwise missing components. Use bullet points if more than two].

Thank you in advance for your help! Once completed, I will make the reproduction package publicly available to be used in future SSRP reproductions. Please let me know if you have any questions.

Best regards,
[Reproducer]

8.1.3 Contacting the original author(s) to request specific missing items of a reproduction package

Template email:

Subject: Reproduction materials for ["Title of the paper"]

Dear [Title (e.g., "Dr.") Last name of Corresponding Author],

I am contacting you regarding reproduction materials for your paper titled [Title] was published in [Journal] in [year] (vol [volume], no. [no.]), [link].

I am a [graduate student/postdoc/other position] at [Institution], and I’m working on reproducing this paper as part of a class assignment. [Add context for why you want to reproduce this particular paper using neutral language (e.g., "This is a seminal paper in my field"), avoiding any statements that would put the respondent on the defensive].

To help me reproduce the paper in full, I hope that you can share the following items: [list items missing from reproduction package, preferably bulleted if more than one (e.g., raw/analytic data, code, protocols for conducting the experiment, etc.)]. I have already searched [locations where you searched for items, with links provided], but I could not locate the items. Unless you tell me otherwise, I will make the reproduction package publicly available to be used in future reproductions. Let me know if any legal or ethical restrictions apply to any of the reproduction materials to consider during this exercise.

Note that I will record the outcome of my reproduction on the Social Science Reproduction Platform(SSRP), an open-source platform for systematically conducting and recording reproductions. Let me know if you would like me to share the outcome of my reproduction with you, and whether you are interested in providing a response.

Since I am required to complete this project by [date], I would appreciate your response by [deadline].

Let me know if you have any questions. Please also feel free to contact my supervisor/instructor [Name (email)] for further details on this exercise. Thank you in advance for your help!

Best regards,
[Reproducer]

8.1.4 Asking for additional guidance when some materials have been shared

Note: Even when a corresponding author has shared a reproduction package, you may still face challenges in interpreting or executing the materials. That shouldn’t discourage you from asking the corresponding author to provide clarifications or to share the missing materials. As in the previous scenario described above, demonstrate that you’ve made an honest effort to reproduce the work using the available resources and try to consolidate your requests in as few emails as possible.

Template email:

Subject: Clarification for reproduction materials for ["Title of the paper"]

Dear [Title (e.g., "Dr.") Last name of Corresponding Author],

Thank you for sharing your materials. They have been immensely helpful.

Unfortunately, I ran into a few issues as I delved into the reproduction, and I think your guidance would be helpful to resolve them. [Describe the issues and how you have tried to resolve them. Describe whatever files or parts of the data or code are missing. Refer to examples 1 and 2 below for more details].

Thank you in advance for your help.

Best regards,
[Reproducer]

1: An example of a well-described issues:

Specifically, I am attempting to reproduce Display Item X (e.g., table 1, figure 3). I found that the following components are required to reproduce Display Item X:

         Display Item X
            └───[code] formatting_table1.R
                ├───display_itemx_part1.txt  
                |   └───[code] output_table1.do           
                |       └───[data] analysis_data01.csv
                |          └───[code] data_cleaning01.R*
                |             └───[data] UNKNOWN
                └───display_itemx_part2.txt  
                    └───[code] output_table2.do           
                        └───[data] analysis_data02.csv
                           └───[code] data_cleaning02.R
                              └───[data] admin_01raw.csv* 

I have marked with an asterisk the items that I could not find in the reproduction materials: data_cleaning01.R and admin_01raw.csv. After accessing these files, I will also identify the name of the raw data set required to obtain output1_part1.txt. This is to let you know that I may need to contact you again if I cannot find this file (labeled as UNKNOWN above) in the reproduction materials.

I understand that this request will require some work for you; I will publish the materials on the Social Science Reproduction Platform, which should help avoid repeated requests in the future.

2. An example of a poorly-described issue:

Your paper does not reproduce. I have tried for several hours now, and can’t get the DO files to run. Could you please share all the missing reproduction materials? Data and code sharing are basic principles of open science, so I am confident that you will do the right thing.

8.1.5 Response when the original author has declined to share due to undisclosed reasons

Note: You can also use this template if a corresponding author has not submitted a response after two or more follow-up emails.

Template email:

Subject: Re: Reproduction materials for ["Title of the paper"]

Dear [Title (e.g., "Dr.") Last name of Corresponding Author],

Thank you for considering my request. I will try to reproduce the paper using the available materials and will record the missing items accordingly on the Social Science Reproduction Platform (SSRP). l will also post my assessment of the reproducibility of the paper in its current form based on the SSRP reproducibility scale.

Let me know if you have any questions.

Best regards,
[Reproducer]

8.1.6 Response when the original author has declined to share due to legal or ethical restrictions of the data

Template email:

Subject: Re: Reproduction materials for ["Title of the paper"]

Dear [Title (e.g., "Dr.") Last name of Corresponding Author],

Thank you for your response and for clarifying the terms of use for the reproduction materials.

Though I understand you are unable to share the raw data, there may be alternative steps you can take that would help me improve the reproducibility of your paper. These include:

1. Sharing the analytic version of the data (the version of the dataset that was used for analysis in the final version of your paper);
   
2. Providing a public description of the steps other researchers can follow to request access to the raw data or materials, including an estimate of the costs and the duration of the process. You can find examples of data availability statements for proprietary or restricted-access data here; and
   
3. Providing access to all data and materials for which the constraints do not apply.

Based on my assessment, your paper would currently rank at [level X] on the SSRP reproducibility scale. However, this score can be easily improved. Being able to provide analytic data would elevate the reproducibility of your paper to [level Y]. Providing public instructions on how other parties can access the data would further elevate its reproducibility to [level Z].

I would be happy to help if you are interested in taking any of the steps I outlined above.

Thank you for your help!

Best regards,
[Reproducer]

8.1.7 Contacting the original author to share the results of your reproduction exercise

Note: Reporting the results of reproductions can be the most contentious part of the process, particularly when the reproducer is unable to fully reproduce the paper or finds significant deviations from the original work. However, if the reproduction can correctly identify the sources of such deviations, it may be viewed as an improved version of the original work.

Regardless of the outcome of the reproduction exercise, the guidance from the introduction of this chapter still stands: reproduce the work of others as you would like for others to reproduce yours, and make sure that is reflected in how you discuss any discrepancies between your and the original work.

Template email:

Subject: Reproducibility Assessment of ["Title of the paper"]

Dear [Title (e.g., "Dr.") Last name of Corresponding Author],

Thank you for your support throughout my project as I worked to verify and advance the reproducibility of [Paper]. I’m writing now to share the results of my project and to invite your feedback.

The results of each step of my reproduction include i) Assessment, ii) Improvements, iii) Robustness Checks, (and iv) Extensions, if applicable).
`[Include the following items in the body of your email: ]`

I look forward to your questions, comments, and suggestions for my work. As discussed previously, I will record the outcomes of my reproduction, along with the improvements, on the Social Science Reproduction Platform.

Best regards,
[Reproducer]

8.1.8 Responding to hostile responses from original authors

Note: Planning your study carefully and transparently and using professional and sensitive language are the best ways to ensure that the interaction will be beneficial to both you and the original author. However, unpleasant interactions may happen despite your best efforts and can range anywhere from dismissive comments to bullying, discrimination, and harassment. Find guidance at the end of this chapter on how to deal with instances of bullying, harassment, or discrimination.

8.1.8.1 Dismissive comments

In cases of dismissive comments, the best course of action may be to simply thank the author for their response and continue with the reproduction.

Template email:

Subject: Re: Reproduction materials for ["Title of the paper"]

Dear [Title (e.g., "Dr.") Last name of Corresponding Author],

Thank you for your response. I will work to reproduce your paper using the available materials and will record my results accordingly on the Social Science Reproduction Platform. l will also post my assessment of the reproducibility of the paper in its current form based on the SSRP reproducibility scale.

Let me know if you have any questions.

Best regards,
[Reproducer]

8.2 For original authors responding to requests from reproducers

This section contains guidance for authors whose work is involved in reproductions on the Social Science Reproduction Platform. We present language that may help various scenarios in which authors find themselves when interacting with reproducers. Though every interaction between authors and reproducers takes place is distinct and may carry its unique challenges, the guiding principle of this chapter always applies: “Treat others and their work as you would like others to treat you and your work!” We hope that these resources will facilitate more efficient and constructive exchanges between the parties involved. Let us know if you need guidance in other scenarios!

8.2.1 Responding to a repeated request that has been addressed in an earlier interaction

Dear [Reproducer],

Thank you for your interest in my work. I have been contacted about this issue by another SSRP reproducer before and provided a response, which I suspect may be already recorded on the SSRP. I’m copying my original response below for your reference. You may find further guidance in the readme file in the reproduction package.

If there are no prior records of these issues on the SSRP, please record the enclosed response [and materials]. This will also help avoid the duplication of effort on the part of others who may be interested in reproducing this work.

Good luck with the remainder of your project and thank you in advance for your cooperation!

Best regards, [Author]

8.2.2 Acknowledging that the author no longer has access to certain part(s) of the reproduction package

Dear [Reproducer],

Thank you for reviewing my work closely. I wish I could be of more help, but unfortunately I no longer have access to the requested materials due to [briefly describe the circumstances that prevent you from providing the materials].

While I recognize that the current standards in the discipline have moved towards computational reproducibility, note that this paper was written when different standards applied. Please feel free to evaluate the paper as is and propose any improvements wherever possible.

I look forward to working with you to address this and improve the overall reproducibility of the paper.

Best regards, [Author]

8.2.3 Acknowledging that some material is still embargoed for future research

Dear [Reproducer],

Thank you for your interest in my work. The data/materials/program that you reference are currently not publicly available because they are embargoed until [embargo period].

[Depending on the restrictions that apply to the reproduction package, consider alternatives to sharing the reproduction materials in full. These include: 1.Sharing the analytic version of the data (the version of the dataset that was used for analysis in the final version of your paper); 2. Providing a public description of the steps that other researchers can follow to request access to the raw data or materials, including an estimate of the costs and the duration of the process. Find examples of data availability statements for proprietary or restricted-access data [here](https://social-science-data-editors.github.io/guidance/Requested_information_dcas.html); and 3. Providing access to all data and materials for which the constraints do not apply.]

I hope you find this helpful. Please feel free to contact me if you have any further questions.

Best regards, [Author]

8.2.4 Responding to incomplete/unclear requests

Dear [Reproducer],

Thank you for your interest in my work. I would be happy to assist you and other reproducers to assess and improve the reproducibility of this paper.

To help me give more concrete guidance, I’d appreciate if you could provide a more specific description >of the items that you need from me. You can find helpful information and resources in Chapter 6 of the Guide, >specifically here [based on the context, you may need to point >the reproducer to a >different scenario and/or provide further information].

Feel free to contact me if you have any further questions. Thank you for your cooperation.

Best regards, [Author]

8.3 Harassment and/or discrimination

The American Economic Association (AEA) and other academic societies have strict policies against harassment and discrimination. Here are some of the behaviors that the AEA Policy on Harassment and Discrimination has listed as unacceptable and could emerge in a hostile exchange regarding a reproduction:

• Intentionally intimidating, threatening, harassing, or abusive actions or remarks (both spoken and in other media)
• Prejudicial actions or comments that undermine the principles of equal opportunity, fair treatment, or free academic exchange
• Deliberate intimidation, stalking, or following
• Real or implied threat of physical harm.

Here are a some steps you can take if you believe you have experienced bullying, discrimination or harassment:

• File a complaint with the AEA Ombudsperson. Any AEA member can file a complaint. You can also join the AEA solely to file a report. The person about whom you are making the complaint need not be an AEA member. A non-AEA member can also file a report if the act of harassment or discrimination was committed by an AEA member or in the context of an AEA-sponsored activity. Learn more about the process here.
• File a report with your institution’s office for the prevention of harassment & discrimination. US-based institutions have internal mechanisms that allow students and faculty to seek support in cases of discrimination and harassment based on race, color, national origin, gender, age, or sexual orientation/identity, including allegations of sexual harassment and sexual violence. Formal titles of this office vary across institutions, but common names include “Office for the Prevention of Harassment and Discrimination” (in institutions that are part of the University of California system), “Office of Equity and Title IX,” etc.
• Contact your institution’s Ombudsperson/Ombuds Office. If you believe that you have experienced academic bullying or other forms of disrespectful behavior that fall outside the scope of harassment and/or discrimination as described above, you should know that university ombuds officers offer a confidential, impartial resource to discuss your concerns and learn about potential next steps available in your case.
• Access mental health services at your institution. While no amount of bullying, discrimination, or harassment is acceptable or the fault of the victim, these unfortunately still occur and can take a toll on victims’ mental health. Many universities offer short-term Counseling & Psychological Services (CAPS) for academic, career, and personal issues.
• Ask for support from your academic supervisor. If you are unsure on how to proceed, consult your academic supervisor on whether continuing the reproduction is appropriate.