Information for Reviewers

As a member of the artifact evaluation committee (AEC), your main goal will be to decide which artifact badges should be awarded to the artifacts you review. We have created a reviewing template, aiming at guiding you for each artifact review.

Following last year’s tradition, we will ask authors to provide as much automation as possible in their artifacts, through “push-button evaluations”. We hope this will streamline the artifact reviewing process.

The artifact evaluation will have a single round, for accepted papers at round #1 and round #2 of ECOOP. The review will be split into two phases:

• the kick-the-tires phase aims at quickly checking the reviewers can run the artifact. Reviewers will be able to ask questions to authors to fix any issue they may encounter.
• the artifact assessment is the main reviewing phase.

Nomination forms

Do you want to participate to ECOOP’s artifact evaluation committee? Please fill the ECOOP’25 Artifact Evaluation PC Self-Nominations form. You will have to review artifacts during the submission period.

The nominations are open and the co-chairs will process them in March.

Important dates are published on the right hand side of this page. Please respect the AEC deadlines; if you foresee any difficulty in doing so, reach out to the AEC chairs as soon as possible.

Call for Artifacts

The call for artifacts will be announced soon. This year, artifact submission will happen for all accepted papers, and evaluation will be done after paper round #2 is completed.

Background

Artifacts of technical research papers can include supplementary material such as tools, datasets, models, tutorial videos, or substantial items associated with the research or study presented in the paper.

These artifacts enhance transparency and support the repeatability of experiments and precise comparison with alternative approaches, thus enabling higher quality in the research area as a whole. Additionally, artifacts enhance the overall understanding of the research by allowing others to review and build upon the work, making it easier for other researchers to perform their experiments, thus helping the original authors disseminate their ideas in detail.

Consequently, artifacts should be taken seriously and recognized separately. We encourage you to submit relevant artifacts along with the manuscript of technical research papers.

The AE process at ECOOP 2025 is a continuation of the AE process at previous ECOOP editions and several other conferences, including ESEC/FSE, OOPSLA, PLDI, ISSTA, HSCC, and SAS.

Badges

• Functional: The artifacts associated with the research are found to be documented, consistent, complete, exercisable, and include appropriate evidence of verification and validation. In particular, all experimental claims made in the paper should be reproducible through the artifact.

• Reusable The artifacts associated with the paper are of a quality that significantly exceeds minimal functionality. That is, they have all the qualities of the Artifacts Evaluated – Functional level. Yet, in addition, they are very carefully documented and well-structured to the extent that reuse and repurposing is facilitated. For the evaluation of the artifact’s reusability, we will rely on the reuse scenarios that the authors describe in their documentation. Providing an open-source implementation and its source code, and relying on public, open-source benchmarks are good steps to ensure the reusability of an artifact.

Irrespective of the artifact evaluation outcome, artifacts may be awarded the “Available” badge.

• Available - Author-created artifacts relevant to this paper have been placed on a publicly accessible archival repository. A DOI for the object is provided. We offer to publish the artifact on DARTS, in which case the available badge will be issued automatically.

A selected number of artifacts going beyond expectations of quality will receive a Distinguished Artifact award. The selection procedure will be based on review scores and feedback from the artifact evaluation committee.

Related resources

The process for evaluation is based on ACM’s Artifact Review and Badging and NISO’s guidelines for reproducibility badging. However, neither ACM nor NISO are involved in the implementation or evaluation process on behalf of ECOOP.

We will soon provide more detailed instructions by publishing the call for artifacts, artifact submission template and the artifact reviewer template. In the meantime, we advise prospective researchers to read the ACM SIGPLAN’s Empirical Evaluation Guidelines.