Title: README.txt Description: README for the autorepairbenchmarks downloads Author: Claire Le Goues Contact: clegoues@cs.cmu.edu Original: June 1, 2014 Last Revised: December 20, 2014 (License information as applies to the entire benchmark set is listed below the overview text) ******************************************************************************* * * Overview ******************************************************************************* * This directory contains the defect scenarios and baseline results for IntroClass and ManyBugs, benchmarks for evaluating automatic program repair research that we describe in an article submitted to IEEE TSE in June, 2014. The purpose of this README is to provide a general overview of the layout of this package. We provide more detailed READMEs with the individual components of this package (i.e., in VirtualMachines/, ManyBugs/ and IntroClass/). For the most part we therefore direct you to those more detailed instructions for information on how to use these benchmarks and results. The article itself also contains useful and actionable information on how this information is organized. This directory contains: * genprog-v2.0-README.txt - http://dijkstra.cs.virginia.edu/genprog/resources/autorepairbenchmarks/genprog-v2.0-README.txt describes how to build and run repair, compiled from the genprog-code source, available elsewhere on the genprog website (genprog.cs.virginia.edu); this README is the same as the one that appears with the code. Running ae and tsprepair requires different flags to the same executable. * IntroClass/ - http://dijkstra.cs.virginia.edu/genprog/resources/autorepairbenchmarks/IntroClass/ contains the IntroClass programs and defects, the baseline results of running GenProg 2.0 and AE on those defects, and scripts to help you use the scenarios either in reproduction of our results or in your own work. IntroClass contains a directory for each of the IntroClass programs; each program directory contains support for running your own experiments as well as the results from our baseline experiments (reported in the article). IntroClass has its own README as well: - http://dijkstra.cs.virginia.edu/genprog/resources/autorepairbenchmarks/IntroClass/README * IntroClass.tar.gz - http://dijkstra.cs.virginia.edu/genprog/resources/autorepairbenchmarks/IntroClass.tar.gz is a downloadable zip file of the IntroClass benchmarks * ManyBugs/ - http://dijkstra.cs.virginia.edu/genprog/resources/autorepairbenchmarks/ManyBugs/ contains the ManyBugs programs and defects and the baseline results of running GenProg 2.0, TSPRepair, and AE on them. The results and benchmark scenarios are in different directories. ManyBugs has its own README, which explains the scenarios and results in more detail: - http://dijkstra.cs.virginia.edu/genprog/resources/autorepairbenchmarks/ManyBugs/README results/ - http://dijkstra.cs.virginia.edu/genprog/resources/autorepairbenchmarks/ManyBugs/results/ results/ is further subdivided into AE, tsprepair, and genprog2.0 results. The results consist of packages that contain the output debug files of the associated run; each debug file contains the command-line arguments that resulted in the run. These packages also contain repaired files, if applicable. scenarios/ - http://dijkstra.cs.virginia.edu/genprog/resources/autorepairbenchmarks/ManyBugs/scenarios/ The scenarios/ directory contains 185 tarballs of the defect scenarios, one per defect per program. The naming scheme indicates the subject program and the revision pair corresponding to the defect in question. Each tarball contains a test.sh file that has all "positive" and "negative" tests individually executable. The ManyBugs directory also contains bug-data.csv, a file that contains categorization information about the defects in the set: - http://dijkstra.cs.virginia.edu/genprog/resources/autorepairbenchmarks/ManyBugs/bug-data.csv * VirtualMachines/ contains: ** the ISOs for the two versions of Fedora that appear on the virtual machines. The baseline results were attained on a machine running Fedora 13. ** genprog_icse2012_aws, which contains a raw data dump of the Fedora 13 Amazon Machine Image. ** genprog_icse2012_virtualbox, which contains a virtual box image corresponding to the Fedora 13 Amazon Machine Image. ** a README that describes how to import and use those images, with additional information the publicly-available Amazon Machine Images we have released in the EC2 cloud. We are very interested in hearing feedback and errata from users who find problems or have questions about these benchmarks and associated information. ******************************************************************************* * * License ******************************************************************************* * The following BSD license applies to the entire download package of scencarios and results; it also appears in LICENSE in this directory. Copyright (c) 2014 Claire Le Goues Neal Holtschulte Edward K. Smith Yuriy Brun Prem Devanbu Stephanie Forrest Wes Weimer All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. The names of the contributors may not be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.