From eb82da891c2dbc88c4f3a4adbe436d9f7d122a62 Mon Sep 17 00:00:00 2001 From: Chris Lattner Date: Tue, 23 May 2006 01:40:20 +0000 Subject: [PATCH] Describe how to add a custom test. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@28430 91177308-0d34-0410-b5e6-96231b3b80d8 --- docs/TestingGuide.html | 85 ++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 82 insertions(+), 3 deletions(-) diff --git a/docs/TestingGuide.html b/docs/TestingGuide.html index b865ff554f0..f036515a7b7 100644 --- a/docs/TestingGuide.html +++ b/docs/TestingGuide.html @@ -24,7 +24,11 @@
  • LLVM Test Suite Tree
  • DejaGNU Structure
  • llvm-test Structure
  • -
  • Running the LLVM Tests
  • +
  • Running the LLVM Tests + +
  • Running the nightly tester
  • @@ -157,8 +161,9 @@ test suite is in the llvm-test module under the main directory.

    -
    Code Fragments -
    + +
    Code Fragments
    +
    @@ -175,7 +180,9 @@ determine correct behavior.

    +
    Whole Programs
    +
    @@ -471,6 +478,78 @@ will help you separate benign warnings from actual test failures.

    + +
    +Writing custom tests for llvm-test
    + + +
    + +

    Assuming you can run llvm-test, (e.g. "gmake TEST=nightly report" +should work), it is really easy to run optimizations or code generator +components against every program in the tree, collecting statistics or running +custom checks for correctness. At base, this is how the nightly tester works, +it's just one example of a general framework.

    + +

    Lets say that you have an LLVM optimization pass, and you want to see how +many times it triggers. First thing you should do is add an LLVM +statistic to your pass, which +will tally counts of things you care about.

    + +

    Following this, you can set up a test and a report that collects these and +formats them for easy viewing. This consists of two files, an +"llvm-test/TEST.XXX.Makefile" fragment (where XXX is the name of your +test) and an "llvm-test/TEST.XXX.report" file that indicates how to +format the output into a table. There are many example reports of various +levels of sophistication included with llvm-test, and the framework is very +general.

    + +

    If you are interested in testing an optimization pass, check out the +"libcalls" test as an example. It can be run like this:

    + +

    +
    +% cd llvm/projects/llvm-test/MultiSource/Benchmarks  # or some other level
    +% make TEST=libcalls report
    +
    +
    + +

    This will do a bunch of stuff, then eventually print a table like this:

    + +
    +
    +Name                                  | total | #exit |
    +...
    +FreeBench/analyzer/analyzer           | 51    | 6     | 
    +FreeBench/fourinarow/fourinarow       | 1     | 1     | 
    +FreeBench/neural/neural               | 19    | 9     | 
    +FreeBench/pifft/pifft                 | 5     | 3     | 
    +MallocBench/cfrac/cfrac               | 1     | *     | 
    +MallocBench/espresso/espresso         | 52    | 12    | 
    +MallocBench/gs/gs                     | 4     | *     | 
    +Prolangs-C/TimberWolfMC/timberwolfmc  | 302   | *     | 
    +Prolangs-C/agrep/agrep                | 33    | 12    | 
    +Prolangs-C/allroots/allroots          | *     | *     | 
    +Prolangs-C/assembler/assembler        | 47    | *     | 
    +Prolangs-C/bison/mybison              | 74    | *     | 
    +...
    +
    +
    + +

    This basically is grepping the -stats output and displaying it in a table. +You can also use the "TEST=libcalls report.html" target to get the table in HTML +form, similarly for report.csv and report.tex.

    + +

    The source for this is in llvm-test/TEST.libcalls.*. The format is pretty +simple: the Makefile indicates how to run the test (in this case, +"opt -simplify-libcalls -stats"), and the report contains one line for +each column of the output. The first value is the header for the column and the +second is the regex to grep the output of the command for. There are lots of +example reports that can do fancy stuff.

    + +
    + +
    Running the nightly tester