This document describes the features of the Makefile-based LLVM -test-suite. This way of interacting with the test-suite is deprecated in favor -of running the test-suite using LNT, but may continue to prove useful for some -users. See the Testing -Guide's test-suite -Quickstart section for more information.
- -The test-suite module contains a number of programs that can be compiled -with LLVM and executed. These programs are compiled using the native compiler -and various LLVM backends. The output from the program compiled with the -native compiler is assumed correct; the results from the other programs are -compared to the native program output and pass if they match.
- -When executing tests, it is usually a good idea to start out with a subset of -the available tests or programs. This makes test run times smaller at first and -later on this is useful to investigate individual test failures. To run some -test only on a subset of programs, simply change directory to the programs you -want tested and run gmake there. Alternatively, you can run a different -test using the TEST variable to change what tests or run on the -selected programs (see below for more info).
- -In addition for testing correctness, the test-suite directory also -performs timing tests of various LLVM optimizations. It also records -compilation times for the compilers and the JIT. This information can be -used to compare the effectiveness of LLVM's optimizations and code -generation.
- -test-suite tests are divided into three types of tests: MultiSource, -SingleSource, and External.
- -The SingleSource directory contains test programs that are only a single -source file in size. These are usually small benchmark programs or small -programs that calculate a particular value. Several such programs are grouped -together in each directory.
The MultiSource directory contains subdirectories which contain entire -programs with multiple source files. Large benchmarks and whole applications -go here.
The External directory contains Makefiles for building code that is external -to (i.e., not distributed with) LLVM. The most prominent members of this -directory are the SPEC 95 and SPEC 2000 benchmark suites. The External -directory does not contain these actual tests, but only the Makefiles that know -how to properly compile these programs from somewhere else. The presence and -location of these external programs is configured by the test-suite -configure script.
Each tree is then subdivided into several categories, including applications, -benchmarks, regression tests, code that is strange grammatically, etc. These -organizations should be relatively self explanatory.
- -Some tests are known to fail. Some are bugs that we have not fixed yet; -others are features that we haven't added yet (or may never add). In the -regression tests, the result for such tests will be XFAIL (eXpected FAILure). -In this way, you can tell the difference between an expected and unexpected -failure.
- -The tests in the test suite have no such feature at this time. If the -test passes, only warnings and other miscellaneous output will be generated. If -a test fails, a large <program> FAILED message will be displayed. This -will help you separate benign warnings from actual test failures.
- -First, all tests are executed within the LLVM object directory tree. They -are not executed inside of the LLVM source tree. This is because the -test suite creates temporary files during execution.
- -To run the test suite, you need to use the following steps:
- -Check out the test-suite module with:
- --% svn co http://llvm.org/svn/llvm-project/test-suite/trunk test-suite --
This will get the test suite into llvm/projects/test-suite.
-Configure and build llvm.
Configure and build llvm-gcc.
Install llvm-gcc somewhere.
Re-configure llvm from the top level of - each build tree (LLVM object directory tree) in which you want - to run the test suite, just as you do before building LLVM.
-During the re-configuration, you must either: (1) - have llvm-gcc you just built in your path, or (2) - specify the directory where your just-built llvm-gcc is - installed using --with-llvmgccdir=$LLVM_GCC_DIR.
-You must also tell the configure machinery that the test suite - is available so it can be configured for your build tree:
--% cd $LLVM_OBJ_ROOT ; $LLVM_SRC_ROOT/configure [--with-llvmgccdir=$LLVM_GCC_DIR] --
[Remember that $LLVM_GCC_DIR is the directory where you - installed llvm-gcc, not its src or obj directory.]
-You can now run the test suite from your build tree as follows:
--% cd $LLVM_OBJ_ROOT/projects/test-suite -% make --
Note that the second and third steps only need to be done once. After you -have the suite checked out and configured, you don't need to do it again (unless -the test code or configure script changes).
- - -In order to run the External tests in the test-suite - module, you must specify --with-externals. This - must be done during the re-configuration step (see above), - and the llvm re-configuration must recognize the - previously-built llvm-gcc. If any of these is missing or - neglected, the External tests won't work.
-In addition to the regular "whole program" tests, the test-suite -module also provides a mechanism for compiling the programs in different ways. -If the variable TEST is defined on the gmake command line, the test system will -include a Makefile named TEST.<value of TEST variable>.Makefile. -This Makefile can modify build rules to yield different results.
- -For example, the LLVM nightly tester uses TEST.nightly.Makefile to -create the nightly test reports. To run the nightly tests, run gmake -TEST=nightly.
- -There are several TEST Makefiles available in the tree. Some of them are -designed for internal LLVM research and will not work outside of the LLVM -research group. They may still be valuable, however, as a guide to writing your -own TEST Makefile for any optimization or analysis passes that you develop with -LLVM.
- -There are a number of ways to run the tests and generate output. The most - simple one is simply running gmake with no arguments. This will - compile and run all programs in the tree using a number of different methods - and compare results. Any failures are reported in the output, but are likely - drowned in the other output. Passes are not reported explicitly.
- -Somewhat better is running gmake TEST=sometest test, which runs - the specified test and usually adds per-program summaries to the output - (depending on which sometest you use). For example, the nightly test - explicitly outputs TEST-PASS or TEST-FAIL for every test after each program. - Though these lines are still drowned in the output, it's easy to grep the - output logs in the Output directories.
- -Even better are the report and report.format targets - (where format is one of html, csv, text or - graphs). The exact contents of the report are dependent on which - TEST you are running, but the text results are always shown at the - end of the run and the results are always stored in the - report.<type>.format file (when running with - TEST=<type>). - - The report also generate a file called - report.<type>.raw.out containing the output of the entire test - run. -
Assuming you can run the test suite, (e.g. "gmake TEST=nightly report" -should work), it is really easy to run optimizations or code generator -components against every program in the tree, collecting statistics or running -custom checks for correctness. At base, this is how the nightly tester works, -it's just one example of a general framework.
- -Lets say that you have an LLVM optimization pass, and you want to see how -many times it triggers. First thing you should do is add an LLVM -statistic to your pass, which -will tally counts of things you care about.
- -Following this, you can set up a test and a report that collects these and -formats them for easy viewing. This consists of two files, a -"test-suite/TEST.XXX.Makefile" fragment (where XXX is the name of your -test) and a "test-suite/TEST.XXX.report" file that indicates how to -format the output into a table. There are many example reports of various -levels of sophistication included with the test suite, and the framework is very -general.
- -If you are interested in testing an optimization pass, check out the -"libcalls" test as an example. It can be run like this:
- -
-% cd llvm/projects/test-suite/MultiSource/Benchmarks # or some other level -% make TEST=libcalls report --
This will do a bunch of stuff, then eventually print a table like this:
- --Name | total | #exit | -... -FreeBench/analyzer/analyzer | 51 | 6 | -FreeBench/fourinarow/fourinarow | 1 | 1 | -FreeBench/neural/neural | 19 | 9 | -FreeBench/pifft/pifft | 5 | 3 | -MallocBench/cfrac/cfrac | 1 | * | -MallocBench/espresso/espresso | 52 | 12 | -MallocBench/gs/gs | 4 | * | -Prolangs-C/TimberWolfMC/timberwolfmc | 302 | * | -Prolangs-C/agrep/agrep | 33 | 12 | -Prolangs-C/allroots/allroots | * | * | -Prolangs-C/assembler/assembler | 47 | * | -Prolangs-C/bison/mybison | 74 | * | -... --
This basically is grepping the -stats output and displaying it in a table. -You can also use the "TEST=libcalls report.html" target to get the table in HTML -form, similarly for report.csv and report.tex.
- -The source for this is in test-suite/TEST.libcalls.*. The format is pretty -simple: the Makefile indicates how to run the test (in this case, -"opt -simplify-libcalls -stats"), and the report contains one line for -each column of the output. The first value is the header for the column and the -second is the regex to grep the output of the command for. There are lots of -example reports that can do fancy stuff.
- -
-