diff --git a/docs/TestingGuide.html b/docs/TestingGuide.html new file mode 100644 index 00000000000..bf740cf9da3 --- /dev/null +++ b/docs/TestingGuide.html @@ -0,0 +1,356 @@ + + +
++ +
+
+
+ cd test
+
+ make
+
+
+
+ + To run only the code fragment tests (i.e. those that do basic testing of + LLVM), run the tests organized by QMTest: +
+
+
+ cd test
+
+ make qmtest
+
+
+
+ + To run only the tests that compile and execute whole programs, run the + Programs tests: +
+
+
+ cd test/Programs
+
+ make
+
+
+ + +
+ Code fragments are not complete programs, and they are never executed + to determine correct behavior. +
+ The tests in the llvm/test/Features and llvm/test/Regression directories + contain code fragments. + +
+ These programs are compiled and then executed using several different + methods (native compiler, LLVM C backend, LLVM JIT, LLVM native code + generation, etc). The output of these programs is compared to ensure + that LLVM is compiling the program correctly. + +
+ In addition to compiling and executing programs, whole program tests + serve as a way of benchmarking LLVM performance, both in terms of the + efficiency of the programs generated as well as the speed with which + LLVM compiles, optimizes, and generates code. + +
+ The test/Programs directory contains all tests which compile and + benchmark whole programs. +
+ +
+ +
+ In addition for testing correctness, the Programs directory + also performs timing tests of various LLVM optimizations. + It also records compilation times for the compilers and the + JIT. This information can be used to compare the + effectiveness of LLVM's optimizations and code generation. +
+ The Programs directory is subdivided into several smaller + subdirectories: +
+ +
+ +
+ +
+ + The QMTest system needs to have several pieces of information + available; these pieces of configuration information are known + collectively as the "context" in QMTest parlance. Since the context + for LLVM is relatively large, the master Makefile in llvm/test + sets it for you. + +
+ + The LLVM database class makes the directory tree underneath llvm/test a + QMTest test database. For each directory that contains tests driven by + QMTest, it knows what type of test the source file is and how to run it. + +
+ + Hence, the QMTest namespace is essentially what you see in + llvm/test/Feature and llvm/test/Regression, but there is some magic that + the database class performs (as described below). + +
+ + The QMTest namespace is currently composed of the following tests and + test suites: + +
+ +
+ +
+ +
+ +
+ + The LLVM database class looks at every file in llvm/test/Feature and + creates a fake test hierarchy containing + Feature.<testtype>.<testname>. + So, if you add an LLVM assembly language file to llvm/test/Feature, it + actually creates 5 news test: assembler/disassembler, assembler, + optimizer, machine code, and C code. + +
+ + If you add a new directory to llvm/test/Regression, you will need to + modify llvm/test/QMTest/llvmdb.py so that it knows what sorts of tests + are in it and how to run them. +
+ In addition to the regular Programs tests, the Programs tree also provides a + mechanism for compiling the programs in different ways. If the variable TEST + is defined on the gmake command line, the test system will include a Makefile + named TEST.<value of TEST variable>.Makefile. This Makefile can modify + build rules that yield different results. +
+ For example, the LLVM nightly tester uses TEST.nightly.Makefile to create the + nightly test reports. To run the nightly tests, run gmake + TEST=nightly. +
+ There are several TEST Makefiles available in the tree. Some of them are + designed for internal LLVM research and will not work outside of the LLVM + research group. They may still be valuable, however, as a guide to writing + your own TEST Makefile for any optimization or analysis passes that you + develop with LLVM. + + +
+ + The master Makefile in llvm/test is capable of running both the + QMTest driven tests and the Programs tests. By default, it will run + all of the tests. +
+ To run only the QMTest driven tests, run make qmtest at the + command line in llvm/tests. To run a specific qmtest, suffix the test name + with ".t" when running make. +
+ For example, to run the Regression.LLC tests, type + make Regression.LLC.t in llvm/tests. +
+ Note that the Makefiles in llvm/test/Features and llvm/test/Regression + are gone. You must now use QMTest from the llvm/test directory to run them. +
+ + To run the Programs test, cd into the llvm/test/Programs directory + and type make. Alternatively, you can type make + TEST=<type> test to run one of the specialized tests in + llvm/test/Programs/TEST.<type>.Makefile. For example, you could run + the nightly tester tests using the following commands: +
+
+ cd llvm/test/Programs
+
+ make TEST=nightly test
+
+
+
+ Regardless of which test you're running, the results are printed on standard + output and standard error. You can redirect these results to a file if you + choose. +
+ Some tests are known to fail. Some are bugs that we have not fixed yet; + others are features that we haven't added yet (or may never add). In QMTest, + the result for such tests will be XFAIL (eXpected FAILure). In this way, you + can tell the difference between an expected and unexpected failure. +
+ The Programs tests have no such feature as of this time. If the test passes, + only warnings and other miscellaneous output will be generated. If a test + fails, a large <program> FAILED message will be displayed. This will + help you separate benign warnings from actual test failures. + +