2004-03-01 18:21:04 +00:00
|
|
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
|
|
|
|
|
"http://www.w3.org/TR/html4/strict.dtd">
|
2003-10-10 18:42:49 +00:00
|
|
|
<html>
|
2003-10-23 18:10:28 +00:00
|
|
|
<head>
|
2004-03-01 18:21:04 +00:00
|
|
|
<title>LLVM Test Suite Guide</title>
|
|
|
|
|
<link rel="stylesheet" href="llvm.css" type="text/css">
|
2003-10-23 18:10:28 +00:00
|
|
|
</head>
|
|
|
|
|
<body>
|
|
|
|
|
|
|
|
|
|
<div class="doc_title">
|
|
|
|
|
LLVM Test Suite Guide
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
<ol>
|
|
|
|
|
<li><a href="#overview">Overview</a></li>
|
|
|
|
|
<li><a href="#Requirements">Requirements</a></li>
|
|
|
|
|
<li><a href="#quick">Quick Start</a></li>
|
2004-03-01 18:21:04 +00:00
|
|
|
<li><a href="#org">LLVM Test Suite Organization</a>
|
2003-10-23 18:10:28 +00:00
|
|
|
<ul>
|
|
|
|
|
<li><a href="#codefragments">Code Fragments</a></li>
|
|
|
|
|
<li><a href="#wholeprograms">Whole Programs</a></li>
|
2004-03-01 18:21:04 +00:00
|
|
|
</ul></li>
|
2003-10-23 18:10:28 +00:00
|
|
|
<li><a href="#tree">LLVM Test Suite Tree</a></li>
|
|
|
|
|
<li><a href="#qmstructure">QMTest Structure</a></li>
|
|
|
|
|
<li><a href="#progstructure">Programs Structure</a></li>
|
|
|
|
|
<li><a href="#run">Running the LLVM Tests</a></li>
|
|
|
|
|
</ol>
|
2003-10-10 18:42:49 +00:00
|
|
|
|
2004-05-23 21:07:27 +00:00
|
|
|
<div class="doc_author">
|
|
|
|
|
<p>Written by John T. Criswell</p>
|
|
|
|
|
</div>
|
2004-03-01 18:21:04 +00:00
|
|
|
|
|
|
|
|
<!--===============================================================-->
|
|
|
|
|
<div class="doc_section"><a name="overview">Overview</a></div>
|
|
|
|
|
<!--===============================================================-->
|
|
|
|
|
|
|
|
|
|
<div class="doc_text">
|
|
|
|
|
|
|
|
|
|
<p>This document is the reference manual for the LLVM test suite. It documents
|
|
|
|
|
the structure of the LLVM test suite, the tools needed to use it, and how to add
|
|
|
|
|
and run tests.</p>
|
|
|
|
|
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
<!--===============================================================-->
|
|
|
|
|
<div class="doc_section"><a name="Requirements">Requirements</a></div>
|
|
|
|
|
<!--===============================================================-->
|
|
|
|
|
|
|
|
|
|
<div class="doc_text">
|
|
|
|
|
|
|
|
|
|
<p>In order to use the LLVM test suite, you will need all of the software
|
|
|
|
|
required to build LLVM, plus the following:</p>
|
|
|
|
|
|
|
|
|
|
<dl>
|
|
|
|
|
<dt><a href="http://www.qmtest.com">QMTest</A></dt>
|
|
|
|
|
<dd>The LLVM test suite uses QMTest to organize and run tests.</dd>
|
|
|
|
|
|
|
|
|
|
<dt><a href="http://www.python.org">Python</A></dt>
|
|
|
|
|
<dd>You will need a Python interpreter that works with QMTest. Python will
|
|
|
|
|
need zlib and SAX support enabled.</dd>
|
|
|
|
|
</dl>
|
|
|
|
|
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
<!--===============================================================-->
|
|
|
|
|
<div class="doc_section"><a name="quick">Quick Start</a></div>
|
|
|
|
|
<!--===============================================================-->
|
|
|
|
|
|
|
|
|
|
<div class="doc_text">
|
|
|
|
|
|
|
|
|
|
<p> The tests are located in the LLVM source tree under the directory
|
|
|
|
|
<tt>llvm/test</tt>. To run all of the tests in LLVM, use the Master Makefile in
|
|
|
|
|
that directory:</p>
|
|
|
|
|
|
|
|
|
|
<pre>
|
|
|
|
|
% gmake -C llvm/test
|
|
|
|
|
</pre>
|
|
|
|
|
|
|
|
|
|
<p>To run only the code fragment tests (i.e. those that do basic testing of
|
|
|
|
|
LLVM), run the tests organized by QMTest:</p>
|
|
|
|
|
|
|
|
|
|
<pre>
|
|
|
|
|
% gmake -C llvm/test qmtest
|
|
|
|
|
</pre>
|
|
|
|
|
|
|
|
|
|
<p>To run only the tests that compile and execute whole programs, run the
|
|
|
|
|
Programs tests:</p>
|
|
|
|
|
|
|
|
|
|
<pre>
|
|
|
|
|
% gmake -C llvm/test/Programs
|
|
|
|
|
</pre>
|
|
|
|
|
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
<!--===============================================================-->
|
|
|
|
|
<div class="doc_section"><a name="org">LLVM Test Suite Organization</a></div>
|
|
|
|
|
<!--===============================================================-->
|
|
|
|
|
|
|
|
|
|
<div class="doc_text">
|
|
|
|
|
|
|
|
|
|
<p>The LLVM test suite contains two major categories of tests: code
|
|
|
|
|
fragments and whole programs.</p>
|
|
|
|
|
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
<div class="doc_subsection"><a name="codefragments">Code Fragments</a>
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
<div class="doc_text">
|
|
|
|
|
|
|
|
|
|
<p>Code fragments are small pieces of code that test a specific feature of LLVM
|
|
|
|
|
or trigger a specific bug in LLVM. They are usually written in LLVM assembly
|
|
|
|
|
language, but can be written in other languages if the test targets a particular
|
|
|
|
|
language front end.</p>
|
|
|
|
|
|
|
|
|
|
<p>Code fragments are not complete programs, and they are never executed to
|
|
|
|
|
determine correct behavior.</p>
|
|
|
|
|
|
|
|
|
|
<p>The tests in the Features and Regression directories contain code
|
|
|
|
|
fragments.</p>
|
|
|
|
|
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
<div class="doc_subsection"><a name="wholeprograms">Whole Programs</a></div>
|
|
|
|
|
|
|
|
|
|
<div class="doc_text">
|
|
|
|
|
|
|
|
|
|
<p>Whole Programs are pieces of code which can be compiled and linked into a
|
|
|
|
|
stand-alone program that can be executed. These programs are generally written
|
|
|
|
|
in high level languages such as C or C++, but sometimes they are written
|
|
|
|
|
straight in LLVM assembly.</p>
|
|
|
|
|
|
|
|
|
|
<p>These programs are compiled and then executed using several different
|
|
|
|
|
methods (native compiler, LLVM C backend, LLVM JIT, LLVM native code generation,
|
|
|
|
|
etc). The output of these programs is compared to ensure that LLVM is compiling
|
|
|
|
|
the program correctly.</p>
|
|
|
|
|
|
|
|
|
|
<p>In addition to compiling and executing programs, whole program tests serve as
|
|
|
|
|
a way of benchmarking LLVM performance, both in terms of the efficiency of the
|
|
|
|
|
programs generated as well as the speed with which LLVM compiles, optimizes, and
|
|
|
|
|
generates code.</p>
|
|
|
|
|
|
|
|
|
|
<p>The Programs directory contains all tests which compile and benchmark whole
|
|
|
|
|
programs.</p>
|
|
|
|
|
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
<!--===============================================================-->
|
|
|
|
|
<div class="doc_section"><a name="tree">LLVM Test Suite Tree</a></div>
|
|
|
|
|
<!--===============================================================-->
|
|
|
|
|
|
|
|
|
|
<div class="doc_text">
|
|
|
|
|
|
|
|
|
|
<p>Each type of test in the LLVM test suite has its own directory. The major
|
|
|
|
|
subtrees of the test suite directory tree are as follows:</p>
|
|
|
|
|
|
|
|
|
|
<ul>
|
|
|
|
|
<li>Features
|
|
|
|
|
<p>This directory contains sample codes that test various features of the
|
|
|
|
|
LLVM language. These pieces of sample code are run through various
|
|
|
|
|
assembler, disassembler, and optimizer passes.</p>
|
|
|
|
|
|
|
|
|
|
<li>Regression
|
|
|
|
|
<p>This directory contains regression tests for LLVM. When a bug is found
|
|
|
|
|
in LLVM, a regression test containing just enough code to reproduce the
|
|
|
|
|
problem should be written and placed somewhere underneath this directory.
|
|
|
|
|
In most cases, this will be a small piece of LLVM assembly language code,
|
|
|
|
|
often distilled from an actual application or benchmark.</p>
|
|
|
|
|
|
|
|
|
|
<li>Programs
|
|
|
|
|
<p>The Programs directory contains programs that can be compiled with LLVM
|
|
|
|
|
and executed. These programs are compiled using the native compiler and
|
|
|
|
|
various LLVM backends. The output from the program compiled with the native
|
|
|
|
|
compiler is assumed correct; the results from the other programs are
|
|
|
|
|
compared to the native program output and pass if they match. </p>
|
|
|
|
|
|
|
|
|
|
<p> In addition for testing correctness, the Programs directory also
|
|
|
|
|
performs timing tests of various LLVM optimizations. It also records
|
|
|
|
|
compilation times for the compilers and the JIT. This information can be
|
|
|
|
|
used to compare the effectiveness of LLVM's optimizations and code
|
|
|
|
|
generation.</p>
|
|
|
|
|
|
|
|
|
|
<p>The Programs directory is subdivided into several smaller subdirectories:
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<ul>
|
|
|
|
|
<li>Programs/SingleSource
|
|
|
|
|
<p>The SingleSource directory contains test programs that are only a
|
|
|
|
|
single source file in size. These are usually small benchmark programs
|
|
|
|
|
or small programs that calculate a particular value. Several such
|
|
|
|
|
programs are grouped together in each directory.</p></li>
|
|
|
|
|
|
|
|
|
|
<li>Programs/MultiSource
|
|
|
|
|
<p>The MultiSource directory contains subdirectories which contain
|
|
|
|
|
entire programs with multiple source files. Large benchmarks and whole
|
|
|
|
|
applications go here.</p></li>
|
|
|
|
|
|
|
|
|
|
<li>Programs/External
|
|
|
|
|
<p>The External directory contains Makefiles for building code that is
|
|
|
|
|
external to (i.e. not distributed with) LLVM. The most prominent member
|
|
|
|
|
of this directory is the SPEC 2000 benchmark suite. The presence and
|
|
|
|
|
location of these external programs is configured by the LLVM
|
|
|
|
|
<tt>configure</tt> script.</p></li>
|
2003-10-23 18:10:28 +00:00
|
|
|
|
2004-03-01 18:21:04 +00:00
|
|
|
</ul></li>
|
2003-10-23 18:10:28 +00:00
|
|
|
|
2004-03-01 18:21:04 +00:00
|
|
|
<li>QMTest
|
|
|
|
|
<p>This directory contains the QMTest information files. Inside this
|
|
|
|
|
directory are QMTest administration files and the Python code that
|
|
|
|
|
implements the LLVM test and database classes.</p>
|
|
|
|
|
|
|
|
|
|
</ul>
|
|
|
|
|
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
<!--===============================================================-->
|
|
|
|
|
<div class="doc_section"><a name="qmstructure">QMTest Structure</a></div>
|
|
|
|
|
<!--===============================================================-->
|
|
|
|
|
|
|
|
|
|
<div class="doc_text">
|
|
|
|
|
|
|
|
|
|
<p>The LLVM test suite is partially driven by QMTest and partially
|
|
|
|
|
driven by GNU Make. Specifically, the Features and Regression tests
|
|
|
|
|
are all driven by QMTest. The Programs directory is currently
|
|
|
|
|
driven by a set of Makefiles.</p>
|
|
|
|
|
|
|
|
|
|
<p>The QMTest system needs to have several pieces of information
|
|
|
|
|
available; these pieces of configuration information are known
|
|
|
|
|
collectively as the "context" in QMTest parlance. Since the context
|
|
|
|
|
for LLVM is relatively large, the master Makefile in llvm/test
|
|
|
|
|
sets it for you.</p>
|
|
|
|
|
|
|
|
|
|
<p>The LLVM database class makes the subdirectories of llvm/test a
|
|
|
|
|
QMTest test database. For each directory that contains tests driven by
|
|
|
|
|
QMTest, it knows what type of test the source file is and how to run it.</p>
|
2003-10-23 18:10:28 +00:00
|
|
|
|
2004-03-01 18:21:04 +00:00
|
|
|
<p>Hence, the QMTest namespace is essentially what you see in the
|
|
|
|
|
Feature and Regression directories, but there is some magic that
|
|
|
|
|
the database class performs (as described below).</p>
|
|
|
|
|
|
|
|
|
|
<p>The QMTest namespace is currently composed of the following tests and test
|
|
|
|
|
suites:</p>
|
|
|
|
|
|
|
|
|
|
<ul>
|
|
|
|
|
<li>Feature
|
|
|
|
|
<p>
|
|
|
|
|
These are the feature tests found in the Feature directory.
|
|
|
|
|
They are broken up into the following categories:
|
|
|
|
|
</p>
|
|
|
|
|
<ul>
|
|
|
|
|
<li>ad
|
|
|
|
|
<p>Assembler/Disassembler tests. These tests verify that a piece of LLVM
|
|
|
|
|
assembly language can be assembled into bytecode and then disassembled
|
|
|
|
|
into the original assembly language code. It does this several times to
|
|
|
|
|
ensure that assembled output can be disassembled and disassembler output
|
|
|
|
|
can be assembled. It also verifies that the give assembly language file
|
|
|
|
|
can be assembled correctly.</p></li>
|
|
|
|
|
|
|
|
|
|
<li>opt
|
|
|
|
|
<p>Optimizer tests. These tests verify that two of the optimizer passes
|
|
|
|
|
completely optimize a program (i.e. after a single pass, they cannot
|
|
|
|
|
optimize a program any further).</p></li>
|
|
|
|
|
|
|
|
|
|
<li>mc
|
|
|
|
|
<p> Machine code tests. These tests verify that the LLVM assembly
|
|
|
|
|
language file can be translated into native assembly code.</p></li>
|
|
|
|
|
|
|
|
|
|
<li>cc
|
|
|
|
|
<p>C code tests. These tests verify that the specified LLVM assembly
|
|
|
|
|
code can be converted into C source code using the C backend.</p></li>
|
|
|
|
|
</ul>
|
|
|
|
|
|
|
|
|
|
<p>The LLVM database class looks at every file in the Feature directory and
|
|
|
|
|
creates a fake test hierarchy containing
|
|
|
|
|
<tt>Feature.<testtype>.<testname></tt>. So, if you add an LLVM
|
|
|
|
|
assembly language file to the Feature directory, it actually creates 5 new
|
|
|
|
|
tests: assembler/disassembler, assembler, optimizer, machine code, and C code.
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<li>Regression
|
|
|
|
|
<p>These are the regression tests. There is one suite for each
|
|
|
|
|
subdirectory of the Regression directory. If you add a new subdirectory
|
|
|
|
|
there, you will need to modify, at least, the <tt>RegressionMap</tt>
|
|
|
|
|
variable in <tt>QMTest/llvmdb.py</tt> so that QMTest knows how to run the
|
|
|
|
|
tests in the new subdirectory.</p>
|
|
|
|
|
|
|
|
|
|
</ul>
|
|
|
|
|
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
<!--===============================================================-->
|
|
|
|
|
<div class="doc_section"><a name="progstructure">Programs Structure</a></div>
|
|
|
|
|
<!--===============================================================-->
|
|
|
|
|
|
|
|
|
|
<div class="doc_text">
|
|
|
|
|
|
|
|
|
|
<p>As mentioned previously, the Programs tree in llvm/test provides three types
|
|
|
|
|
of tests: MultiSource, SingleSource, and External. Each tree is then subdivided
|
|
|
|
|
into several categories, including applications, benchmarks, regression tests,
|
|
|
|
|
code that is strange grammatically, etc. These organizations should be
|
|
|
|
|
relatively self explanatory.</p>
|
|
|
|
|
|
|
|
|
|
<p>In addition to the regular Programs tests, the Programs tree also provides a
|
|
|
|
|
mechanism for compiling the programs in different ways. If the variable TEST is
|
|
|
|
|
defined on the gmake command line, the test system will include a Makefile named
|
|
|
|
|
<tt>TEST.<value of TEST variable>.Makefile</tt>. This Makefile can modify
|
|
|
|
|
build rules to yield different results.</p>
|
|
|
|
|
|
|
|
|
|
<p>For example, the LLVM nightly tester uses <tt>TEST.nightly.Makefile</tt> to
|
|
|
|
|
create the nightly test reports. To run the nightly tests, run <tt>gmake
|
|
|
|
|
TEST=nightly</tt>.</p>
|
|
|
|
|
|
|
|
|
|
<p>There are several TEST Makefiles available in the tree. Some of them are
|
|
|
|
|
designed for internal LLVM research and will not work outside of the LLVM
|
|
|
|
|
research group. They may still be valuable, however, as a guide to writing your
|
|
|
|
|
own TEST Makefile for any optimization or analysis passes that you develop with
|
|
|
|
|
LLVM.</p>
|
|
|
|
|
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
<!--===============================================================-->
|
|
|
|
|
<div class="doc_section"><a name="run">Running the LLVM Tests</a></div>
|
|
|
|
|
<!--===============================================================-->
|
|
|
|
|
|
|
|
|
|
<div class="doc_text">
|
|
|
|
|
|
|
|
|
|
<p>First, all tests are executed within the LLVM object directory tree. They
|
|
|
|
|
<i>are not</i> executed inside of the LLVM source tree. This is because the
|
|
|
|
|
test suite creates temporary files during execution. </p>
|
|
|
|
|
|
|
|
|
|
<p>The master Makefile in llvm/test is capable of running both the QMTest driven
|
|
|
|
|
tests and the Programs tests. By default, it will run all of the tests.</p>
|
|
|
|
|
|
|
|
|
|
<p>To run only the QMTest driven tests, run <tt>gmake qmtest</tt> at the
|
|
|
|
|
command line in llvm/tests. To run a specific qmtest, suffix the test name with
|
|
|
|
|
".t" when running gmake.</p>
|
|
|
|
|
|
|
|
|
|
<p>For example, to run the Regression.LLC tests, type <tt>gmake
|
|
|
|
|
Regression.LLC.t</tt> in llvm/tests.</p>
|
|
|
|
|
|
|
|
|
|
<p>Note that the Makefiles in llvm/test/Features and llvm/test/Regression are
|
|
|
|
|
gone. You must now use QMTest from the llvm/test directory to run them.</p>
|
|
|
|
|
|
|
|
|
|
<p>To run the Programs test, cd into the llvm/test/Programs directory and type
|
|
|
|
|
<tt>gmake</tt>. Alternatively, you can type <tt>gmake TEST=<type>
|
|
|
|
|
test</tt> to run one of the specialized tests in
|
|
|
|
|
llvm/test/Programs/TEST.<type>.Makefile. For example, you could run the
|
|
|
|
|
nightly tester tests using the following commands:</p>
|
|
|
|
|
|
|
|
|
|
<pre>
|
|
|
|
|
% cd llvm/test/Programs
|
|
|
|
|
% gmake TEST=nightly test
|
|
|
|
|
</pre>
|
|
|
|
|
|
|
|
|
|
<p>Regardless of which test you're running, the results are printed on standard
|
|
|
|
|
output and standard error. You can redirect these results to a file if you
|
|
|
|
|
choose.</p>
|
|
|
|
|
|
|
|
|
|
<p>Some tests are known to fail. Some are bugs that we have not fixed yet;
|
|
|
|
|
others are features that we haven't added yet (or may never add). In QMTest,
|
|
|
|
|
the result for such tests will be XFAIL (eXpected FAILure). In this way, you
|
|
|
|
|
can tell the difference between an expected and unexpected failure.</p>
|
|
|
|
|
|
|
|
|
|
<p>The Programs tests have no such feature as of this time. If the test passes,
|
|
|
|
|
only warnings and other miscellaneous output will be generated. If a test
|
|
|
|
|
fails, a large <program> FAILED message will be displayed. This will help
|
|
|
|
|
you separate benign warnings from actual test failures.</p>
|
|
|
|
|
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
<!-- *********************************************************************** -->
|
2003-10-23 18:10:28 +00:00
|
|
|
|
2004-03-01 18:21:04 +00:00
|
|
|
<hr>
|
|
|
|
|
<address>
|
|
|
|
|
<a href="http://jigsaw.w3.org/css-validator/check/referer"><img
|
|
|
|
|
src="http://jigsaw.w3.org/css-validator/images/vcss" alt="Valid CSS!"></a>
|
|
|
|
|
<a href="http://validator.w3.org/check/referer"><img
|
|
|
|
|
src="http://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01!" /></a>
|
|
|
|
|
|
|
|
|
|
John T. Criswell<br>
|
|
|
|
|
<a href="http://llvm.cs.uiuc.edu">The LLVM Compiler Infrastructure</a><br>
|
|
|
|
|
Last modified: $Date$
|
|
|
|
|
</address>
|
2003-10-23 18:10:28 +00:00
|
|
|
</body>
|
2003-10-10 18:42:49 +00:00
|
|
|
</html>
|
