mirror of
https://github.com/c64scene-ar/llvm-6502.git
synced 2025-01-14 16:33:28 +00:00
* HTML 4.01 Strict compliance
* Removed unnecessary tabs in the entire file git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@12051 91177308-0d34-0410-b5e6-96231b3b80d8
This commit is contained in:
parent
daa4cb0f03
commit
1d83e11101
@ -1,11 +1,10 @@
|
||||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
|
||||
"http://www.w3.org/TR/html4/strict.dtd">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
|
||||
<link rel="stylesheet" href="llvm.css" type="text/css" media="screen" />
|
||||
<title>LLVM Test Suite Guide</title>
|
||||
<title>LLVM Test Suite Guide</title>
|
||||
<link rel="stylesheet" href="llvm.css" type="text/css">
|
||||
</head>
|
||||
|
||||
<body>
|
||||
|
||||
<div class="doc_title">
|
||||
@ -16,414 +15,369 @@
|
||||
<li><a href="#overview">Overview</a></li>
|
||||
<li><a href="#Requirements">Requirements</a></li>
|
||||
<li><a href="#quick">Quick Start</a></li>
|
||||
<li><a href="#org">LLVM Test Suite Organization</a></li>
|
||||
<li><a href="#org">LLVM Test Suite Organization</a>
|
||||
<ul>
|
||||
<li><a href="#codefragments">Code Fragments</a></li>
|
||||
<li><a href="#wholeprograms">Whole Programs</a></li>
|
||||
</ul>
|
||||
</ul></li>
|
||||
<li><a href="#tree">LLVM Test Suite Tree</a></li>
|
||||
<li><a href="#qmstructure">QMTest Structure</a></li>
|
||||
<li><a href="#progstructure">Programs Structure</a></li>
|
||||
<li><a href="#run">Running the LLVM Tests</a></li>
|
||||
<p><b>Written by John T. Criswell</b></p>
|
||||
</ol>
|
||||
|
||||
<!--===============================================================-->
|
||||
<div class="doc_section"><a name="overview">Overview</a></div>
|
||||
<!--===============================================================-->
|
||||
<p><b>Written by John T. Criswell</b></p>
|
||||
|
||||
<div class="doc_text">
|
||||
<p>
|
||||
This document is the reference manual for the LLVM test suite. It
|
||||
documents the structure of the LLVM test suite, the tools needed to
|
||||
use it, and how to add and run tests.
|
||||
</p>
|
||||
</div>
|
||||
<!--===============================================================-->
|
||||
<div class="doc_section"><a name="overview">Overview</a></div>
|
||||
<!--===============================================================-->
|
||||
|
||||
<!--===============================================================-->
|
||||
<div class="doc_section"><a name="Requirements">Requirements</a></div>
|
||||
<!--===============================================================-->
|
||||
<div class="doc_text">
|
||||
|
||||
<div class="doc_text">
|
||||
<p>
|
||||
In order to use the LLVM test suite, you will need all of the software
|
||||
required to build LLVM, plus the following:
|
||||
</p>
|
||||
<dl compact>
|
||||
<dt><A HREF="http://www.qmtest.com">QMTest</A></dt>
|
||||
<dd>The LLVM test suite uses QMTest to organize and
|
||||
run tests.</dd>
|
||||
<p>This document is the reference manual for the LLVM test suite. It documents
|
||||
the structure of the LLVM test suite, the tools needed to use it, and how to add
|
||||
and run tests.</p>
|
||||
|
||||
<dt><A HREF="http://www.python.org">Python</A></dt>
|
||||
<dd>You will need a Python interpreter that works with
|
||||
QMTest. Python will need zlib and SAX support
|
||||
enabled.</dd>
|
||||
</dl>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!--===============================================================-->
|
||||
<div class="doc_section"><a name="quick">Quick Start</a></div>
|
||||
<!--===============================================================-->
|
||||
<!--===============================================================-->
|
||||
<div class="doc_section"><a name="Requirements">Requirements</a></div>
|
||||
<!--===============================================================-->
|
||||
|
||||
<div class="doc_text">
|
||||
<p>
|
||||
The tests are located in the LLVM source tree under the directory
|
||||
<tt>llvm/test</tt>. To run all of the tests in LLVM, use the Master
|
||||
Makefile in that directory:
|
||||
</p>
|
||||
<pre>
|
||||
% gmake -C llvm/test
|
||||
</pre>
|
||||
<div class="doc_text">
|
||||
|
||||
<p>
|
||||
To run only the code fragment tests (i.e. those that do basic testing of
|
||||
LLVM), run the tests organized by QMTest:
|
||||
</p>
|
||||
<p>In order to use the LLVM test suite, you will need all of the software
|
||||
required to build LLVM, plus the following:</p>
|
||||
|
||||
<pre>
|
||||
% gmake -C llvm/test qmtest
|
||||
</pre>
|
||||
<dl>
|
||||
<dt><a href="http://www.qmtest.com">QMTest</A></dt>
|
||||
<dd>The LLVM test suite uses QMTest to organize and run tests.</dd>
|
||||
|
||||
<p>
|
||||
To run only the tests that compile and execute whole programs, run the
|
||||
Programs tests:
|
||||
</p>
|
||||
<dt><a href="http://www.python.org">Python</A></dt>
|
||||
<dd>You will need a Python interpreter that works with QMTest. Python will
|
||||
need zlib and SAX support enabled.</dd>
|
||||
</dl>
|
||||
|
||||
<pre>
|
||||
% gmake -C llvm/test/Programs
|
||||
</pre>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!--===============================================================-->
|
||||
<div class="doc_section"><h2><a name="org">LLVM Test Suite
|
||||
Organization </a></h2></div>
|
||||
<!--===============================================================-->
|
||||
<!--===============================================================-->
|
||||
<div class="doc_section"><a name="quick">Quick Start</a></div>
|
||||
<!--===============================================================-->
|
||||
|
||||
<div class="doc_text">
|
||||
<p>The LLVM test suite contains two major categories of tests: code
|
||||
fragments and whole programs.</p>
|
||||
</div>
|
||||
<div class="doc_text">
|
||||
|
||||
<div class="doc_subsection"><a name="codefragments">Code Fragments</a>
|
||||
</div>
|
||||
<p> The tests are located in the LLVM source tree under the directory
|
||||
<tt>llvm/test</tt>. To run all of the tests in LLVM, use the Master Makefile in
|
||||
that directory:</p>
|
||||
|
||||
<div class="doc_text">
|
||||
<p>
|
||||
Code fragments are small pieces of code that test a specific
|
||||
feature of LLVM or trigger a specific bug in LLVM. They are
|
||||
usually written in LLVM assembly language, but can be
|
||||
written in other languages if the test targets a
|
||||
particular language front end.
|
||||
</p><p>
|
||||
Code fragments are not complete programs, and they are
|
||||
never executed to determine correct behavior.
|
||||
</p><p>
|
||||
The tests in the Features and
|
||||
Regression directories contain code fragments.
|
||||
</p>
|
||||
</div>
|
||||
<pre>
|
||||
% gmake -C llvm/test
|
||||
</pre>
|
||||
|
||||
<div class="doc_subsection"><a name="wholeprograms">Whole Programs</a>
|
||||
</div>
|
||||
<p>To run only the code fragment tests (i.e. those that do basic testing of
|
||||
LLVM), run the tests organized by QMTest:</p>
|
||||
|
||||
<div class="doc_text">
|
||||
<p>
|
||||
Whole Programs are pieces of code which can be compiled and
|
||||
linked into a stand-alone program that can be executed. These
|
||||
programs are generally written in high level languages such as C
|
||||
or C++, but sometimes they are written straight in LLVM
|
||||
assembly.
|
||||
</p><p>
|
||||
These programs are compiled and then executed using several
|
||||
different methods (native compiler, LLVM C backend, LLVM JIT,
|
||||
LLVM native code generation, etc). The output of these programs
|
||||
is compared to ensure that LLVM is compiling the program
|
||||
correctly.
|
||||
</p><p>
|
||||
In addition to compiling and executing programs, whole program
|
||||
tests serve as a way of benchmarking LLVM performance, both in
|
||||
terms of the efficiency of the programs generated as well as the
|
||||
speed with which LLVM compiles, optimizes, and generates code.
|
||||
</p><p>
|
||||
The Programs directory contains all tests which compile and
|
||||
benchmark whole programs.
|
||||
</p>
|
||||
</div>
|
||||
<pre>
|
||||
% gmake -C llvm/test qmtest
|
||||
</pre>
|
||||
|
||||
<!--===============================================================-->
|
||||
<div class="doc_section"><h2><a name="tree">LLVM Test Suite Tree</a>
|
||||
</div>
|
||||
<!--===============================================================-->
|
||||
<p>To run only the tests that compile and execute whole programs, run the
|
||||
Programs tests:</p>
|
||||
|
||||
<div class="doc_text">
|
||||
<p>Each type of test in the LLVM test suite has its own directory. The
|
||||
major subtrees of the test suite directory tree are as follows:</p>
|
||||
<pre>
|
||||
% gmake -C llvm/test/Programs
|
||||
</pre>
|
||||
|
||||
</div>
|
||||
|
||||
<!--===============================================================-->
|
||||
<div class="doc_section"><a name="org">LLVM Test Suite Organization</a></div>
|
||||
<!--===============================================================-->
|
||||
|
||||
<div class="doc_text">
|
||||
|
||||
<p>The LLVM test suite contains two major categories of tests: code
|
||||
fragments and whole programs.</p>
|
||||
|
||||
</div>
|
||||
|
||||
<div class="doc_subsection"><a name="codefragments">Code Fragments</a>
|
||||
</div>
|
||||
|
||||
<div class="doc_text">
|
||||
|
||||
<p>Code fragments are small pieces of code that test a specific feature of LLVM
|
||||
or trigger a specific bug in LLVM. They are usually written in LLVM assembly
|
||||
language, but can be written in other languages if the test targets a particular
|
||||
language front end.</p>
|
||||
|
||||
<p>Code fragments are not complete programs, and they are never executed to
|
||||
determine correct behavior.</p>
|
||||
|
||||
<p>The tests in the Features and Regression directories contain code
|
||||
fragments.</p>
|
||||
|
||||
</div>
|
||||
|
||||
<div class="doc_subsection"><a name="wholeprograms">Whole Programs</a></div>
|
||||
|
||||
<div class="doc_text">
|
||||
|
||||
<p>Whole Programs are pieces of code which can be compiled and linked into a
|
||||
stand-alone program that can be executed. These programs are generally written
|
||||
in high level languages such as C or C++, but sometimes they are written
|
||||
straight in LLVM assembly.</p>
|
||||
|
||||
<p>These programs are compiled and then executed using several different
|
||||
methods (native compiler, LLVM C backend, LLVM JIT, LLVM native code generation,
|
||||
etc). The output of these programs is compared to ensure that LLVM is compiling
|
||||
the program correctly.</p>
|
||||
|
||||
<p>In addition to compiling and executing programs, whole program tests serve as
|
||||
a way of benchmarking LLVM performance, both in terms of the efficiency of the
|
||||
programs generated as well as the speed with which LLVM compiles, optimizes, and
|
||||
generates code.</p>
|
||||
|
||||
<p>The Programs directory contains all tests which compile and benchmark whole
|
||||
programs.</p>
|
||||
|
||||
</div>
|
||||
|
||||
<!--===============================================================-->
|
||||
<div class="doc_section"><a name="tree">LLVM Test Suite Tree</a></div>
|
||||
<!--===============================================================-->
|
||||
|
||||
<div class="doc_text">
|
||||
|
||||
<p>Each type of test in the LLVM test suite has its own directory. The major
|
||||
subtrees of the test suite directory tree are as follows:</p>
|
||||
|
||||
<ul>
|
||||
<li>Features
|
||||
<p>This directory contains sample codes that test various features of the
|
||||
LLVM language. These pieces of sample code are run through various
|
||||
assembler, disassembler, and optimizer passes.</p>
|
||||
|
||||
<li>Regression
|
||||
<p>This directory contains regression tests for LLVM. When a bug is found
|
||||
in LLVM, a regression test containing just enough code to reproduce the
|
||||
problem should be written and placed somewhere underneath this directory.
|
||||
In most cases, this will be a small piece of LLVM assembly language code,
|
||||
often distilled from an actual application or benchmark.</p>
|
||||
|
||||
<li>Programs
|
||||
<p>The Programs directory contains programs that can be compiled with LLVM
|
||||
and executed. These programs are compiled using the native compiler and
|
||||
various LLVM backends. The output from the program compiled with the native
|
||||
compiler is assumed correct; the results from the other programs are
|
||||
compared to the native program output and pass if they match. </p>
|
||||
|
||||
<p> In addition for testing correctness, the Programs directory also
|
||||
performs timing tests of various LLVM optimizations. It also records
|
||||
compilation times for the compilers and the JIT. This information can be
|
||||
used to compare the effectiveness of LLVM's optimizations and code
|
||||
generation.</p>
|
||||
|
||||
<p>The Programs directory is subdivided into several smaller subdirectories:
|
||||
</p>
|
||||
|
||||
<ul>
|
||||
<li>Programs/SingleSource
|
||||
<p>The SingleSource directory contains test programs that are only a
|
||||
single source file in size. These are usually small benchmark programs
|
||||
or small programs that calculate a particular value. Several such
|
||||
programs are grouped together in each directory.</p></li>
|
||||
|
||||
<li>Programs/MultiSource
|
||||
<p>The MultiSource directory contains subdirectories which contain
|
||||
entire programs with multiple source files. Large benchmarks and whole
|
||||
applications go here.</p></li>
|
||||
|
||||
<li>Programs/External
|
||||
<p>The External directory contains Makefiles for building code that is
|
||||
external to (i.e. not distributed with) LLVM. The most prominent member
|
||||
of this directory is the SPEC 2000 benchmark suite. The presence and
|
||||
location of these external programs is configured by the LLVM
|
||||
<tt>configure</tt> script.</p></li>
|
||||
|
||||
<ul>
|
||||
<li>Features
|
||||
<p>
|
||||
This directory contains sample codes that test various features
|
||||
of the LLVM language. These pieces of sample code are run
|
||||
through various assembler, disassembler, and optimizer passes.
|
||||
</p>
|
||||
</ul></li>
|
||||
|
||||
<li>Regression
|
||||
<p>
|
||||
This directory contains regression tests for LLVM. When a bug
|
||||
is found in LLVM, a regression test containing just enough
|
||||
code to reproduce the problem should be written and placed
|
||||
somewhere underneath this directory. In most cases, this
|
||||
will be a small piece of LLVM assembly language code, often
|
||||
distilled from an actual application or benchmark.
|
||||
</p>
|
||||
<li>QMTest
|
||||
<p>This directory contains the QMTest information files. Inside this
|
||||
directory are QMTest administration files and the Python code that
|
||||
implements the LLVM test and database classes.</p>
|
||||
|
||||
<li>Programs
|
||||
<p>
|
||||
The Programs directory contains programs that can be compiled
|
||||
with LLVM and executed. These programs are compiled using the
|
||||
native compiler and various LLVM backends. The output from the
|
||||
program compiled with the native compiler is assumed correct;
|
||||
the results from the other programs are compared to the native
|
||||
program output and pass if they match.
|
||||
</p><p>
|
||||
In addition for testing correctness, the Programs directory
|
||||
also performs timing tests of various LLVM optimizations.
|
||||
It also records compilation times for the compilers and the
|
||||
JIT. This information can be used to compare the
|
||||
effectiveness of LLVM's optimizations and code generation.
|
||||
</p><p>
|
||||
The Programs directory is subdivided into several smaller
|
||||
subdirectories:
|
||||
</p>
|
||||
</ul>
|
||||
|
||||
<ul>
|
||||
<li>Programs/SingleSource
|
||||
<p>
|
||||
The SingleSource directory contains test programs that
|
||||
are only a single source file in size. These are
|
||||
usually small benchmark programs or small programs that
|
||||
calculate a particular value. Several such programs are
|
||||
grouped together in each directory.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<li>Programs/MultiSource
|
||||
<p>
|
||||
The MultiSource directory contains subdirectories which
|
||||
contain entire programs with multiple source files.
|
||||
Large benchmarks and whole applications go here.
|
||||
</p>
|
||||
<!--===============================================================-->
|
||||
<div class="doc_section"><a name="qmstructure">QMTest Structure</a></div>
|
||||
<!--===============================================================-->
|
||||
|
||||
<li>Programs/External
|
||||
<p>
|
||||
The External directory contains Makefiles for building
|
||||
code that is external to (i.e. not distributed with)
|
||||
LLVM. The most prominent member of this directory is
|
||||
the SPEC 2000 benchmark suite. The presence and
|
||||
location of these external programs is configured by the
|
||||
LLVM <tt>configure</tt> script.
|
||||
</p>
|
||||
</ul>
|
||||
<div class="doc_text">
|
||||
|
||||
<p>
|
||||
<p>The LLVM test suite is partially driven by QMTest and partially
|
||||
driven by GNU Make. Specifically, the Features and Regression tests
|
||||
are all driven by QMTest. The Programs directory is currently
|
||||
driven by a set of Makefiles.</p>
|
||||
|
||||
<li>QMTest
|
||||
<p>
|
||||
This directory contains the QMTest information files. Inside
|
||||
this directory are QMTest administration files and the Python
|
||||
code that implements the LLVM test and database classes.
|
||||
</p>
|
||||
</ul>
|
||||
</div>
|
||||
<p>The QMTest system needs to have several pieces of information
|
||||
available; these pieces of configuration information are known
|
||||
collectively as the "context" in QMTest parlance. Since the context
|
||||
for LLVM is relatively large, the master Makefile in llvm/test
|
||||
sets it for you.</p>
|
||||
|
||||
<!--===============================================================-->
|
||||
<div class="doc_section"><h2><a name="qmstructure">QMTest Structure</a>
|
||||
</div>
|
||||
<!--===============================================================-->
|
||||
<p>The LLVM database class makes the subdirectories of llvm/test a
|
||||
QMTest test database. For each directory that contains tests driven by
|
||||
QMTest, it knows what type of test the source file is and how to run it.</p>
|
||||
|
||||
<div class="doc_text">
|
||||
<p>
|
||||
The LLVM test suite is partially driven by QMTest and partially
|
||||
driven by GNU Make. Specifically, the Features and Regression tests
|
||||
are all driven by QMTest. The Programs directory is currently
|
||||
driven by a set of Makefiles.
|
||||
</p><p>
|
||||
The QMTest system needs to have several pieces of information
|
||||
available; these pieces of configuration information are known
|
||||
collectively as the "context" in QMTest parlance. Since the context
|
||||
for LLVM is relatively large, the master Makefile in llvm/test
|
||||
sets it for you.
|
||||
</p><p>
|
||||
The LLVM database class makes the subdirectories of llvm/test a
|
||||
QMTest test database. For each directory that contains tests driven by
|
||||
QMTest, it knows what type of test the source file is and how to run it.
|
||||
</p><p>
|
||||
Hence, the QMTest namespace is essentially what you see in the
|
||||
Feature and Regression directories, but there is some magic that
|
||||
the database class performs (as described below).
|
||||
</p><p>
|
||||
The QMTest namespace is currently composed of the following tests and
|
||||
test suites:
|
||||
</p>
|
||||
<p>Hence, the QMTest namespace is essentially what you see in the
|
||||
Feature and Regression directories, but there is some magic that
|
||||
the database class performs (as described below).</p>
|
||||
|
||||
<ul>
|
||||
<li>Feature
|
||||
<p>
|
||||
These are the feature tests found in the Feature directory.
|
||||
They are broken up into the following categories:
|
||||
</p>
|
||||
<ul>
|
||||
<li>ad
|
||||
<p>
|
||||
Assembler/Disassembler tests. These tests verify that a
|
||||
piece of LLVM assembly language can be assembled into
|
||||
bytecode and then disassembled into the original
|
||||
assembly language code. It does this several times to
|
||||
ensure that assembled output can be disassembled and
|
||||
disassembler output can be assembled. It also verifies
|
||||
that the give assembly language file can be assembled
|
||||
correctly.
|
||||
</p>
|
||||
<p>The QMTest namespace is currently composed of the following tests and test
|
||||
suites:</p>
|
||||
|
||||
<li>opt
|
||||
<p>
|
||||
Optimizer tests. These tests verify that two of the
|
||||
optimizer passes completely optimize a program (i.e.
|
||||
after a single pass, they cannot optimize a program
|
||||
any further).
|
||||
</p>
|
||||
<ul>
|
||||
<li>Feature
|
||||
<p>
|
||||
These are the feature tests found in the Feature directory.
|
||||
They are broken up into the following categories:
|
||||
</p>
|
||||
<ul>
|
||||
<li>ad
|
||||
<p>Assembler/Disassembler tests. These tests verify that a piece of LLVM
|
||||
assembly language can be assembled into bytecode and then disassembled
|
||||
into the original assembly language code. It does this several times to
|
||||
ensure that assembled output can be disassembled and disassembler output
|
||||
can be assembled. It also verifies that the give assembly language file
|
||||
can be assembled correctly.</p></li>
|
||||
|
||||
<li>mc
|
||||
<p>
|
||||
Machine code tests. These tests verify that the LLVM
|
||||
assembly language file can be translated into native
|
||||
assembly code.
|
||||
</p>
|
||||
<li>opt
|
||||
<p>Optimizer tests. These tests verify that two of the optimizer passes
|
||||
completely optimize a program (i.e. after a single pass, they cannot
|
||||
optimize a program any further).</p></li>
|
||||
|
||||
<li>cc
|
||||
<p>
|
||||
C code tests. These tests verify that the specified
|
||||
LLVM assembly code can be converted into C source code
|
||||
using the C backend.
|
||||
</p>
|
||||
</ul>
|
||||
<li>mc
|
||||
<p> Machine code tests. These tests verify that the LLVM assembly
|
||||
language file can be translated into native assembly code.</p></li>
|
||||
|
||||
<p>
|
||||
The LLVM database class looks at every file in the Feature
|
||||
directory and creates a fake test hierarchy containing
|
||||
<tt>Feature.<testtype>.<testname></tt>. So, if you
|
||||
add an LLVM assembly language file to the Feature directory, it
|
||||
actually creates 5 new tests: assembler/disassembler, assembler,
|
||||
optimizer, machine code, and C code.
|
||||
</p>
|
||||
<li>cc
|
||||
<p>C code tests. These tests verify that the specified LLVM assembly
|
||||
code can be converted into C source code using the C backend.</p></li>
|
||||
</ul>
|
||||
|
||||
<li>Regression
|
||||
<p>
|
||||
These are the regression tests. There is one suite for each
|
||||
subdirectory of the Regression directory. If you add a new
|
||||
subdirectory there, you will need to modify, at least, the
|
||||
<tt>RegressionMap</tt> variable in <tt>QMTest/llvmdb.py</tt> so
|
||||
that QMTest knows how to run the tests in the new subdirectory.
|
||||
</p>
|
||||
</ul>
|
||||
</div>
|
||||
<p>The LLVM database class looks at every file in the Feature directory and
|
||||
creates a fake test hierarchy containing
|
||||
<tt>Feature.<testtype>.<testname></tt>. So, if you add an LLVM
|
||||
assembly language file to the Feature directory, it actually creates 5 new
|
||||
tests: assembler/disassembler, assembler, optimizer, machine code, and C code.
|
||||
</p>
|
||||
|
||||
<!--===============================================================-->
|
||||
<div class="doc_section"><h2><a name="progstructure">Programs
|
||||
Structure</a></div>
|
||||
<!--===============================================================-->
|
||||
<li>Regression
|
||||
<p>These are the regression tests. There is one suite for each
|
||||
subdirectory of the Regression directory. If you add a new subdirectory
|
||||
there, you will need to modify, at least, the <tt>RegressionMap</tt>
|
||||
variable in <tt>QMTest/llvmdb.py</tt> so that QMTest knows how to run the
|
||||
tests in the new subdirectory.</p>
|
||||
|
||||
<div class="doc_text">
|
||||
<p>
|
||||
As mentioned previously, the Programs tree in llvm/test provides three
|
||||
types of tests: MultiSource, SingleSource, and External. Each tree is
|
||||
then subdivided into several categories, including applications,
|
||||
benchmarks, regression tests, code that is strange grammatically, etc.
|
||||
These organizations should be relatively self explanatory.
|
||||
</p><p>
|
||||
In addition to the regular Programs tests, the Programs tree also
|
||||
provides a mechanism for compiling the programs in different ways. If
|
||||
the variable TEST is defined on the gmake command line, the test system
|
||||
will include a Makefile named <tt>TEST.<value of TEST
|
||||
variable>.Makefile</tt>. This Makefile can modify build rules to
|
||||
yield different results.
|
||||
</p><p>
|
||||
For example, the LLVM nightly tester uses <tt>TEST.nightly.Makefile</tt>
|
||||
to create the nightly test reports. To run the nightly tests, run
|
||||
<tt>gmake TEST=nightly</tt>.
|
||||
</p><p>
|
||||
There are several TEST Makefiles available in the tree. Some of them
|
||||
are designed for internal LLVM research and will not work outside of the
|
||||
LLVM research group. They may still be valuable, however, as a guide to
|
||||
writing your own TEST Makefile for any optimization or analysis passes
|
||||
that you develop with LLVM.
|
||||
</p>
|
||||
</div>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
||||
<!--===============================================================-->
|
||||
<div class="doc_section"><h2><a name="run">Running the LLVM Tests</a>
|
||||
</div>
|
||||
<!--===============================================================-->
|
||||
<!--===============================================================-->
|
||||
<div class="doc_section"><a name="progstructure">Programs Structure</a></div>
|
||||
<!--===============================================================-->
|
||||
|
||||
<div class="doc_text">
|
||||
<p>
|
||||
First, all tests are executed within the LLVM object directory tree.
|
||||
They <i>are not</i> executed inside of the LLVM source tree. This is
|
||||
because the test suite creates temporary files during execution.
|
||||
</p><p>
|
||||
The master Makefile in llvm/test is capable of running both the
|
||||
QMTest driven tests and the Programs tests. By default, it will run
|
||||
all of the tests.
|
||||
</p><p>
|
||||
To run only the QMTest driven tests, run <tt>gmake qmtest</tt> at the
|
||||
command line in llvm/tests. To run a specific qmtest, suffix the test
|
||||
name with ".t" when running gmake.
|
||||
</p><p>
|
||||
For example, to run the Regression.LLC tests, type
|
||||
<tt>gmake Regression.LLC.t</tt> in llvm/tests.
|
||||
</p><p>
|
||||
Note that the Makefiles in llvm/test/Features and llvm/test/Regression
|
||||
are gone. You must now use QMTest from the llvm/test directory to run
|
||||
them.
|
||||
</p><p>
|
||||
To run the Programs test, cd into the llvm/test/Programs directory and
|
||||
type <tt>gmake</tt>. Alternatively, you can type <tt>gmake
|
||||
TEST=<type> test</tt> to run one of the specialized tests in
|
||||
llvm/test/Programs/TEST.<type>.Makefile. For example, you could
|
||||
run the nightly tester tests using the following commands:
|
||||
</p>
|
||||
<div class="doc_text">
|
||||
|
||||
<pre>
|
||||
% cd llvm/test/Programs
|
||||
% gmake TEST=nightly test
|
||||
</pre>
|
||||
<p>As mentioned previously, the Programs tree in llvm/test provides three types
|
||||
of tests: MultiSource, SingleSource, and External. Each tree is then subdivided
|
||||
into several categories, including applications, benchmarks, regression tests,
|
||||
code that is strange grammatically, etc. These organizations should be
|
||||
relatively self explanatory.</p>
|
||||
|
||||
<p>
|
||||
Regardless of which test you're running, the results are printed on
|
||||
standard output and standard error. You can redirect these results to a
|
||||
file if you choose.
|
||||
</p><p>
|
||||
Some tests are known to fail. Some are bugs that we have not fixed yet;
|
||||
others are features that we haven't added yet (or may never add). In
|
||||
QMTest, the result for such tests will be XFAIL (eXpected FAILure). In
|
||||
this way, you can tell the difference between an expected and unexpected
|
||||
failure.
|
||||
</p><p>
|
||||
The Programs tests have no such feature as of this time. If the test
|
||||
passes, only warnings and other miscellaneous output will be generated.
|
||||
If a test fails, a large <program> FAILED message will be
|
||||
displayed. This will help you separate benign warnings from actual test
|
||||
failures.
|
||||
</p>
|
||||
</div>
|
||||
<p>In addition to the regular Programs tests, the Programs tree also provides a
|
||||
mechanism for compiling the programs in different ways. If the variable TEST is
|
||||
defined on the gmake command line, the test system will include a Makefile named
|
||||
<tt>TEST.<value of TEST variable>.Makefile</tt>. This Makefile can modify
|
||||
build rules to yield different results.</p>
|
||||
|
||||
<p>For example, the LLVM nightly tester uses <tt>TEST.nightly.Makefile</tt> to
|
||||
create the nightly test reports. To run the nightly tests, run <tt>gmake
|
||||
TEST=nightly</tt>.</p>
|
||||
|
||||
<p>There are several TEST Makefiles available in the tree. Some of them are
|
||||
designed for internal LLVM research and will not work outside of the LLVM
|
||||
research group. They may still be valuable, however, as a guide to writing your
|
||||
own TEST Makefile for any optimization or analysis passes that you develop with
|
||||
LLVM.</p>
|
||||
|
||||
</div>
|
||||
|
||||
<!--===============================================================-->
|
||||
<div class="doc_section"><a name="run">Running the LLVM Tests</a></div>
|
||||
<!--===============================================================-->
|
||||
|
||||
<div class="doc_text">
|
||||
|
||||
<p>First, all tests are executed within the LLVM object directory tree. They
|
||||
<i>are not</i> executed inside of the LLVM source tree. This is because the
|
||||
test suite creates temporary files during execution. </p>
|
||||
|
||||
<p>The master Makefile in llvm/test is capable of running both the QMTest driven
|
||||
tests and the Programs tests. By default, it will run all of the tests.</p>
|
||||
|
||||
<p>To run only the QMTest driven tests, run <tt>gmake qmtest</tt> at the
|
||||
command line in llvm/tests. To run a specific qmtest, suffix the test name with
|
||||
".t" when running gmake.</p>
|
||||
|
||||
<p>For example, to run the Regression.LLC tests, type <tt>gmake
|
||||
Regression.LLC.t</tt> in llvm/tests.</p>
|
||||
|
||||
<p>Note that the Makefiles in llvm/test/Features and llvm/test/Regression are
|
||||
gone. You must now use QMTest from the llvm/test directory to run them.</p>
|
||||
|
||||
<p>To run the Programs test, cd into the llvm/test/Programs directory and type
|
||||
<tt>gmake</tt>. Alternatively, you can type <tt>gmake TEST=<type>
|
||||
test</tt> to run one of the specialized tests in
|
||||
llvm/test/Programs/TEST.<type>.Makefile. For example, you could run the
|
||||
nightly tester tests using the following commands:</p>
|
||||
|
||||
<pre>
|
||||
% cd llvm/test/Programs
|
||||
% gmake TEST=nightly test
|
||||
</pre>
|
||||
|
||||
<p>Regardless of which test you're running, the results are printed on standard
|
||||
output and standard error. You can redirect these results to a file if you
|
||||
choose.</p>
|
||||
|
||||
<p>Some tests are known to fail. Some are bugs that we have not fixed yet;
|
||||
others are features that we haven't added yet (or may never add). In QMTest,
|
||||
the result for such tests will be XFAIL (eXpected FAILure). In this way, you
|
||||
can tell the difference between an expected and unexpected failure.</p>
|
||||
|
||||
<p>The Programs tests have no such feature as of this time. If the test passes,
|
||||
only warnings and other miscellaneous output will be generated. If a test
|
||||
fails, a large <program> FAILED message will be displayed. This will help
|
||||
you separate benign warnings from actual test failures.</p>
|
||||
|
||||
</div>
|
||||
|
||||
<!-- *********************************************************************** -->
|
||||
|
||||
<hr><font size="-1">
|
||||
<address>John T. Criswell</address>
|
||||
<a href="http://llvm.cs.uiuc.edu">The LLVM Compiler Infrastructure</a>
|
||||
<br>
|
||||
Last modified: $Date$
|
||||
</font>
|
||||
<hr>
|
||||
<address>
|
||||
<a href="http://jigsaw.w3.org/css-validator/check/referer"><img
|
||||
src="http://jigsaw.w3.org/css-validator/images/vcss" alt="Valid CSS!"></a>
|
||||
<a href="http://validator.w3.org/check/referer"><img
|
||||
src="http://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01!" /></a>
|
||||
|
||||
John T. Criswell<br>
|
||||
<a href="http://llvm.cs.uiuc.edu">The LLVM Compiler Infrastructure</a><br>
|
||||
Last modified: $Date$
|
||||
</address>
|
||||
</body>
|
||||
</html>
|
||||
|
Loading…
x
Reference in New Issue
Block a user