For PR1319:

Rewrite much of the DejaGnu section to bring it in line with the new
facilities in llvm.exp.


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@36015 91177308-0d34-0410-b5e6-96231b3b80d8
This commit is contained in:
Reid Spencer 2007-04-14 21:46:15 +00:00
parent d733ae8f5d
commit bbb2a7a2cf

View File

@ -275,81 +275,199 @@ location of these external programs is configured by the llvm-test
<!--=========================================================================-->
<div class="doc_section"><a name="dgstructure">DejaGNU Structure</a></div>
<!--=========================================================================-->
<div class="doc_text">
<p>The LLVM test suite is partially driven by DejaGNU and partially
driven by GNU Make. Specifically, the Features and Regression tests
are all driven by DejaGNU. The <tt>llvm-test</tt>
module is currently driven by a set of Makefiles.</p>
<p>The LLVM test suite is partially driven by DejaGNU and partially driven by
GNU Make. Specifically, the Features and Regression tests are all driven by
DejaGNU. The <tt>llvm-test</tt> module is currently driven by a set of
Makefiles.</p>
<p>The DejaGNU structure is very simple, but does require some
information to be set. This information is gathered via <tt>configure</tt> and
is written to a file, <tt>site.exp</tt> in <tt>llvm/test</tt>. The
<tt>llvm/test</tt>
Makefile does this work for you.</p>
<p>The DejaGNU structure is very simple, but does require some information to
be set. This information is gathered via <tt>configure</tt> and is written
to a file, <tt>site.exp</tt> in <tt>llvm/test</tt>. The <tt>llvm/test</tt>
Makefile does this work for you.</p>
<p>In order for DejaGNU to work, each directory of tests must have a
<tt>dg.exp</tt> file. This file is a program written in tcl that calls
the <tt>llvm-runtests</tt> procedure on each test file. The
llvm-runtests procedure is defined in
<tt>llvm/test/lib/llvm-dg.exp</tt>. Any directory that contains only
directories does not need the <tt>dg.exp</tt> file.</p>
<p>In order for DejaGNU to work, each directory of tests must have a
<tt>dg.exp</tt> file. DejaGNU looks for this file to determine how to run the
tests. This file is just a Tcl script and it can do anything you want, but
we've standardized it for the LLVM regression tests. It simply loads a Tcl
library (<tt>test/lib/llvm.exp</tt>) and calls the <tt>llvm_runtests</tt>
function defined in that library with a list of file names to run. The names
are obtained by using Tcl's glob command. Any directory that contains only
directories does not need the <tt>dg.exp</tt> file.</p>
<p>In order for a test to be run, it must contain information within
the test file on how to run the test. These are called <tt>RUN</tt>
lines. Run lines are specified in the comments of the test program
using the keyword <tt>RUN</tt> followed by a colon, and lastly the
commands to execute. These commands will be executed in a bash script,
so any bash syntax is acceptable. You can specify as many RUN lines as
necessary. Each RUN line translates to one line in the resulting bash
script. Below is an example of legal RUN lines in a <tt>.ll</tt>
file:</p>
<pre>
; RUN: llvm-as < %s | llvm-dis > %t1
; RUN: llvm-dis < %s.bc-13 > %t2
; RUN: diff %t1 %t2
</pre>
<p>There are a couple patterns within a <tt>RUN</tt> line that the
llvm-runtest procedure looks for and replaces with the appropriate
syntax:</p>
<p>The <tt>llvm-runtests</tt> function lookas at each file that is passed to
it and gathers any lines together that match "RUN:". This are the "RUN" lines
that specify how the test is to be run. So, each test script must contain
RUN lines if it is to do anything. If there are no RUN lines, the
<tt>llvm-runtests</tt> function will issue an error and the test will
fail.</p>
<dl style="margin-left: 25px">
<dt>%p</dt>
<dd>The path to the source directory. This is for locating
any supporting files that are not generated by the test, but used by
the test.</dd>
<dt>%s</dt>
<dd>The test file.</dd>
<p>RUN lines are specified in the comments of the test program using the
keyword <tt>RUN</tt> followed by a colon, and lastly the command (pipeline)
to execute. Together, these lines form the "script" that
<tt>llvm-runtests</tt> executes to run the test case. The syntax of the
RUN lines is similar to a shell's syntax for pipelines including I/O
redirection and variable substitution. However, even though these lines
may <i>look</i> like a shell script, they are not. RUN lines are interpreted
directly by the Tcl <tt>exec</tt> command. They are never executed by a
shell. Consequently the syntax differs from normal shell script syntax in a
few ways. You can specify as many RUN lines as needed.</p>
<dt>%t</dt>
<dd>Temporary filename: testscript.test_filename.tmp, where
test_filename is the name of the test file. All temporary files are
placed in the Output directory within the directory the test is
located.</dd>
<p>Each RUN line is executed on its own, distinct from other lines unless
its last character is <tt>\</tt>. This continuation character causes the RUN
line to be concatenated with the next one. In this way you can build up long
pipelines of commands without making huge line lengths. The lines ending in
<tt>\</tt> are concatenated until a RUN line that doesn't end in <tt>\</tt> is
found. This concatenated set or RUN lines then constitutes one execution.
Tcl will substitute variables and arrange for the pipeline to be executed. If
any process in the pipeline fails, the entire line (and test case) fails too.
</p>
<dt>%prcontext</dt>
<dd>Path to a script that performs grep -C. Use this since not all
platforms support grep -C.</dd>
<p> Below is an example of legal RUN lines in a <tt>.ll</tt> file:</p>
<pre>
; RUN: llvm-as &lt; %s | llvm-dis &gt; %t1
; RUN: llvm-dis &lt; %s.bc-13 &gt; %t2
; RUN: diff %t1 %t2
</pre>
</div>
<dt>%llvmgcc</dt> <dd>Full path to the llvm-gcc executable.</dd>
<dt>%llvmgxx</dt> <dd>Full path to the llvm-g++ executable.</dd>
</dl>
<!-- _______________________________________________________________________ -->
<div class="doc_subsection"><a name="dgvars">Vars And Substitutions</a></div>
<div class="doc_text">
<p>With a RUN line there are a number of substitutions that are permitted. In
general, any Tcl variable that is available in the <tt>substitute</tt>
function (in <tt>test/lib/llvm.exp</tt>) can be substituted into a RUN line.
To make a substitution just write the variable's name preceded by a $.
Additionally, for compatibility reasons with previous versions of the test
library, certain names can be accessed with an alternate syntax: a % prefix.
These alternates are deprecated and may go away in a future version.
</p>
Here are the available variable names. The alternate syntax is listed in
parentheses.</p>
<dl style="margin-left: 25px">
<dt><b>$test</b> (%s)</dt>
<dd>The full path to the test case's source. This is suitable for passing
on the command line as the input to an llvm tool.</dd>
<dt><b>$srcdir</b></dt>
<dd>The source directory from where the "<tt>make check</tt>" was run.</dd>
<dt><b>objdir</b></dt>
<dd>The object directory that corresponds to the </tt>$srcdir</tt>.</dd>
<dt><b>subdir</b></dt>
<dd>A partial path from the <tt>test</tt> directory that contains the
sub-directory that contains the test source being executed.</dd>
<dt><b>srcroot</b></dt>
<dd>The root directory of the LLVM src tree.</dd>
<dt><b>objroot</b></dt>
<dd>The root directory of the LLVM object tree. This could be the same
as the srcroot.</dd>
<dt><b>path</b><dt>
<dd>The path to the directory that contains the test case source. This is
for locating any supporting files that are not generated by the test, but
used by the test.</dd>
<dt><b>tmp</b></dt>
<dd>The path to a temporary file name that could be used for this test case.
The file name won't conflict with other test cases. You can append to it if
you need multiple temporaries. This is useful as the destination of some
redirected output.</dd>
<dt><b>llvmlibsdir</b> (%llvmlibsdir)</dt>
<dd>The directory where the LLVM libraries are located.</dd>
<dt><b>target_triplet</b> (%target_triplet)</dt>
<dd>The target triplet that corresponds to the current host machine (the one
running the test cases). This should probably be called "host".<dd>
<dt><b>prcontext</b> (%prcontext)</dt>
<dd>Path to the prcontext tcl script that prints some context around a
line that matches a pattern. This isn't strictly necessary as the test suite
is run with its PATH altered to include the test/Scripts directory where
the prcontext script is located. Note that this script is similar to
<tt>grep -C</tt> but you should use the <tt>prcontext</tt> script because
not all platforms support <tt>grep -C</tt>.</dd>
<dt><b>llvmgcc</b> (%llvmgcc)</dt>
<dd>The full path to the <tt>llvm-gcc</tt> executable as specified in the
configured LLVM environment</dd>
<dt><b>llvmgxx</b> (%llvmgxx)</dt>
<dd>The full path to the <tt>llvm-gxx</tt> executable as specified in the
configured LLVM environment</dd>
<dt><b>llvmgcc_version</b> (%llvmgcc_version)</dt>
<dd>The full version number of the <tt>llvm-gcc</tt> executable.</dd>
<dt><b>llvmgccmajvers</b> (%llvmgccmajvers)</dt>
<dd>The major version number of the <tt>llvm-gcc</tt> executable.</dd>
<dt><b>gccpath</b></dt>
<dd>The full path to the C compiler used to <i>build </i> LLVM. Note that
this might not be gcc.</dd>
<dt><b>gxxpath</b></dt>
<dd>The full path to the C++ compiler used to <i>build </i> LLVM. Note that
this might not be g++.</dd>
<dt><b>compile_c</b> (%compile_c)</dt>
<dd>The full command line used to compile LLVM C source code. This has all
the configured -I, -D and optimization options.</dd>
<dt><b>compile_cxx</b> (%compile_cxx)</dt>
<dd>The full command used to compile LLVM C++ source code. This has
all the configured -I, -D and optimization options.</dd>
<dt><b>link</b> (%link)</dt>
<dd>This full link command used to link LLVM executables. This has all the
configured -I, -L and -l options.</dd>
<dt><b>shlibext</b> (%shlibext)</dt>
<dd>The suffix for the host platforms share library (dll) files. This
includes the period as the first character.</dd>
</dl>
<p>To add more variables, two things need to be changed. First, add a line in
the <tt>test/Makefile</tt> that creates the <tt>site.exp</tt> file. This will
"set" the variable as a global in the site.exp file. Second, in the
<tt>test/lib/llvm.exp</tt> file, in the substitute proc, add the variable name
to the list of "global" declarations at the beginning of the proc. That's it,
the variable can then be used in test scripts.</p>
</div>
<!-- _______________________________________________________________________ -->
<div class="doc_subsection"><a name="dgfeatures">Other Features</a></div>
<div class="doc_text">
<p>To make RUN line writing easier, there are several shell scripts located
in the <tt>llvm/test/Scripts</tt> directory. For example:</p>
<dl>
<dt><b>ignore</b></dt>
<dd>This script runs its arguments and then always returns 0. This is useful
in cases where the test needs to cause a tool to generate an error (e.g. to
check the error output). However, any program in a pipeline that returns a
non-zero result will cause the test to fail. This script overcomes that
issue and nicely documents that the test case is purposefully ignoring the
result code of the tool</dd>
<dt><b>not</b></dt>
<dd>This script runs its arguments and then inverts the result code from
it. Zero result codes become 1. Non-zero result codes become 0. This is
useful to invert the result of a grep. For example "not grep X" means
succeed only if you don't find X in the input.</dd>
</dl>
<p>There are also several scripts in the llvm/test/Scripts directory
that you might find useful when writing <tt>RUN</tt> lines.</p>
<p>Sometimes it is necessary to mark a test case as "expected fail" or XFAIL.
You can easily mark a test as XFAIL just by including <tt>XFAIL: </tt> on a
line near the top of the file. This signals that the test case should succeed
if the test fails. Such test cases are counted separately by DejaGnu. To
specify an expected fail, use the XFAIL keyword in the comments of the test
program followed by a colon and one or more regular expressions (separated by
a comma). The regular expressions allow you to XFAIL the test conditionally
by host platform. The regular expressions following the : are matched against
the target triplet or llvmgcc version number for the host machine. If there is
a match, the test is expected to fail. If not, the test is expected to
succeed. To XFAIL everywhere just specify <tt>XFAIL: *</tt>. When matching
the llvm-gcc version, you can specify the major (e.g. 3) or full version
(i.e. 3.4) number. Here is an example of an <tt>XFAIL</tt> line:</p>
<pre>
; XFAIL: darwin,sun,llvmgcc4
</pre>
<p>Lastly, you can easily mark a test that is expected to fail on a
specific platform or with a specific version of llvmgcc by using the
<tt>XFAIL</tt> keyword. Xfail lines are
specified in the comments of the test program using <tt>XFAIL</tt>,
followed by a colon, and one or more regular expressions (separated by
a comma) that will match against the target triplet or llvmgcc version for the
machine. You can use * to match all targets. You can specify the major or full
version (i.e. 3.4) for llvmgcc. Here is an example of an
<tt>XFAIL</tt> line:</p>
<pre>
; XFAIL: darwin,sun,llvmgcc4
</pre>
<p>To make the output more useful, the <tt>llvm_runtest</tt> function wil
scan the lines of the test case for ones that contain a pattern that matches
PR[0-9]+. This is the syntax for specifying a PR (Problem Report) number that
is related to the test case. The numer after "PR" specifies the LLVM bugzilla
number. When a PR number is specified, it will be used in the pass/fail
reporting. This is useful to quickly get some context when a test fails.</p>
<p>Finally, any line that contains "END." will cause the special
interpretation of lines to terminate. This is generally done right after the
last RUN: line. This has two side effects: (a) it prevents special
interpretation of lines that are part of the test program, not the
instructions to the test case, and (b) it speeds things up for really big test
cases by avoiding interpretation of the remainder of the file.</p>
</div>