mirror of
https://github.com/c64scene-ar/llvm-6502.git
synced 2024-12-28 19:31:58 +00:00
For PR1319:
Rewrite much of the DejaGnu section to bring it in line with the new facilities in llvm.exp. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@36015 91177308-0d34-0410-b5e6-96231b3b80d8
This commit is contained in:
Reid Spencer
parent
d733ae8f5d
commit
bbb2a7a2cf
@ -275,81 +275,199 @@ location of these external programs is configured by the llvm-test
|
||||
<!--=========================================================================-->
|
||||
<div class="doc_section"><a name="dgstructure">DejaGNU Structure</a></div>
|
||||
<!--=========================================================================-->
|
||||
|
||||
<div class="doc_text">
|
||||
<p>The LLVM test suite is partially driven by DejaGNU and partially
|
||||
driven by GNU Make. Specifically, the Features and Regression tests
|
||||
are all driven by DejaGNU. The <tt>llvm-test</tt>
|
||||
module is currently driven by a set of Makefiles.</p>
|
||||
<p>The LLVM test suite is partially driven by DejaGNU and partially driven by
|
||||
GNU Make. Specifically, the Features and Regression tests are all driven by
|
||||
DejaGNU. The <tt>llvm-test</tt> module is currently driven by a set of
|
||||
Makefiles.</p>
|
||||
|
||||
<p>The DejaGNU structure is very simple, but does require some
|
||||
information to be set. This information is gathered via <tt>configure</tt> and
|
||||
is written to a file, <tt>site.exp</tt> in <tt>llvm/test</tt>. The
|
||||
<tt>llvm/test</tt>
|
||||
Makefile does this work for you.</p>
|
||||
<p>The DejaGNU structure is very simple, but does require some information to
|
||||
be set. This information is gathered via <tt>configure</tt> and is written
|
||||
to a file, <tt>site.exp</tt> in <tt>llvm/test</tt>. The <tt>llvm/test</tt>
|
||||
Makefile does this work for you.</p>
|
||||
|
||||
<p>In order for DejaGNU to work, each directory of tests must have a
|
||||
<tt>dg.exp</tt> file. This file is a program written in tcl that calls
|
||||
the <tt>llvm-runtests</tt> procedure on each test file. The
|
||||
llvm-runtests procedure is defined in
|
||||
<tt>llvm/test/lib/llvm-dg.exp</tt>. Any directory that contains only
|
||||
directories does not need the <tt>dg.exp</tt> file.</p>
|
||||
<p>In order for DejaGNU to work, each directory of tests must have a
|
||||
<tt>dg.exp</tt> file. DejaGNU looks for this file to determine how to run the
|
||||
tests. This file is just a Tcl script and it can do anything you want, but
|
||||
we've standardized it for the LLVM regression tests. It simply loads a Tcl
|
||||
library (<tt>test/lib/llvm.exp</tt>) and calls the <tt>llvm_runtests</tt>
|
||||
function defined in that library with a list of file names to run. The names
|
||||
are obtained by using Tcl's glob command. Any directory that contains only
|
||||
directories does not need the <tt>dg.exp</tt> file.</p>
|
||||
|
||||
<p>In order for a test to be run, it must contain information within
|
||||
the test file on how to run the test. These are called <tt>RUN</tt>
|
||||
lines. Run lines are specified in the comments of the test program
|
||||
using the keyword <tt>RUN</tt> followed by a colon, and lastly the
|
||||
commands to execute. These commands will be executed in a bash script,
|
||||
so any bash syntax is acceptable. You can specify as many RUN lines as
|
||||
necessary. Each RUN line translates to one line in the resulting bash
|
||||
script. Below is an example of legal RUN lines in a <tt>.ll</tt>
|
||||
file:</p>
|
||||
<pre>
|
||||
; RUN: llvm-as < %s | llvm-dis > %t1
|
||||
; RUN: llvm-dis < %s.bc-13 > %t2
|
||||
; RUN: diff %t1 %t2
|
||||
</pre>
|
||||
<p>There are a couple patterns within a <tt>RUN</tt> line that the
|
||||
llvm-runtest procedure looks for and replaces with the appropriate
|
||||
syntax:</p>
|
||||
<p>The <tt>llvm-runtests</tt> function lookas at each file that is passed to
|
||||
it and gathers any lines together that match "RUN:". This are the "RUN" lines
|
||||
that specify how the test is to be run. So, each test script must contain
|
||||
RUN lines if it is to do anything. If there are no RUN lines, the
|
||||
<tt>llvm-runtests</tt> function will issue an error and the test will
|
||||
fail.</p>
|
||||
|
||||
<dl style="margin-left: 25px">
|
||||
<dt>%p</dt>
|
||||
<dd>The path to the source directory. This is for locating
|
||||
any supporting files that are not generated by the test, but used by
|
||||
the test.</dd>
|
||||
<dt>%s</dt>
|
||||
<dd>The test file.</dd>
|
||||
<p>RUN lines are specified in the comments of the test program using the
|
||||
keyword <tt>RUN</tt> followed by a colon, and lastly the command (pipeline)
|
||||
to execute. Together, these lines form the "script" that
|
||||
<tt>llvm-runtests</tt> executes to run the test case. The syntax of the
|
||||
RUN lines is similar to a shell's syntax for pipelines including I/O
|
||||
redirection and variable substitution. However, even though these lines
|
||||
may <i>look</i> like a shell script, they are not. RUN lines are interpreted
|
||||
directly by the Tcl <tt>exec</tt> command. They are never executed by a
|
||||
shell. Consequently the syntax differs from normal shell script syntax in a
|
||||
few ways. You can specify as many RUN lines as needed.</p>
|
||||
|
||||
<dt>%t</dt>
|
||||
<dd>Temporary filename: testscript.test_filename.tmp, where
|
||||
test_filename is the name of the test file. All temporary files are
|
||||
placed in the Output directory within the directory the test is
|
||||
located.</dd>
|
||||
<p>Each RUN line is executed on its own, distinct from other lines unless
|
||||
its last character is <tt>\</tt>. This continuation character causes the RUN
|
||||
line to be concatenated with the next one. In this way you can build up long
|
||||
pipelines of commands without making huge line lengths. The lines ending in
|
||||
<tt>\</tt> are concatenated until a RUN line that doesn't end in <tt>\</tt> is
|
||||
found. This concatenated set or RUN lines then constitutes one execution.
|
||||
Tcl will substitute variables and arrange for the pipeline to be executed. If
|
||||
any process in the pipeline fails, the entire line (and test case) fails too.
|
||||
</p>
|
||||
|
||||
<dt>%prcontext</dt>
|
||||
<dd>Path to a script that performs grep -C. Use this since not all
|
||||
platforms support grep -C.</dd>
|
||||
<p> Below is an example of legal RUN lines in a <tt>.ll</tt> file:</p>
|
||||
<pre>
|
||||
; RUN: llvm-as < %s | llvm-dis > %t1
|
||||
; RUN: llvm-dis < %s.bc-13 > %t2
|
||||
; RUN: diff %t1 %t2
|
||||
</pre>
|
||||
</div>
|
||||
|
||||
<dt>%llvmgcc</dt> <dd>Full path to the llvm-gcc executable.</dd>
|
||||
<dt>%llvmgxx</dt> <dd>Full path to the llvm-g++ executable.</dd>
|
||||
</dl>
|
||||
<!-- _______________________________________________________________________ -->
|
||||
<div class="doc_subsection"><a name="dgvars">Vars And Substitutions</a></div>
|
||||
<div class="doc_text">
|
||||
<p>With a RUN line there are a number of substitutions that are permitted. In
|
||||
general, any Tcl variable that is available in the <tt>substitute</tt>
|
||||
function (in <tt>test/lib/llvm.exp</tt>) can be substituted into a RUN line.
|
||||
To make a substitution just write the variable's name preceded by a $.
|
||||
Additionally, for compatibility reasons with previous versions of the test
|
||||
library, certain names can be accessed with an alternate syntax: a % prefix.
|
||||
These alternates are deprecated and may go away in a future version.
|
||||
</p>
|
||||
Here are the available variable names. The alternate syntax is listed in
|
||||
parentheses.</p>
|
||||
<dl style="margin-left: 25px">
|
||||
<dt><b>$test</b> (%s)</dt>
|
||||
<dd>The full path to the test case's source. This is suitable for passing
|
||||
on the command line as the input to an llvm tool.</dd>
|
||||
<dt><b>$srcdir</b></dt>
|
||||
<dd>The source directory from where the "<tt>make check</tt>" was run.</dd>
|
||||
<dt><b>objdir</b></dt>
|
||||
<dd>The object directory that corresponds to the </tt>$srcdir</tt>.</dd>
|
||||
<dt><b>subdir</b></dt>
|
||||
<dd>A partial path from the <tt>test</tt> directory that contains the
|
||||
sub-directory that contains the test source being executed.</dd>
|
||||
<dt><b>srcroot</b></dt>
|
||||
<dd>The root directory of the LLVM src tree.</dd>
|
||||
<dt><b>objroot</b></dt>
|
||||
<dd>The root directory of the LLVM object tree. This could be the same
|
||||
as the srcroot.</dd>
|
||||
<dt><b>path</b><dt>
|
||||
<dd>The path to the directory that contains the test case source. This is
|
||||
for locating any supporting files that are not generated by the test, but
|
||||
used by the test.</dd>
|
||||
<dt><b>tmp</b></dt>
|
||||
<dd>The path to a temporary file name that could be used for this test case.
|
||||
The file name won't conflict with other test cases. You can append to it if
|
||||
you need multiple temporaries. This is useful as the destination of some
|
||||
redirected output.</dd>
|
||||
<dt><b>llvmlibsdir</b> (%llvmlibsdir)</dt>
|
||||
<dd>The directory where the LLVM libraries are located.</dd>
|
||||
<dt><b>target_triplet</b> (%target_triplet)</dt>
|
||||
<dd>The target triplet that corresponds to the current host machine (the one
|
||||
running the test cases). This should probably be called "host".<dd>
|
||||
<dt><b>prcontext</b> (%prcontext)</dt>
|
||||
<dd>Path to the prcontext tcl script that prints some context around a
|
||||
line that matches a pattern. This isn't strictly necessary as the test suite
|
||||
is run with its PATH altered to include the test/Scripts directory where
|
||||
the prcontext script is located. Note that this script is similar to
|
||||
<tt>grep -C</tt> but you should use the <tt>prcontext</tt> script because
|
||||
not all platforms support <tt>grep -C</tt>.</dd>
|
||||
<dt><b>llvmgcc</b> (%llvmgcc)</dt>
|
||||
<dd>The full path to the <tt>llvm-gcc</tt> executable as specified in the
|
||||
configured LLVM environment</dd>
|
||||
<dt><b>llvmgxx</b> (%llvmgxx)</dt>
|
||||
<dd>The full path to the <tt>llvm-gxx</tt> executable as specified in the
|
||||
configured LLVM environment</dd>
|
||||
<dt><b>llvmgcc_version</b> (%llvmgcc_version)</dt>
|
||||
<dd>The full version number of the <tt>llvm-gcc</tt> executable.</dd>
|
||||
<dt><b>llvmgccmajvers</b> (%llvmgccmajvers)</dt>
|
||||
<dd>The major version number of the <tt>llvm-gcc</tt> executable.</dd>
|
||||
<dt><b>gccpath</b></dt>
|
||||
<dd>The full path to the C compiler used to <i>build </i> LLVM. Note that
|
||||
this might not be gcc.</dd>
|
||||
<dt><b>gxxpath</b></dt>
|
||||
<dd>The full path to the C++ compiler used to <i>build </i> LLVM. Note that
|
||||
this might not be g++.</dd>
|
||||
<dt><b>compile_c</b> (%compile_c)</dt>
|
||||
<dd>The full command line used to compile LLVM C source code. This has all
|
||||
the configured -I, -D and optimization options.</dd>
|
||||
<dt><b>compile_cxx</b> (%compile_cxx)</dt>
|
||||
<dd>The full command used to compile LLVM C++ source code. This has
|
||||
all the configured -I, -D and optimization options.</dd>
|
||||
<dt><b>link</b> (%link)</dt>
|
||||
<dd>This full link command used to link LLVM executables. This has all the
|
||||
configured -I, -L and -l options.</dd>
|
||||
<dt><b>shlibext</b> (%shlibext)</dt>
|
||||
<dd>The suffix for the host platforms share library (dll) files. This
|
||||
includes the period as the first character.</dd>
|
||||
</dl>
|
||||
<p>To add more variables, two things need to be changed. First, add a line in
|
||||
the <tt>test/Makefile</tt> that creates the <tt>site.exp</tt> file. This will
|
||||
"set" the variable as a global in the site.exp file. Second, in the
|
||||
<tt>test/lib/llvm.exp</tt> file, in the substitute proc, add the variable name
|
||||
to the list of "global" declarations at the beginning of the proc. That's it,
|
||||
the variable can then be used in test scripts.</p>
|
||||
</div>
|
||||
|
||||
<!-- _______________________________________________________________________ -->
|
||||
<div class="doc_subsection"><a name="dgfeatures">Other Features</a></div>
|
||||
<div class="doc_text">
|
||||
<p>To make RUN line writing easier, there are several shell scripts located
|
||||
in the <tt>llvm/test/Scripts</tt> directory. For example:</p>
|
||||
<dl>
|
||||
<dt><b>ignore</b></dt>
|
||||
<dd>This script runs its arguments and then always returns 0. This is useful
|
||||
in cases where the test needs to cause a tool to generate an error (e.g. to
|
||||
check the error output). However, any program in a pipeline that returns a
|
||||
non-zero result will cause the test to fail. This script overcomes that
|
||||
issue and nicely documents that the test case is purposefully ignoring the
|
||||
result code of the tool</dd>
|
||||
<dt><b>not</b></dt>
|
||||
<dd>This script runs its arguments and then inverts the result code from
|
||||
it. Zero result codes become 1. Non-zero result codes become 0. This is
|
||||
useful to invert the result of a grep. For example "not grep X" means
|
||||
succeed only if you don't find X in the input.</dd>
|
||||
</dl>
|
||||
|
||||
<p>There are also several scripts in the llvm/test/Scripts directory
|
||||
that you might find useful when writing <tt>RUN</tt> lines.</p>
|
||||
<p>Sometimes it is necessary to mark a test case as "expected fail" or XFAIL.
|
||||
You can easily mark a test as XFAIL just by including <tt>XFAIL: </tt> on a
|
||||
line near the top of the file. This signals that the test case should succeed
|
||||
if the test fails. Such test cases are counted separately by DejaGnu. To
|
||||
specify an expected fail, use the XFAIL keyword in the comments of the test
|
||||
program followed by a colon and one or more regular expressions (separated by
|
||||
a comma). The regular expressions allow you to XFAIL the test conditionally
|
||||
by host platform. The regular expressions following the : are matched against
|
||||
the target triplet or llvmgcc version number for the host machine. If there is
|
||||
a match, the test is expected to fail. If not, the test is expected to
|
||||
succeed. To XFAIL everywhere just specify <tt>XFAIL: *</tt>. When matching
|
||||
the llvm-gcc version, you can specify the major (e.g. 3) or full version
|
||||
(i.e. 3.4) number. Here is an example of an <tt>XFAIL</tt> line:</p>
|
||||
<pre>
|
||||
; XFAIL: darwin,sun,llvmgcc4
|
||||
</pre>
|
||||
|
||||
<p>Lastly, you can easily mark a test that is expected to fail on a
|
||||
specific platform or with a specific version of llvmgcc by using the
|
||||
<tt>XFAIL</tt> keyword. Xfail lines are
|
||||
specified in the comments of the test program using <tt>XFAIL</tt>,
|
||||
followed by a colon, and one or more regular expressions (separated by
|
||||
a comma) that will match against the target triplet or llvmgcc version for the
|
||||
machine. You can use * to match all targets. You can specify the major or full
|
||||
version (i.e. 3.4) for llvmgcc. Here is an example of an
|
||||
<tt>XFAIL</tt> line:</p>
|
||||
<pre>
|
||||
; XFAIL: darwin,sun,llvmgcc4
|
||||
</pre>
|
||||
<p>To make the output more useful, the <tt>llvm_runtest</tt> function wil
|
||||
scan the lines of the test case for ones that contain a pattern that matches
|
||||
PR[0-9]+. This is the syntax for specifying a PR (Problem Report) number that
|
||||
is related to the test case. The numer after "PR" specifies the LLVM bugzilla
|
||||
number. When a PR number is specified, it will be used in the pass/fail
|
||||
reporting. This is useful to quickly get some context when a test fails.</p>
|
||||
|
||||
<p>Finally, any line that contains "END." will cause the special
|
||||
interpretation of lines to terminate. This is generally done right after the
|
||||
last RUN: line. This has two side effects: (a) it prevents special
|
||||
interpretation of lines that are part of the test program, not the
|
||||
instructions to the test case, and (b) it speeds things up for really big test
|
||||
cases by avoiding interpretation of the remainder of the file.</p>
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user