From da51c4cbd53943d8473562231b7753dadd89421c Mon Sep 17 00:00:00 2001 From: Bill Wendling Date: Sat, 22 Sep 2007 09:16:44 +0000 Subject: [PATCH] Formatting changes. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@42223 91177308-0d34-0410-b5e6-96231b3b80d8 --- docs/TestingGuide.html | 198 ++++++++++++++++++++++++++++++----------- 1 file changed, 147 insertions(+), 51 deletions(-) diff --git a/docs/TestingGuide.html b/docs/TestingGuide.html index cb880373de3..af72c845a44 100644 --- a/docs/TestingGuide.html +++ b/docs/TestingGuide.html @@ -116,20 +116,30 @@ the llvm-test directory will be automatically configured. Alternatively, you can configure the test-suite module manually.

To run all of the simple tests in LLVM using DejaGNU, use the master Makefile in the llvm/test directory:

+ +
 % gmake -C llvm/test
-or
+
+ +

or

+ +
 % gmake check
+

To run only a subdirectory of tests in llvm/test using DejaGNU (ie. Regression/Transforms), just set the TESTSUITE variable to the path of the subdirectory (relative to llvm/test):

+ +
 % gmake -C llvm/test TESTSUITE=Regression/Transforms
+

Note: If you are running the tests with objdir != subdir, you must have run the complete testsuite before you can specify a @@ -138,6 +148,7 @@ subdirectory.

To run the comprehensive test suite (tests that compile and execute whole programs), run the llvm-test tests:

+
 % cd llvm/projects
 % svn co http://llvm.org/svn/llvm-project/test-suite/trunk llvm-test
@@ -145,6 +156,7 @@ programs), run the llvm-test tests:

 % ./configure --with-llvmsrc=$LLVM_SRC_ROOT --with-llvmobj=$LLVM_OBJ_ROOT
 % gmake
+
@@ -324,11 +336,14 @@ location of these external programs is configured by the llvm-test

Below is an example of legal RUN lines in a .ll file:

- 
-  ; RUN: llvm-as < %s | llvm-dis > %t1
-  ; RUN: llvm-dis < %s.bc-13 > %t2
-  ; RUN: diff %t1 %t2
-
+ +
+
+; RUN: llvm-as < %s | llvm-dis > %t1
+; RUN: llvm-dis < %s.bc-13 > %t2
+; RUN: diff %t1 %t2
+
+

As with a Unix shell, the RUN: lines permit pipelines and I/O redirection to be used. However, the usage is slightly different than for Bash. To check @@ -351,43 +366,66 @@ location of these external programs is configured by the llvm-test

There are some quoting rules that you must pay attention to when writing your RUN lines. In general nothing needs to be quoted. Tcl won't strip off any ' or " so they will get passed to the invoked program. For example:

- 
-     ... | grep 'find this string'
-
+ +
+
+... | grep 'find this string'
+
+
+

This will fail because the ' characters are passed to grep. This would instruction grep to look for 'find in the files this and string'. To avoid this use curly braces to tell Tcl that it should treat everything enclosed as one value. So our example would become:

- 
-     ... | grep {find this string}
-
+ +
+
+... | grep {find this string}
+
+
+

Additionally, the characters [ and ] are treated specially by Tcl. They tell Tcl to interpret the content as a command to execute. Since these characters are often used in regular expressions this can have disastrous results and cause the entire test run in a directory to fail. For example, a common idiom is to look for some basicblock number:

- 
-     ... | grep bb[2-8]
-
+ +
+
+... | grep bb[2-8]
+
+
+

This, however, will cause Tcl to fail because its going to try to execute a program named "2-8". Instead, what you want is this:

- 
-     ... | grep {bb\[2-8\]}
-
+ +
+
+... | grep {bb\[2-8\]}
+
+
+

Finally, if you need to pass the \ character down to a program, then it must be doubled. This is another Tcl special character. So, suppose you had: - 

-     ... | grep 'i32\*'
-
+ +
+
+... | grep 'i32\*'
+
+
+

This will fail to match what you want (a pointer to i32). First, the ' do not get stripped off. Second, the \ gets stripped off by Tcl so what grep sees is: 'i32*'. That's not likely to match anything. To resolve this you must use \\ and the {}, like this:

- 
-     ... | grep {i32\\*}
-
+ +
+
+... | grep {i32\\*}
+
+
@@ -404,36 +442,47 @@ location of these external programs is configured by the llvm-test

Here are the available variable names. The alternate syntax is listed in parentheses.

+
$test (%s)
The full path to the test case's source. This is suitable for passing on the command line as the input to an llvm tool.
+
$srcdir
The source directory from where the "make check" was run.
+
objdir
The object directory that corresponds to the $srcdir.
+
subdir
A partial path from the test directory that contains the sub-directory that contains the test source being executed.
+
srcroot
The root directory of the LLVM src tree.
+
objroot
The root directory of the LLVM object tree. This could be the same as the srcroot.
+
path
The path to the directory that contains the test case source. This is for locating any supporting files that are not generated by the test, but used by the test.
+
tmp
The path to a temporary file name that could be used for this test case. The file name won't conflict with other test cases. You can append to it if you need multiple temporaries. This is useful as the destination of some redirected output.
+
llvmlibsdir (%llvmlibsdir)
The directory where the LLVM libraries are located.
+
target_triplet (%target_triplet)
The target triplet that corresponds to the current host machine (the one running the test cases). This should probably be called "host".
+
prcontext (%prcontext)
Path to the prcontext tcl script that prints some context around a line that matches a pattern. This isn't strictly necessary as the test suite @@ -441,31 +490,41 @@ location of these external programs is configured by the llvm-test the prcontext script is located. Note that this script is similar to grep -C but you should use the prcontext script because not all platforms support grep -C.
+
llvmgcc (%llvmgcc)
The full path to the llvm-gcc executable as specified in the configured LLVM environment
+
llvmgxx (%llvmgxx)
The full path to the llvm-gxx executable as specified in the configured LLVM environment
+
llvmgcc_version (%llvmgcc_version)
The full version number of the llvm-gcc executable.
+
llvmgccmajvers (%llvmgccmajvers)
The major version number of the llvm-gcc executable.
+
gccpath
The full path to the C compiler used to build LLVM. Note that this might not be gcc.
+
gxxpath
The full path to the C++ compiler used to build LLVM. Note that this might not be g++.
+
compile_c (%compile_c)
The full command line used to compile LLVM C source code. This has all the configured -I, -D and optimization options.
+
compile_cxx (%compile_cxx)
The full command used to compile LLVM C++ source code. This has all the configured -I, -D and optimization options.
+
link (%link)
This full link command used to link LLVM executables. This has all the configured -I, -L and -l options.
+
shlibext (%shlibext)
The suffix for the host platforms share library (dll) files. This includes the period as the first character.
@@ -491,6 +550,7 @@ location of these external programs is configured by the llvm-test non-zero result will cause the test to fail. This script overcomes that issue and nicely documents that the test case is purposefully ignoring the result code of the tool +
not
This script runs its arguments and then inverts the result code from it. Zero result codes become 1. Non-zero result codes become 0. This is @@ -511,9 +571,12 @@ location of these external programs is configured by the llvm-test succeed. To XFAIL everywhere just specify XFAIL: *. When matching the llvm-gcc version, you can specify the major (e.g. 3) or full version (i.e. 3.4) number. Here is an example of an XFAIL line:

- 
-   ; XFAIL: darwin,sun,llvmgcc4
-
+ +
+
+; XFAIL: darwin,sun,llvmgcc4
+
+

To make the output more useful, the llvm_runtest function wil scan the lines of the test case for ones that contain a pattern that matches @@ -573,12 +636,14 @@ specify the following configuration options:

uses the default value /home/vadve/shared/benchmarks/speccpu2000/benchspec.

+

--enable-spec95
--enable-spec95=<directory>
Enable the use of SPEC95 when testing LLVM. It is similar to the --enable-spec2000 option.

+

--enable-povray
--enable-povray=<directory>
@@ -598,12 +663,12 @@ specify the following configuration options:

are not executed inside of the LLVM source tree. This is because the test suite creates temporary files during execution.

-

The master Makefile in llvm/test is capable of running only the DejaGNU -driven tests. By default, it will run all of these tests.

+

The master Makefile in llvm/test is capable of running only the +DejaGNU driven tests. By default, it will run all of these tests.

To run only the DejaGNU driven tests, run gmake at the command line in llvm/test. To run a specific directory of tests, use -the TESTSUITE variable. +the TESTSUITE variable.

For example, to run the Regression tests, type @@ -613,40 +678,71 @@ the TESTSUITE variable. llvm/test/Regression. You must use DejaGNU from the llvm/test directory to run them.

-

To run the llvm-test suite, you need to use the following steps: -

+

To run the llvm-test suite, you need to use the following steps:

+
    -
  1. cd into the llvm/projects directory
    2. -
  2. check out the test-suite module with:
    - svn co http://llvm.org/svn/llvm-project/test-suite/trunk llvm-test
    - This will get the test suite into llvm/projects/llvm-test
    3. -
  3. configure the test suite. You can do this one of two ways: +
  4. cd into the llvm/projects directory
    5. + +

  5. Check out the test-suite module with:

    + +
    +
    +% svn co http://llvm.org/svn/llvm-project/test-suite/trunk llvm-test
+
    +
    + +

    This will get the test suite into llvm/projects/llvm-test

    + +

  6. Configure the test suite. You can do this one of two ways:

    +
      -
    1. Use the regular llvm configure:
      - cd $LLVM_OBJ_ROOT ; $LLVM_SRC_ROOT/configure
      - This will ensure that the projects/llvm-test directory is also - properly configured.
      2. -
    2. Use the configure script found in the llvm-test source - directory:
      - $LLVM_SRC_ROOT/projects/llvm-test/configure - --with-llvmsrc=$LLVM_SRC_ROOT --with-llvmobj=$LLVM_OBJ_ROOT +
    3. Use the regular llvm configure:

      + +
      +
      +% cd $LLVM_OBJ_ROOT ; $LLVM_SRC_ROOT/configure
+
      +
      + +

      This will ensure that the projects/llvm-test directory is + also properly configured.

      4. + +

    4. Use the configure script found in the llvm-test + source directory:

      + +
      +
      +% $LLVM_SRC_ROOT/projects/llvm-test/configure \
+  --with-llvmsrc=$LLVM_SRC_ROOT               \
+  --with-llvmobj=$LLVM_OBJ_ROOT
+
      +
    -
  7. gmake
    8. +
  8. gmake

Note that the second and third steps only need to be done once. After you have the suite checked out and configured, you don't need to do it again (unless the test code or configure script changes).

To make a specialized test (use one of the -llvm-test/TEST.<type>.Makefiles), just run:
-gmake TEST=<type> test
For example, you could run the -nightly tester tests using the following commands:

+llvm-test/TEST.<type>.Makefiles), just run:

+
- % cd llvm/projects/llvm-test
- % gmake TEST=nightly test
+% gmake TEST=<type> test
+
+ +

For example, you could run the nightly tester tests using the following +commands:

+ +
+
+% cd llvm/projects/llvm-test
+% gmake TEST=nightly test
+
+

Regardless of which test you're running, the results are printed on standard output and standard error. You can redirect these results to a file if you