From 5026c7f91d23405fe3fc3d5d5618f2f1b9398299 Mon Sep 17 00:00:00 2001 From: Tanya Lattner Date: Mon, 6 Dec 2004 02:11:52 +0000 Subject: [PATCH] Added DejaGNU information to the testing guide. This should go into the 1.4 release. I left the QMTest stuff in place but made it appear to be optional. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@18564 91177308-0d34-0410-b5e6-96231b3b80d8 --- docs/TestingGuide.html | 113 +++++++++++++++++++++++++++++++++++++---- 1 file changed, 102 insertions(+), 11 deletions(-) diff --git a/docs/TestingGuide.html b/docs/TestingGuide.html index 0f3d1b020b2..21e5ba8379b 100644 --- a/docs/TestingGuide.html +++ b/docs/TestingGuide.html @@ -22,6 +22,7 @@
  • LLVM Test Suite Tree
    • +
  • DejaGNU Structure
  • QMTest Structure
  • llvm-test Structure
  • Running the LLVM Tests
    • @@ -29,8 +30,8 @@
    -

    Written by John T. Criswell and Reid Spencer

    +

    Written by John T. Criswell, Reid Spencer, and Tanya Lattner

    @@ -55,15 +56,22 @@ and run tests.

    required to build LLVM, plus the following:

    +
    DejaGNU
    +
    The Feature and Regressions tests are organized and run by DejaGNU.
    +
    Expect
    +
    Expect is required by DejaGNU.
    +
    tclsh
    +
    Tclsh is required by DejaGNU.
    +
    QMTest
    The LLVM test suite uses QMTest to organize and run tests. Note: you will need QMTest 2.0.3 (source tar.gz file) to be successful. The tests do not run with -any other version.
    +any other version. (optional, required only for QMTest)
    Python
    You will need a Python interpreter that works with QMTest. Python will -need zlib and SAX support enabled.
    +need zlib and SAX support enabled. (optional, required only for QMTest)
    F2C
    For now, LLVM does not have a Fortran front-end, but using F2C, we can run @@ -106,14 +114,23 @@ programs in C and C++ is in the llvm-test module. This module should be checked out to the llvm/projects directory. When you configure the llvm module, the llvm-test module will be automatically configured. Or you can do it manually.

    -

    To run all of the simple tests in LLVM, use the master Makefile in the +

    To run all of the simple tests in LLVM using DejaGNU, use the master Makefile in the llvm/test directory:

     % gmake -C llvm/test
    +or
    +
    +% gmake check
+
    +

    To run only a subdirectory of tests in llvm/test using DejaGNU (ie. Regression/Transforms). Just substitute the path to the subdirectory:

    +
    +% gmake -C llvm/test TESTSUITE=Regression/Transforms
+
    +
    Note: If you are running the tests with objdir != subdir you must have run the complete testsuite before you can specify a subdirectory.
    -

    To run only the code fragment tests (i.e. those that do basic testing of -LLVM), run the tests organized by QMTest:

    +

    To run the simple tests (i.e. those that do basic testing of +LLVM), using QMTest:

     % gmake -C llvm/test qmtest
    @@ -169,7 +186,7 @@ language front end.

    Code fragments are not complete programs, and they are never executed to determine correct behavior.

    -

    Thes code fragment tests are located in the llvm/test/Features and +

    These code fragment tests are located in the llvm/test/Features and llvm/test/Regression directories.

    @@ -260,6 +277,82 @@ test and database classes.

    + + +
    DejaGNU Structure
    + + +
    +

    The LLVM test suite is partially driven by DejaGNU and partially +driven by GNU Make. Specifically, the Features and Regression tests +are all driven by DejaGNU (and optionally QMTest). The llvm-test +module is currently driven by a set of Makefiles.

    + +

    The DejaGNU structure is very simple, but does require some +information to be set. This information is gathered via configure and +is written to a file, site.exp in llvm/test. The llvm/test +Makefile does this work for you.

    + +

    In order for DejaGNU to work, each directory of tests must have a +dg.exp file. This file is a program written in tcl that calls +the llvm-runtests procedure on each test file. The +llvm-runtests procedure is defined in +llvm/test/lib/llvm-dg.exp. Any directory that contains only +directories does not need the dg.exp file.

    + +

    In order for a test to be run, it must contain information within +the test file on how to run the test. These are called RUN +lines. Run lines are specified in the comments of the test program +using the keyword RUN followed by a colon, and lastly the +commands to execute. These commands will be executed in a bash script, +so any bash syntax is acceptable. You can specify as many RUN lines as +necessary. Each RUN line translates to one line in the resulting bash +script. Below is an example of legal RUN lines in a .ll +file:

    +
    +; RUN: llvm-as < %s | llvm-dis > %t1
+; RUN: llvm-dis < %s.bc-13 > %t2
+; RUN: diff %t1 %t2
+
    +

    There are a couple patterns within a RUN line that the +llvm-runtest procedure looks for and replaces with the appropriate +syntax:

    +
      +
      %p
      +
      The path to the source directory. This is for locating +any supporting files that are not generated by the test, but used by +the test.
      +
      %s
      +
      The test file.
      + +
      $t
      +
      Temporary filename: testscript.test_filename.tmp, where +test_filename is the name of the test file. All temporary files are +placed in the Output directory within the directory the test is +located.
      + +
      %prcontext
      +
      Path to a script that performs grep -C. Use this since not all +platforms support grep -C.
      + +
      %llvmgcc
      Full path to the llvmgcc executable.
      +
      %llvmgxx
      Full path to the llvmg++ executable.
      +
    + +

    There are also several scripts in the llvm/test/Scripts directory +that you might find useful when writing RUN lines.

    + +

    Lastly, you can easily mark a test that is expected to fail on a +specific platform by using the XFAIL keyword. Xfail lines are +specified in the comments of the test program using XFAIL, +followed by a colon, and one or more regular expressions (separated by +a comma) that will match against the target triplet for the +machine. You can use * to match all targets. Here is an example of an +XFAIL line:

    +
    +; XFAIL: darwin,sun
+
    +
    @@ -268,9 +361,7 @@ test and database classes.

    -

    The LLVM test suite is partially driven by QMTest and partially driven by GNU -Make. Specifically, the Features and Regression tests are all driven by QMTest. -The llvm-test module is currently driven by a set of Makefiles.

    +

    The Feature and Regression tests can also be run using QMTest.

    The QMTest system needs to have several pieces of information available; these pieces of configuration information are known collectively as the