6502/llvm-6502
1
0
Fork 0
mirror of https://github.com/c64scene-ar/llvm-6502.git synced 2025-05-20 05:38:50 +00:00
Code Issues Projects Releases Wiki Activity

Documentation for lit: more formatting: use 'option' and 'program' directives.

Browse Source 
This enables cross-referencing and now '--' in option names are no more turned into en dashes.


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@168906 91177308-0d34-0410-b5e6-96231b3b80d8
This commit is contained in:
Dmitri Gribenko 2012-11-29 18:03:08 +00:00
parent 6eda081345
commit bc5fb06785
1 changed files with 144 additions and 139 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

241
docs/CommandGuide/lit.rst
View File

@ -4,60 +4,61 @@ lit - LLVM Integrated Tester
SYNOPSIS SYNOPSIS
-------- --------
**lit** [*options*] [*tests*] :program:`lit` [*options*] [*tests*]
DESCRIPTION DESCRIPTION
----------- -----------
**lit** is a portable tool for executing LLVM and Clang style test suites, :program:`lit` is a portable tool for executing LLVM and Clang style test
summarizing their results, and providing indication of failures. **lit** is suites, summarizing their results, and providing indication of failures.
designed to be a lightweight testing tool with as simple a user interface as :program:`lit` is designed to be a lightweight testing tool with as simple a
possible. user interface as possible.
**lit** should be run with one or more *tests* to run specified on the command :program:`lit` should be run with one or more *tests* to run specified on the
line. Tests can be either individual test files or directories to search for command line. Tests can be either individual test files or directories to
tests (see :ref:`test-discovery`). search for tests (see :ref:`test-discovery`).
Each specified test will be executed (potentially in parallel) and once all Each specified test will be executed (potentially in parallel) and once all
tests have been run **lit** will print summary information on the number of tests tests have been run :program:`lit` will print summary information on the number
which passed or failed (see :ref:`test-status-results`). The **lit** program will of tests which passed or failed (see :ref:`test-status-results`). The
execute with a non-zero exit code if any tests fail. :program:`lit` program will execute with a non-zero exit code if any tests
fail.
By default **lit** will use a succinct progress display and will only print By default :program:`lit` will use a succinct progress display and will only
summary information for test failures. See :ref:`output-options` for options print summary information for test failures. See :ref:`output-options` for
controlling the **lit** progress display and output. options controlling the :program:`lit` progress display and output.
**lit** also includes a number of options for controlling how tests are executed :program:`lit` also includes a number of options for controlling how tests are
(specific features may depend on the particular test format). See executed (specific features may depend on the particular test format). See
:ref:`execution-options` for more information. :ref:`execution-options` for more information.
Finally, **lit** also supports additional options for only running a subset of Finally, :program:`lit` also supports additional options for only running a
the options specified on the command line, see :ref:`selection-options` for subset of the options specified on the command line, see
more information. :ref:`selection-options` for more information.
Users interested in the **lit** architecture or designing a **lit** testing Users interested in the :program:`lit` architecture or designing a
implementation should see :ref:`lit-infrastructure`. :program:`lit` testing implementation should see :ref:`lit-infrastructure`.
GENERAL OPTIONS GENERAL OPTIONS
--------------- ---------------
**-h**, **--help** .. option:: -h, --help
Show the **lit** help message. Show the :program:`lit` help message.
**-j** *N*, **--threads**\ =\ *N* .. option:: -j N, --threads=N
Run *N* tests in parallel. By default, this is automatically chosen to match Run ``N`` tests in parallel. By default, this is automatically chosen to
the number of detected available CPUs. match the number of detected available CPUs.
**--config-prefix**\ =\ *NAME* .. option:: --config-prefix=NAME
Search for *NAME.cfg* and *NAME.site.cfg* when searching for test suites, Search for :file:`{NAME}.cfg` and :file:`{NAME}.site.cfg` when searching for
instead of *lit.cfg* and *lit.site.cfg*. test suites, instead of :file:`lit.cfg` and :file:`lit.site.cfg`.
**--param** *NAME*, **--param** *NAME*\ =\ *VALUE* .. option:: --param NAME, --param NAME=VALUE
Add a user defined parameter *NAME* with the given *VALUE* (or the empty Add a user defined parameter ``NAME`` with the given ``VALUE`` (or the empty
string if not given). The meaning and use of these parameters is test suite string if not given). The meaning and use of these parameters is test suite
dependent. dependent.
@ -66,20 +67,20 @@ GENERAL OPTIONS
OUTPUT OPTIONS OUTPUT OPTIONS
-------------- --------------
**-q**, **--quiet** .. option:: -q, --quiet
Suppress any output except for test failures. Suppress any output except for test failures.
**-s**, **--succinct** .. option:: -s, --succinct
Show less output, for example don't show information on tests that pass. Show less output, for example don't show information on tests that pass.
**-v**, **--verbose** .. option:: -v, --verbose
Show more information on test failures, for example the entire test output Show more information on test failures, for example the entire test output
instead of just the test result. instead of just the test result.
**--no-progress-bar** .. option:: --no-progress-bar
Do not use curses based progress bar. Do not use curses based progress bar.
@ -88,78 +89,81 @@ OUTPUT OPTIONS
EXECUTION OPTIONS EXECUTION OPTIONS
----------------- -----------------
**--path**\ =\ *PATH* .. option:: --path=PATH
Specify an addition *PATH* to use when searching for executables in tests. Specify an additional ``PATH`` to use when searching for executables in tests.
**--vg** .. option:: --vg
Run individual tests under valgrind (using the memcheck tool). The Run individual tests under valgrind (using the memcheck tool). The
*--error-exitcode* argument for valgrind is used so that valgrind failures will ``--error-exitcode`` argument for valgrind is used so that valgrind failures
cause the program to exit with a non-zero status. will cause the program to exit with a non-zero status.
When this option is enabled, **lit** will also automatically provide a When this option is enabled, :program:`lit` will also automatically provide a
"valgrind" feature that can be used to conditionally disable (or expect failure "``valgrind``" feature that can be used to conditionally disable (or expect
in) certain tests. failure in) certain tests.
**--vg-arg**\ =\ *ARG* .. option:: --vg-arg=ARG
When *--vg* is used, specify an additional argument to pass to valgrind itself. When :option:`--vg` is used, specify an additional argument to pass to
:program:`valgrind` itself.
**--vg-leak** .. option:: --vg-leak
When *--vg* is used, enable memory leak checks. When this option is enabled, When :option:`--vg` is used, enable memory leak checks. When this option is
**lit** will also automatically provide a "vg_leak" feature that can be enabled, :program:`lit` will also automatically provide a "``vg_leak``"
used to conditionally disable (or expect failure in) certain tests. feature that can be used to conditionally disable (or expect failure in)
certain tests.
**--time-tests** .. option:: --time-tests
Track the wall time individual tests take to execute and includes the results in Track the wall time individual tests take to execute and includes the results
the summary output. This is useful for determining which tests in a test suite in the summary output. This is useful for determining which tests in a test
take the most time to execute. Note that this option is most useful with *-j suite take the most time to execute. Note that this option is most useful
1*. with ``-j 1``.
.. _selection-options: .. _selection-options:
SELECTION OPTIONS SELECTION OPTIONS
----------------- -----------------
**--max-tests**\ =\ *N* .. option:: --max-tests=N
Run at most *N* tests and then terminate. Run at most ``N`` tests and then terminate.
**--max-time**\ =\ *N* .. option:: --max-time=N
Spend at most *N* seconds (approximately) running tests and then terminate. Spend at most ``N`` seconds (approximately) running tests and then terminate.
**--shuffle** .. option:: --shuffle
Run the tests in a random order. Run the tests in a random order.
ADDITIONAL OPTIONS ADDITIONAL OPTIONS
------------------ ------------------
**--debug** .. option:: --debug
Run **lit** in debug mode, for debugging configuration issues and **lit** itself. Run :program:`lit` in debug mode, for debugging configuration issues and
:program:`lit` itself.
**--show-suites** .. option:: --show-suites
List the discovered test suites as part of the standard output. List the discovered test suites as part of the standard output.
**--no-tcl-as-sh** .. option:: --no-tcl-as-sh
Run Tcl scripts internally (instead of converting to shell scripts). Run Tcl scripts internally (instead of converting to shell scripts).
**--repeat**\ =\ *N* .. option:: --repeat=N
Run each test *N* times. Currently this is primarily useful for timing tests, Run each test ``N`` times. Currently this is primarily useful for timing
other results are not collated in any reasonable fashion. tests, other results are not collated in any reasonable fashion.
EXIT STATUS EXIT STATUS
----------- -----------
**lit** will exit with an exit code of 1 if there are any FAIL or XPASS :program:`lit` will exit with an exit code of 1 if there are any FAIL or XPASS
results. Otherwise, it will exit with the status 0. Other exit codes are used results. Otherwise, it will exit with the status 0. Other exit codes are used
for non-test related failures (for example a user error or an internal program for non-test related failures (for example a user error or an internal program
error). error).
@ -169,28 +173,28 @@ error).
TEST DISCOVERY TEST DISCOVERY
-------------- --------------
The inputs passed to **lit** can be either individual tests, or entire The inputs passed to :program:`lit` can be either individual tests, or entire
directories or hierarchies of tests to run. When **lit** starts up, the first directories or hierarchies of tests to run. When :program:`lit` starts up, the
thing it does is convert the inputs into a complete list of tests to run as part first thing it does is convert the inputs into a complete list of tests to run
of *test discovery*. as part of *test discovery*.
In the **lit** model, every test must exist inside some *test suite*. **lit** In the :program:`lit` model, every test must exist inside some *test suite*.
resolves the inputs specified on the command line to test suites by searching :program:`lit` resolves the inputs specified on the command line to test suites
upwards from the input path until it finds a *lit.cfg* or *lit.site.cfg* by searching upwards from the input path until it finds a :file:`lit.cfg` or
file. These files serve as both a marker of test suites and as configuration :file:`lit.site.cfg` file. These files serve as both a marker of test suites
files which **lit** loads in order to understand how to find and run the tests and as configuration files which :program:`lit` loads in order to understand
inside the test suite. how to find and run the tests inside the test suite.
Once **lit** has mapped the inputs into test suites it traverses the list of Once :program:`lit` has mapped the inputs into test suites it traverses the
inputs adding tests for individual files and recursively searching for tests in list of inputs adding tests for individual files and recursively searching for
directories. tests in directories.
This behavior makes it easy to specify a subset of tests to run, while still This behavior makes it easy to specify a subset of tests to run, while still
allowing the test suite configuration to control exactly how tests are allowing the test suite configuration to control exactly how tests are
interpreted. In addition, **lit** always identifies tests by the test suite they interpreted. In addition, :program:`lit` always identifies tests by the test
are in, and their relative path inside the test suite. For appropriately suite they are in, and their relative path inside the test suite. For
configured projects, this allows **lit** to provide convenient and flexible appropriately configured projects, this allows :program:`lit` to provide
support for out-of-tree builds. convenient and flexible support for out-of-tree builds.
.. _test-status-results: .. _test-status-results:
@ -238,12 +242,12 @@ section for more information.
LIT INFRASTRUCTURE LIT INFRASTRUCTURE
------------------ ------------------
This section describes the **lit** testing architecture for users interested in This section describes the :program:`lit` testing architecture for users interested in
creating a new **lit** testing implementation, or extending an existing one. creating a new :program:`lit` testing implementation, or extending an existing one.
**lit** proper is primarily an infrastructure for discovering and running :program:`lit` proper is primarily an infrastructure for discovering and running
arbitrary tests, and to expose a single convenient interface to these arbitrary tests, and to expose a single convenient interface to these
tests. **lit** itself doesn't know how to run tests, rather this logic is tests. :program:`lit` itself doesn't know how to run tests, rather this logic is
defined by *test suites*. defined by *test suites*.
TEST SUITES TEST SUITES
@ -253,11 +257,11 @@ As described in :ref:`test-discovery`, tests are always located inside a *test
suite*. Test suites serve to define the format of the tests they contain, the suite*. Test suites serve to define the format of the tests they contain, the
logic for finding those tests, and any additional information to run the tests. logic for finding those tests, and any additional information to run the tests.
**lit** identifies test suites as directories containing *lit.cfg* or :program:`lit` identifies test suites as directories containing ``lit.cfg`` or
*lit.site.cfg* files (see also **--config-prefix**). Test suites are initially ``lit.site.cfg`` files (see also :option:`--config-prefix`). Test suites are
discovered by recursively searching up the directory hierarchy for all the input initially discovered by recursively searching up the directory hierarchy for
files passed on the command line. You can use **--show-suites** to display the all the input files passed on the command line. You can use
discovered test suites at startup. :option:`--show-suites` to display the discovered test suites at startup.
Once a test suite is discovered, its config file is loaded. Config files Once a test suite is discovered, its config file is loaded. Config files
themselves are Python modules which will be executed. When the config file is themselves are Python modules which will be executed. When the config file is
@ -306,7 +310,7 @@ executed, two important global variables are predefined:
**parent** The parent configuration, this is the config object for the directory **parent** The parent configuration, this is the config object for the directory
containing the test suite, or None. containing the test suite, or None.
**root** The root configuration. This is the top-most **lit** configuration in **root** The root configuration. This is the top-most :program:`lit` configuration in
the project. the project.
**on_clone** The config is actually cloned for every subdirectory inside a test **on_clone** The config is actually cloned for every subdirectory inside a test
@ -320,41 +324,42 @@ executed, two important global variables are predefined:
TEST DISCOVERY TEST DISCOVERY
~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~
Once test suites are located, **lit** recursively traverses the source directory Once test suites are located, :program:`lit` recursively traverses the source
(following *test_src_root*) looking for tests. When **lit** enters a directory (following *test_src_root*) looking for tests. When :program:`lit`
sub-directory, it first checks to see if a nested test suite is defined in that enters a sub-directory, it first checks to see if a nested test suite is
directory. If so, it loads that test suite recursively, otherwise it defined in that directory. If so, it loads that test suite recursively,
instantiates a local test config for the directory (see otherwise it instantiates a local test config for the directory (see
:ref:`local-configuration-files`). :ref:`local-configuration-files`).
Tests are identified by the test suite they are contained within, and the Tests are identified by the test suite they are contained within, and the
relative path inside that suite. Note that the relative path may not refer to an relative path inside that suite. Note that the relative path may not refer to
actual file on disk; some test formats (such as *GoogleTest*) define "virtual an actual file on disk; some test formats (such as *GoogleTest*) define
tests" which have a path that contains both the path to the actual test file and "virtual tests" which have a path that contains both the path to the actual
a subpath to identify the virtual test. test file and a subpath to identify the virtual test.
.. _local-configuration-files: .. _local-configuration-files:
LOCAL CONFIGURATION FILES LOCAL CONFIGURATION FILES
~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~
When **lit** loads a subdirectory in a test suite, it instantiates a local test When :program:`lit` loads a subdirectory in a test suite, it instantiates a
configuration by cloning the configuration for the parent direction --- the root local test configuration by cloning the configuration for the parent direction
of this configuration chain will always be a test suite. Once the test --- the root of this configuration chain will always be a test suite. Once the
configuration is cloned **lit** checks for a *lit.local.cfg* file in the test configuration is cloned :program:`lit` checks for a *lit.local.cfg* file
subdirectory. If present, this file will be loaded and can be used to specialize in the subdirectory. If present, this file will be loaded and can be used to
the configuration for each individual directory. This facility can be used to specialize the configuration for each individual directory. This facility can
define subdirectories of optional tests, or to change other configuration be used to define subdirectories of optional tests, or to change other
parameters --- for example, to change the test format, or the suffixes which configuration parameters --- for example, to change the test format, or the
identify test files. suffixes which identify test files.
TEST RUN OUTPUT FORMAT TEST RUN OUTPUT FORMAT
~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~
The **lit** output for a test run conforms to the following schema, in both The :program:`lit` output for a test run conforms to the following schema, in
short and verbose modes (although in short mode no PASS lines will be shown). both short and verbose modes (although in short mode no PASS lines will be
This schema has been chosen to be relatively easy to reliably parse by a machine shown). This schema has been chosen to be relatively easy to reliably parse by
(for example in buildbot log scraping), and for other tools to generate. a machine (for example in buildbot log scraping), and for other tools to
generate.
Each test result is expected to appear on a line that matches: Each test result is expected to appear on a line that matches:
@ -402,8 +407,8 @@ B, C, and D, and a log message for the failing test C:
LIT EXAMPLE TESTS LIT EXAMPLE TESTS
~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~
The **lit** distribution contains several example implementations of test suites The :program:`lit` distribution contains several example implementations of
in the *ExampleTests* directory. test suites in the *ExampleTests* directory.
SEE ALSO SEE ALSO
-------- --------