Commit Graph

268 Commits

Author SHA1 Message Date
Daniel Dunbar
2849503ab2 [lit] Add an --output option, for writing results in a machine readable form.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@190738 91177308-0d34-0410-b5e6-96231b3b80d8
2013-09-14 01:19:17 +00:00
Daniel Dunbar
ff058f0a70 [lit] Add support for attach arbitrary metrics to test results.
- This is a work-in-progress and all details are subject to change, but I am
   trying to build up support for allowing lit to be used as a driver for
   performance tests (or other tests which might want to record information
   beyond simple PASS/FAIL).

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@190535 91177308-0d34-0410-b5e6-96231b3b80d8
2013-09-11 17:45:11 +00:00
Daniel Dunbar
4b3e5f24ec [lit] Allow config files to pass arbitrary values to child configs.
- This aligns with how existing test suites end up wanting to use the local
   config files, conceptually it makes sense to consider them to be inherited.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189885 91177308-0d34-0410-b5e6-96231b3b80d8
2013-09-03 23:32:55 +00:00
Daniel Dunbar
e0ff96c112 [lit] [tests] Add missing test input file.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189561 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-29 03:02:34 +00:00
Daniel Dunbar
cb4bfb3409 [lit] Fix internal shell's argv[0] handling.
- At least on OS X, it is important for correct behavior of /bin/[ that argv[0]
   is passed as written, and not as the full executable path.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189559 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-29 02:52:10 +00:00
Daniel Dunbar
4ac723b53f [lit] Add support for multiprocessing, under --use-processes for now.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189556 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-29 00:54:23 +00:00
Daniel Dunbar
df44de6d91 [lit] Implement --max-time support by using provider cancel method.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189555 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-29 00:54:19 +00:00
Daniel Dunbar
07f0f16bfd [lit] Refactor test execution logic into lit.run.Run.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189554 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-29 00:54:15 +00:00
Daniel Dunbar
b11b690d3f [lit] Put display lock inside the ThreadResultsConsumer.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189553 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-29 00:54:09 +00:00
Daniel Dunbar
ec8e059254 [lit] Factor out a results consumer interface for test execution.
- Also, change TestProvider interface to operate on test indices.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189552 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-29 00:54:02 +00:00
Daniel Dunbar
8c59003cc3 [lit] Move top-level execute code into Run object.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189551 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-29 00:48:55 +00:00
Daniel Dunbar
5b2efc28fd [lit] Factor out Run class to capture configuration + discovered tests.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189550 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-29 00:48:45 +00:00
Daniel Dunbar
8253cc047d [lit] Change lit.Test.ResultCode to be unique across pickling.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189549 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-29 00:48:39 +00:00
Daniel Dunbar
6f25e872e4 [lit] Add a TODO.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189546 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-29 00:41:15 +00:00
Daniel Dunbar
501cba385c [lit] Update shtest format to return lit.Test.Result objects.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189545 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-29 00:41:09 +00:00
Daniel Dunbar
8a1d9b207a [lit] Update LitTestCase to support lit.Test.Result.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189544 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-29 00:40:59 +00:00
Daniel Dunbar
4db169d31e [lit] Fix a couple lingering Py3 compat issues in ProgressBar.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188951 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-21 22:26:47 +00:00
Daniel Dunbar
f72bc79226 [lit] Lift XFAIL handling to core infrastructure.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188949 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-21 22:26:42 +00:00
Daniel Dunbar
d3bf8a2c0a [lit] Allow formats to return lit.Test.Result instances directly.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188948 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-21 22:26:40 +00:00
Daniel Dunbar
ccd21b26dd [lit] Factor out a separate Test.Result() object.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188947 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-21 22:26:37 +00:00
Daniel Dunbar
0dd41a99e3 [lit] Simplify --time-tests code.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188946 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-21 22:26:34 +00:00
Daniel Dunbar
7b0cdf7bf4 [lit] Extract TestFormat base class, for future use.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188945 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-21 22:26:26 +00:00
Daniel Dunbar
70360e6762 [lit] Rewrite TODO list, and elaborate on some things.
- If anyone is interested in lit's feature set, I'd appreciate any comments on
   the elaborated items.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188590 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-16 23:30:23 +00:00
Daniel Dunbar
23114d0da7 [lit] Remove some done TODOs.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188502 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-15 22:52:20 +00:00
Daniel Dunbar
4fa5939268 [lit] Add test coverage of gtest format.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188417 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-14 22:21:11 +00:00
Daniel Dunbar
80a06baec2 [lit] Ensure test output is converted to strings where possible.
- This cleans up the text output of failing tests when run under PY3.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188416 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-14 22:21:01 +00:00
Daniel Dunbar
35d5e9044c [lit] Support parsing scripts with inconsistent or invalid encodings.
- For whatever reason, we have a lot of test files with bogus unicode
   characters. This patch allows those scripts to still be parsed on Python3 by
   changing the parsing logic to work on binary files, and only require the
   actual script commands to be convertible to ascii.

 - This patch has been tweaked to now ensure that the command strings are not of
   unicode type on Python 2.6-7.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188398 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-14 18:22:41 +00:00
Daniel Dunbar
14a5c695a9 Revert r188376, "[lit] Support parsing scripts with inconsistent or invalid encodings.", this doesn't work yet for bots using the internal shell.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188379 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-14 15:55:25 +00:00
Daniel Dunbar
47e7c040f4 [lit] Support parsing scripts with inconsistent or invalid encodings.
- For whatever reason, we have a lot of test files with bogus unicode
   characters. This patch allows those scripts to still be parsed on Python3 by
   changing the parsing logic to work on binary files, and only require the
   actual script commands to be convertible to ascii.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188376 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-14 15:24:58 +00:00
Daniel Dunbar
af5a9e3f1e [lit] Fix a relative import issue I missed earlier.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188360 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-14 05:07:16 +00:00
Daniel Dunbar
213ab86ee6 [lit] Avoid StringIO.
- We barely used it, and it is very hard to use in a 2.5-3 compatible
   way because of changing expectations for its input types.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188359 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-14 05:07:13 +00:00
Daniel Dunbar
6b78ef3762 [lit] Fix tests to execute lit with same python as invoked with.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188358 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-14 05:07:09 +00:00
Daniel Dunbar
5403214852 [lit] Factor ShTest format script command parsing from other processing.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188357 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-14 05:07:04 +00:00
Daniel Dunbar
493e8d4bf5 [lit] Move executeCommand() into lit.util.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188356 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-14 05:07:01 +00:00
Daniel Dunbar
f76c664f6b [lit] Move formats into their own subpackage.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188355 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-14 05:06:55 +00:00
NAKAMURA Takumi
e908486f5b Lit: Introduce "%/[STpst] into parseIntegratedTestScript(), to normalize substitutions.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188348 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-14 02:26:31 +00:00
Daniel Dunbar
13e26da155 [lit] Support use of setup.py from other directories.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188309 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-13 19:08:48 +00:00
Daniel Dunbar
ea549f0e12 [lit] Remove old ExamplesTests directory.
- The actual tests have better coverage than those, and they weren't useful
   anymore.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188110 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-09 21:39:36 +00:00
Daniel Dunbar
de404a2809 [lit] Move ManyTests examples to lit/examples/many-tests.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188109 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-09 21:39:28 +00:00
Daniel Dunbar
a5e463ea5f [lit] Drop deprecated aliases for lit and old module names.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188108 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-09 21:39:24 +00:00
Daniel Dunbar
4981a6ca97 [lit] Update lit's own tests to use lit_config and lit package, as appropriate.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188107 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-09 21:39:17 +00:00
Daniel Dunbar
efdcf800fe [lit] Fix typo.
- Noticed by edward-san (IRC).

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188096 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-09 19:39:42 +00:00
Daniel Dunbar
128ce319ec [lit] Rename lit.{TestFormats,Util} to their aliased names {formats,util}.
- With compatibility hack in lit.__init__, so this hopefully shouldn't break
   anything.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188040 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-09 00:37:15 +00:00
Daniel Dunbar
38649827c3 [lit] Inject the lit specific config object as 'lit_config' when loading config files.
- Injecting it as 'lit' is gross, since that name should be used to refer to
   the actual package. For now both are available so it is possibly to cleanup
   test config files incrementally.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188039 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-09 00:37:05 +00:00
Daniel Dunbar
6c749c5fbc [lit] Split TestingConfig.frompath() into separate ctor and load methods.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188038 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-09 00:36:58 +00:00
Daniel Dunbar
49e51429c1 [lit] Eliminate mustExist parameter from TestingConfig.frompath().
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188034 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-09 00:09:02 +00:00
Daniel Dunbar
3279653eb8 [lit] Only create config copies when a local config file is present.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188033 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-09 00:08:56 +00:00
Daniel Dunbar
b3c0c58ca4 [lit] Eliminate parent argument from TestingConfig.frompath(), which is effectively unused.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188032 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-09 00:08:46 +00:00
Daniel Dunbar
abb9de5257 [lit] Change --show-{tests,suites} to exit after printing.
- This is a more sensible behavior than printing and also running tests.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188009 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-08 20:59:25 +00:00
Daniel Dunbar
59c6b1073c [lit] Remove --repeat option, which wasn't that useful.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188008 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-08 20:59:20 +00:00