Commit Graph

333 Commits

Author SHA1 Message Date
Arnold Schwaighofer
5a0272c759 lit: Allow configurations to restrict the set of tests to run
By setting limit_to_features to a non empty list of features a configuration can
restrict the set of tests to run to only include tests that require a feature in
this list.

rdar://21082253

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@238766 91177308-0d34-0410-b5e6-96231b3b80d8
2015-06-01 17:50:03 +00:00
Matthias Braun
cccd128a8a Lit: Allow overriding llvm tool paths+arguments, make -D an alias for --param
These changes allow usages where you want to pass an additional
commandline option to all invocations of a specific llvm tool. Example:

> llvm-lit -Dllc=llc -enable-misched -verify-machineinstrs

Differential Revision: http://reviews.llvm.org/D9487

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@236461 91177308-0d34-0410-b5e6-96231b3b80d8
2015-05-04 21:36:36 +00:00
Reid Kleckner
285ec33f69 [lit] Allow disabling an entire gtest suite, as is done in tsan
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@234336 91177308-0d34-0410-b5e6-96231b3b80d8
2015-04-07 18:14:10 +00:00
Reid Kleckner
93d8e23940 [lit] Skip gtest names starting with DISABLED_
The sanitizer test suite uses this idiom to disable a test.  Now that we
actually check if a test ran after invoking it, we see that zero tests
ran, and complain.

Instead, ignore tests starting with DISABLED_ completely. Fixes the
sanitizer test suite failures on Windows.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@234247 91177308-0d34-0410-b5e6-96231b3b80d8
2015-04-06 22:16:58 +00:00
Reid Kleckner
0123b6ee73 [lit] Fix running gtest type-parameterized tests on Windows
The '/' character in the test name of a type-parameterized test is not a
path separator, and should not be '\' on Windows. We were passing a test
name to --gtest_filter which found no tests, so the exit code was zero,
indicating a passed test.

This bug has been here since r84387 in 2009, when Jeff Yasskin added the
original lit support for type-paratermized tests. Somewhere along the
line some of the ValueMapTests started failing, but we can fix those
separately.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@234242 91177308-0d34-0410-b5e6-96231b3b80d8
2015-04-06 21:49:55 +00:00
Reid Kleckner
2dc2d3f820 lit: Add 'cd' support to the internal shell and port some tests
The internal shell was already threading around a 'cwd' parameter. We
just have to make it mutable so that we can update it as the test script
executes.

If the shell ever grows support for environment variable substitution,
we could also implement support for export.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@231017 91177308-0d34-0410-b5e6-96231b3b80d8
2015-03-02 21:33:18 +00:00
Reid Kleckner
304821b21b Don't warn or note if bash is missing
We haven't needed bash on Windows to run the test suite for a long time
now.

Patch by Michael Edwards!

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@228221 91177308-0d34-0410-b5e6-96231b3b80d8
2015-02-04 22:36:52 +00:00
Sylvestre Ledru
8931c49efa Fix some bashims. More information on https://wiki.ubuntu.com/DashAsBinSh. Reported initially on https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=772302 & https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=772301
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@227744 91177308-0d34-0410-b5e6-96231b3b80d8
2015-02-01 14:55:43 +00:00
Eric Fiselier
b727b13f61 [lit] Format JSONMetricValue strings better.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@226672 91177308-0d34-0410-b5e6-96231b3b80d8
2015-01-21 16:38:31 +00:00
Rafael Espindola
7c845683ea Don't set LD_PRELOAD to ''. It doesn't work on OpenBSD.
Patch by Brad Smith.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@225890 91177308-0d34-0410-b5e6-96231b3b80d8
2015-01-14 00:39:56 +00:00
Eric Fiselier
1af01214b5 [LIT] Remove string decoding in gtest discovery code. lit.util.capture now does decoding.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@225693 91177308-0d34-0410-b5e6-96231b3b80d8
2015-01-12 20:43:34 +00:00
Eric Fiselier
2d88dc293b [LIT] Decode string result in lit.util.capture
Summary: I think this is probably a bug, but I'm putting this up for review just to be sure. I think that `lit.util.capture` should decode the resulting string in the same way `lit.util.executeCommand` does.

Reviewers: ddunbar, EricWF

Reviewed By: EricWF

Subscribers: llvm-commits

Differential Revision: http://reviews.llvm.org/D6769

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@225681 91177308-0d34-0410-b5e6-96231b3b80d8
2015-01-12 20:09:34 +00:00
Rafael Espindola
50c8c5686e Pass LSAN_OPTIONS down so that it is possible to add suppressions.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@224777 91177308-0d34-0410-b5e6-96231b3b80d8
2014-12-23 18:39:02 +00:00
Eric Fiselier
2bea59bd02 Split executeShTest into two parts so that it can be better leveraged by libc++
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@224672 91177308-0d34-0410-b5e6-96231b3b80d8
2014-12-20 03:23:53 +00:00
Eric Fiselier
74430387cd [LIT] Add JSONMetricValue type to wrap types supported by the json encoder.
Summary:
The following types can be encoded and decoded by the json library:
`dict`, `list`, `tuple`, `str`, `unicode`, `int`, `long`, `float`, `bool`, `NoneType`. 

`JSONMetricValue` can be constructed with any of these types, and used as part of Test.Result.
This patch also adds a toMetricValue function that converts a value into a MetricValue.




Reviewers: ddunbar, EricWF

Reviewed By: EricWF

Subscribers: cfe-commits, llvm-commits

Differential Revision: http://reviews.llvm.org/D6576

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@224628 91177308-0d34-0410-b5e6-96231b3b80d8
2014-12-19 22:29:12 +00:00
Rafael Espindola
fe6c7ee0f0 Now that we require a newer python, use the new exception syntax.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@224130 91177308-0d34-0410-b5e6-96231b3b80d8
2014-12-12 15:32:23 +00:00
Rafael Espindola
708bc15ff0 Require python 2.7.
We were already requiring 2.5, which meant that people on old linux distros
had to upgrade anyway.

Requiring python 2.6 will make supporting 3.X easier as we can use the 3.X
exception syntax.

According to the discussion on llvmdev, there is not much value is requiring
just 2.6, we may as well just require 2.7.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@224129 91177308-0d34-0410-b5e6-96231b3b80d8
2014-12-12 15:29:31 +00:00
Eric Fiselier
bd11260981 [LIT] Add support for UNSUPPORTED tag to TestRunner.parseIntegratedTestScript
Summary:
This patch gives me just enough to leverage the existing functionality in `TestRunner` for use in `libc++` and `libc++abi` .

It does the following:
* Adds the `UNSUPPORTED` tag to `TestRunner.parseIntegratedTestScript`.
* Allows `parseIntegratedTestScript` to return an empty script if a script is not required by the caller.



Reviewers: ddunbar, EricWF

Reviewed By: EricWF

Subscribers: cfe-commits, llvm-commits

Differential Revision: http://reviews.llvm.org/D6589

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@223915 91177308-0d34-0410-b5e6-96231b3b80d8
2014-12-10 03:42:09 +00:00
David Majnemer
726bd41602 lit: Don't use python 2.6 features
LLVM supports python 2.5, this fixes a (somewhat) recent regression.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@223626 91177308-0d34-0410-b5e6-96231b3b80d8
2014-12-08 08:33:30 +00:00
Chris Matthews
ab1837aef3 Fix for xunit output to work around issue in Jenkins when tests are at the root level
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@223562 91177308-0d34-0410-b5e6-96231b3b80d8
2014-12-06 01:13:49 +00:00
Chris Matthews
689f10d1b5 Fix corner cases in lit xunit for paths with dots
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@223549 91177308-0d34-0410-b5e6-96231b3b80d8
2014-12-06 00:21:08 +00:00
Chris Matthews
0baca9564c Add a test-case for lit xunit output
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@223307 91177308-0d34-0410-b5e6-96231b3b80d8
2014-12-03 23:01:35 +00:00
Chris Matthews
06a1a5f256 Don't allow test-suite names with . in them in xml output
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@223305 91177308-0d34-0410-b5e6-96231b3b80d8
2014-12-03 22:59:15 +00:00
Chris Matthews
dcedd15f58 Give lit a --xunit-xml-output option for saving results in xunit format
--xunit-xml-output saves test results to disk in JUnit's xml format. This will allow Jenkins to report the details of a lit run.
  
  Based on a patch by David Chisnall.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@223163 91177308-0d34-0410-b5e6-96231b3b80d8
2014-12-02 22:19:21 +00:00
Rafael Espindola
3ada4ec04c Handle ASAN_OPTIONS and UBSAN_OPTIONS in TestingConfig.py
Currently they are passed to tests of llvm itself, but not, for example, lld.

With this patch the options are visible in every test.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@221198 91177308-0d34-0410-b5e6-96231b3b80d8
2014-11-03 23:04:56 +00:00
Rafael Espindola
86156e8997 refactor duplicated code. NFC.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@221191 91177308-0d34-0410-b5e6-96231b3b80d8
2014-11-03 22:17:49 +00:00
Rafael Espindola
6199c0cf17 [lit] Forward LD_PRELOAD to tests.
With this patch I can use asan to test the gold plugin without having
to build gold itself with asan.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@221183 91177308-0d34-0410-b5e6-96231b3b80d8
2014-11-03 21:24:43 +00:00
Daniel Dunbar
539be96ea4 [lit] Fix Python-3 compatibility, patch by Dan Liew.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@220357 91177308-0d34-0410-b5e6-96231b3b80d8
2014-10-22 01:26:06 +00:00
Daniel Dunbar
1c76fb5008 [lit] Bump version number.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@220355 91177308-0d34-0410-b5e6-96231b3b80d8
2014-10-22 00:48:23 +00:00
Daniel Dunbar
ac1af12c28 Fix ShTest parsing error when a keyword line doesn't end with a newline.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@220354 91177308-0d34-0410-b5e6-96231b3b80d8
2014-10-22 00:34:31 +00:00
Dan Liew
559074a285 Teach Lit to catch OSError exceptions when creating a process during the
execution of a shell command. This can happen for example if the
``RUN:`` line calls a python script which can work correctly under
Linux/OSX but will not work under Windows. A more useful error message
is now shown rather than an unhelpful backtrace.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@220227 91177308-0d34-0410-b5e6-96231b3b80d8
2014-10-20 20:14:28 +00:00
Peter Collingbourne
798ace2e58 Initial version of Go bindings.
This code is based on the existing LLVM Go bindings project hosted at:
https://github.com/go-llvm/llvm

Note that all contributors to the gollvm project have agreed to relicense
their changes under the LLVM license and submit them to the LLVM project.

Differential Revision: http://reviews.llvm.org/D5684

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@219976 91177308-0d34-0410-b5e6-96231b3b80d8
2014-10-16 22:48:02 +00:00
Jordan Rose
c919e57cbf [lit] Parse all strings as UTF-8 rather than ASCII.
As far as I can tell UTF-8 has been supported since the beginning of Python's
codec support, and it's the de facto standard for text these days, at least
for primarily-English text. This allows us to put Unicode into lit RUN lines.

rdar://problem/18311663

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@217688 91177308-0d34-0410-b5e6-96231b3b80d8
2014-09-12 16:46:05 +00:00
Eric Fiselier
e1167f1319 [LIT] Move display of unsupported and xfail tests to summary.
Summary:
This patch changes the way xfail and unsupported tests are displayed. 
This output is only displayed when the --show-unsupported/--show-xfail flags are passed to lit.

Currently xfail/unsupported tests are printed during the run of the test-suite. I think its better to display this information during the summary instead.
This patch removes the printing of these tests from when they are run to the summary.


Reviewers: ddunbar, EricWF

Reviewed By: EricWF

Subscribers: llvm-commits

Differential Revision: http://reviews.llvm.org/D4842

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@215809 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-16 02:16:25 +00:00
Eric Fiselier
2785e77db1 [lit] Add --show-xfail flag to LIT.
Summary:
This patch add a --show-xfail flag. If this flag is specified then each xfail test will be printed to output.
When it is not given xfail tests are ignored. Ignoring xfail tests is the current behavior.

This flag is meant to mirror the --show-unsupported flag that was recently added.

Reviewers: ddunbar, EricWF

Reviewed By: EricWF

Subscribers: llvm-commits

Differential Revision: http://reviews.llvm.org/D4750

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@214609 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-02 01:29:52 +00:00
Eric Fiselier
d409968d60 [lit] Add --show-unsupported flag to LIT
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213227 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-17 05:53:00 +00:00
Hans Wennborg
4a5916aca9 lit: simplify population of the actual_inputs array
Add all inputs to the array, except those starting with @, which
are treated as response files and expanded.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211119 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-17 18:17:46 +00:00
Hans Wennborg
baf9879290 Revert "lit: warn when passed invalid pathname" (r210597)
It was pointed out that this breaks the "virtual test discovery"
mechanism, which allows for narming tests in the test exec root.

Reverting until I can figure out how to fix this.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211048 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-16 20:18:41 +00:00
Hans Wennborg
a327ac3e51 lit: warn when passed invalid pathname
It would previously say things like

  warning: input 'test/Frontend/foo.c' contained no tests

and have the user pull their hair trying to figure out what's wrong with that
file. This patch changes the message to the much clearer:

  warning: no such file or directory: 'test/Frontend/foo.c'

Differential Revision: http://reviews.llvm.org/D4097

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@210597 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-10 22:51:58 +00:00
Alexander Potapenko
22471f5731 Add detection of OS X relocatable SDK to compiler-rt as a lit.util function
Clang's lit cfg already detects the currently selected SDK via
"xcrun --show-sdk-path". The same thing should be done for compiler-rt tests,
to make them work on recent OS X versions. Instead of duplicating the detection
code, this patch extracts the detection function into a lit.util method.

Patch by Kuba Brecka (kuba.brecka@gmail.com),
reviewed at http://reviews.llvm.org/D4072


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@210534 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-10 14:22:00 +00:00
Joerg Sonnenberger
df40801dff Fix test syntax to work with non-bash /bin/sh.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@206119 91177308-0d34-0410-b5e6-96231b3b80d8
2014-04-12 21:13:41 +00:00
Daniel Dunbar
d111694af2 [lit] Fix non-function style print statement.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@203573 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-11 14:05:49 +00:00
Daniel Dunbar
453474e38c [lit] Bump dev version number.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@203498 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-10 21:58:16 +00:00
Daniel Dunbar
a594517e5b [lit] Add a README.txt.
- Also, update MANIFEST.in and utils/check-sdist.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@203497 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-10 21:58:12 +00:00
Daniel Dunbar
a99759357a [lit] Add --version option.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@203496 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-10 21:57:48 +00:00
Alp Toker
ca179f4de8 lit: Provide source locations in cfg files with older Python versions
This commit prospectively brings the benefits of r198766 to older supported
Python versions (2.5+).

Tested with Python 2.6, 2.7, 3.1 and 3.3 (!)

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199009 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-11 14:34:18 +00:00
Alp Toker
4c28d2024f lit: execfile() isn't present in Python 3.3
On the other hand, exec(compile()) doesn't work in older Python versions in the
2.x series.

This commit introduces exec(compile()) with a fallback to plain exec(). That'll
hopefully hit the sweet spot in terms of version support.

Followup to r198766 which added enhanced source locations for lit cfg parsing.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199006 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-11 13:27:28 +00:00
Alp Toker
598887906b lit: Provide file location in cfg error messages
Python doesn't do a good job at diagnosing string exec() so use execfile()
where available.

This should be a timesaver when trying to get to the bottom of build bot
failures.

Before:

    File "llvm/utils/lit/lit/TestingConfig.py", line 93, in load_from_path
      exec("exec data in cfg_globals")
    File "<string>", line 1, in <module>
    File "<string>", line 194, in <module>
  NameError: name 'typo' is not defined

After:

    File "llvm/utils/lit/lit/TestingConfig.py", line 95, in load_from_path
    execfile(path, cfg_globals)
    File "clang/test/lit.cfg", line 194, in <module>
      typo
      ^~~~
  NameError: name 'typo' is not defined

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@198766 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-08 14:20:59 +00:00
Alp Toker
9c7316f760 Python compatibility fix for r198150
Remove the stat call error reporting for now.

It wasn't essential so silent fallback should be fine here.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@198155 91177308-0d34-0410-b5e6-96231b3b80d8
2013-12-29 06:51:10 +00:00
Alp Toker
aa29cf421a Prospective Python 3 fix for r198150
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@198152 91177308-0d34-0410-b5e6-96231b3b80d8
2013-12-29 05:51:07 +00:00