Commit Graph

11427 Commits

Author SHA1 Message Date
Hans Wennborg
09a31f3154 Allow switch-to-lookup table for tables with holes by adding bitmask check
This allows us to generate table lookups for code such as:

  unsigned test(unsigned x) {
    switch (x) {
      case 100: return 0;
      case 101: return 1;
      case 103: return 2;
      case 105: return 3;
      case 107: return 4;
      case 109: return 5;
      case 110: return 6;
      default: return f(x);
    }
  }

Since cases 102, 104, etc. are not constants, the lookup table has holes
in those positions. We therefore guard the table lookup with a bitmask check.

Patch by Jasper Neumann!

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@203694 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-12 18:35:40 +00:00
Evan Cheng
9225686155 Revert r203488 and r203520.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@203687 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-12 18:09:37 +00:00
Eli Bendersky
4c871b7880 Revive SizeOptLevel-explaining comments that were dropped in r203669
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@203675 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-12 16:44:17 +00:00
Eli Bendersky
ce306f9f99 Move duplicated code into a helper function (exposed through overload).
There's a bit of duplicated "magic" code in opt.cpp and Clang's CodeGen that
computes the inliner threshold from opt level and size opt level.

This patch moves the code to a function that lives alongside the inliner itself,
providing a convenient overload to the inliner creation.

A separate patch can be committed to Clang to use this once it's committed to
LLVM. Standalone tools that use the inlining pass can also avoid duplicating
this code and fearing it will go out of sync.

Note: this patch also restructures the conditinal logic of the computation to
be cleaner.



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@203669 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-12 16:12:36 +00:00
Alon Mishne
e74c0bf111 Cloning a function now also clones its debug metadata if 'ModuleLevelChanges' is true.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@203662 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-12 14:42:51 +00:00
Erik Verbruggen
0638609d66 Fix crash in PRE.
After r203553 overflow intrinsics and their non-intrinsic (normal)
instruction get hashed to the same value. This patch prevents PRE from
moving an instruction into a predecessor block, and trying to add a phi
node that gets two different types (the intrinsic result and the
non-intrinsic result), resulting in a failing assert.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@203574 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-11 15:07:32 +00:00
Tim Northover
ca396e391e IR: add a second ordering operand to cmpxhg for failure
The syntax for "cmpxchg" should now look something like:

	cmpxchg i32* %addr, i32 42, i32 3 acquire monotonic

where the second ordering argument gives the required semantics in the case
that no exchange takes place. It should be no stronger than the first ordering
constraint and cannot be either "release" or "acq_rel" (since no store will
have taken place).

rdar://problem/15996804

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@203559 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-11 10:48:52 +00:00
Erik Verbruggen
fb411c8b8c GVN: fix hashing of extractvalue.
My last commit did not add the indexes to the hashed value for
extractvalue. Adding that back in.


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@203558 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-11 10:21:30 +00:00
Erik Verbruggen
cf5240642b GVN: merge overflow intrinsics with non-overflow instructions.
When an overflow intrinsic is followed by a non-overflow instruction,
replace the latter with an extract. For example:

  %sadd = tail call { i32, i1 } @llvm.sadd.with.overflow.i32(i32 %a, i32 %b)
  %sadd3 = add i32 %a, %b

Here the add statement will be replaced by an extract.

When an overflow intrinsic follows a non-overflow instruction, a clone
of the intrinsic is inserted before the normal instruction, which makes
it the same as the previous case. Subsequent runs of GVN can then clean
up the duplicate instructions and insert the extract.

This fixes PR8817.


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@203553 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-11 09:36:48 +00:00
Duncan P. N. Exon Smith
3139a0a41a Cleanup whitespace
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@203529 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-11 02:44:45 +00:00
Evan Cheng
fc77954998 Follow up to r203488. Code clean up to eliminate a lot of copy+paste.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@203520 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-11 00:24:20 +00:00
Diego Novillo
87393cfd6b Use discriminator information in sample profiles.
Summary:
When the sample profiles include discriminator information,
use the discriminator values to distinguish instruction weights
in different basic blocks.

This modifies the BodySamples mapping to map <line, discriminator> pairs
to weights. Instructions on the same line but different blocks, will
use different discriminator values. This, in turn, means that the blocks
may have different weights.

Other changes in this patch:

- Add tests for positive values of line offset, discriminator and samples.
- Change data types from uint32_t to unsigned and int and do additional
  validation.

Reviewers: chandlerc

CC: llvm-commits

Differential Revision: http://llvm-reviews.chandlerc.com/D2857

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@203508 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-10 22:41:28 +00:00
Benjamin Kramer
8da0b7358d MemCpyOpt: When merging memsets also merge the trivial case of two memsets with the same destination.
The testcase is from PR19092, but I think the bug described there is actually a clang issue.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@203489 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-10 21:05:13 +00:00
Evan Cheng
d89b0f200c For functions with ARM target specific calling convention, when simplify-libcall
optimize a call to a llvm intrinsic to something that invovles a call to a C
library call, make sure it sets the right calling convention on the call.

e.g.
extern double pow(double, double);
double t(double x) {
  return pow(10, x);
}

Compiles to something like this for AAPCS-VFP:
define arm_aapcs_vfpcc double @t(double %x) #0 {
entry:
  %0 = call double @llvm.pow.f64(double 1.000000e+01, double %x)
  ret double %0
}

declare double @llvm.pow.f64(double, double) #1

Simplify libcall (part of instcombine) will turn the above into:
define arm_aapcs_vfpcc double @t(double %x) #0 {
entry:
  %__exp10 = call double @__exp10(double %x) #1
  ret double %__exp10
}

declare double @__exp10(double)

The pre-instcombine code works because calls to LLVM builtins are special.
Instruction selection will chose the right calling convention for the call.
However, the code after instcombine is wrong. The call to __exp10 will use
the C calling convention.

I can think of 3 options to fix this.

1. Make "C" calling convention just work since the target should know what CC
   is being used.

   This doesn't work because each function can use different CC with the "pcs"
   attribute.

2. Have Clang add the right CC keyword on the calls to LLVM builtin.

   This will work but it doesn't match the LLVM IR specification which states
   these are "Standard C Library Intrinsics".

3. Fix simplify libcall so the resulting calls to the C routines will have the
   proper CC keyword. e.g.
   %__exp10 = call arm_aapcs_vfpcc double @__exp10(double %x) #1

   This works and is the solution I implemented here.

Both solutions #2 and #3 would work. After carefully considering the pros and
cons, I decided to implement #3 for the following reasons.

1. It doesn't change the "spec" of the intrinsics.
2. It's a self-contained fix.

There are a couple of potential downsides.
1. There could be other places in the optimizer that is broken in the same way
   that's not addressed by this.
2. There could be other calling conventions that need to be propagated by
   simplify-libcall that's not handled.

But for now, this is the fix that I'm most comfortable with.


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@203488 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-10 20:49:45 +00:00
Benjamin Kramer
4b484628f4 SimplifyCFG: Simplify the weight scaling algorithm.
No change in functionality.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@203413 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-09 14:42:55 +00:00
Ahmed Charles
9c3328fc7f Fix build break.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@203366 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-09 03:50:36 +00:00
Chandler Carruth
36b699f2b1 [C++11] Add range based accessors for the Use-Def chain of a Value.
This requires a number of steps.
1) Move value_use_iterator into the Value class as an implementation
   detail
2) Change it to actually be a *Use* iterator rather than a *User*
   iterator.
3) Add an adaptor which is a User iterator that always looks through the
   Use to the User.
4) Wrap these in Value::use_iterator and Value::user_iterator typedefs.
5) Add the range adaptors as Value::uses() and Value::users().
6) Update *all* of the callers to correctly distinguish between whether
   they wanted a use_iterator (and to explicitly dig out the User when
   needed), or a user_iterator which makes the Use itself totally
   opaque.

Because #6 requires churning essentially everything that walked the
Use-Def chains, I went ahead and added all of the range adaptors and
switched them to range-based loops where appropriate. Also because the
renaming requires at least churning every line of code, it didn't make
any sense to split these up into multiple commits -- all of which would
touch all of the same lies of code.

The result is still not quite optimal. The Value::use_iterator is a nice
regular iterator, but Value::user_iterator is an iterator over User*s
rather than over the User objects themselves. As a consequence, it fits
a bit awkwardly into the range-based world and it has the weird
extra-dereferencing 'operator->' that so many of our iterators have.
I think this could be fixed by providing something which transforms
a range of T&s into a range of T*s, but that *can* be separated into
another patch, and it isn't yet 100% clear whether this is the right
move.

However, this change gets us most of the benefit and cleans up
a substantial amount of code around Use and User. =]

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@203364 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-09 03:16:01 +00:00
Benjamin Kramer
ba2a549016 [C++11] Revert uses of lambdas with array_pod_sort.
Looks like GCC implements the lambda->function pointer conversion differently.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@203294 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-07 21:52:38 +00:00
Benjamin Kramer
e1362f1ebd [C++11] Convert sort predicates into lambdas.
No functionality change.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@203288 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-07 21:35:39 +00:00
Tim Northover
69d2b2aa5a InstCombine: form shuffles from wider range of insert/extractelements
Sequences of insertelement/extractelements are sometimes used to build
vectorsr; this code tries to put them back together into shuffles, but
could only produce a completely uniform shuffle types (<N x T> from two
<N x T> sources).

This should allow shuffles with different numbers of elements on the
input and output sides as well.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@203229 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-07 10:24:44 +00:00
Ahmed Charles
f4ccd11075 Replace OwningPtr<T> with std::unique_ptr<T>.
This compiles with no changes to clang/lld/lldb with MSVC and includes
overloads to various functions which are used by those projects and llvm
which have OwningPtr's as parameters. This should allow out of tree
projects some time to move. There are also no changes to libs/Target,
which should help out of tree targets have time to move, if necessary.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@203083 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-06 05:51:42 +00:00
Chandler Carruth
67f6bf70d2 [Layering] Move InstVisitor.h into the IR library as it is pretty
obviously coupled to the IR.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@203064 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-06 03:23:41 +00:00
Chandler Carruth
f4ec8bfaec [Layering] Move DebugInfo.h into the IR library where its implementation
already lives.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@203046 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-06 00:46:21 +00:00
Chandler Carruth
7cf9764966 [Layering] Move DIBuilder.h into the IR library where its implementation
already lives.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@203038 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-06 00:22:06 +00:00
Arnold Schwaighofer
9d84b4d70c LoopVectorizer: Preserve fast-math flags
Fixes PR19045.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@203008 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-05 21:10:47 +00:00
Chandler Carruth
b810517338 [Layering] Move DebugLoc.h into the IR library. The implementation
already lived there and it is where it belongs -- this is the in-memory
debug location representation.

This is just cleanup -- Modules can actually cope with this, but that
doesn't make it right. After chatting with folks that have out-of-tree
stuff, going ahead and moving the rest of the headers seems preferable.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202960 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-05 10:30:38 +00:00
Chandler Carruth
2e816f0d56 [C++11] Make this interface accept const Use pointers and use override
to ensure we don't mess up any of the overrides. Necessary for cleaning
up the Value use iterators and enabling range-based traversing of use
lists.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202958 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-05 10:21:48 +00:00
Ahmed Charles
1a6eca243f [C++11] Replace OwningPtr::take() with OwningPtr::release().
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202957 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-05 10:19:29 +00:00
Craig Topper
7b62be28cb [C++11] Add 'override' keyword to virtual methods that override their base class.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202953 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-05 09:10:37 +00:00
Chandler Carruth
19d764fb05 [Modules] Move the ConstantRange class into the IR library. This is
a bit surprising, as the class is almost entirely abstracted away from
any particular IR, however it encodes the comparsion predicates which
mutate ranges as ICmp predicate codes. This is reasonable as they're
used for both instructions and constants. Thus, it belongs in the IR
library with instructions and constants.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202838 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-04 12:24:34 +00:00
Chandler Carruth
5b74a01aad [Modules] Move the PredIteratorCache into the IR library -- it is
hardcoded to use IR BasicBlocks.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202835 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-04 12:09:19 +00:00
Chandler Carruth
ff956e7568 [Modules] Move the NoFolder into the IR library as it creates
instructions.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202834 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-04 12:05:47 +00:00
Chandler Carruth
d90488f7e8 [Modules] Move the TargetFolder into the Analysis library. Historically,
this would have been required because of the use of DataLayout, but that
has moved into the IR proper. It is still required because this folder
uses the constant folding in the analysis library (which uses the
datalayout) as the more aggressive basis of its folder.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202832 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-04 11:59:06 +00:00
Chandler Carruth
03e36d752c [Modules] Move CFG.h to the IR library as it defines graph traits over
IR types.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202827 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-04 11:45:46 +00:00
Chandler Carruth
7225e27b4c [Modules] Move ValueMap to the IR library. While this class does not
directly care about the Value class (it is templated so that the key can
be any arbitrary Value subclass), it is in fact concretely tied to the
Value class through the ValueHandle's CallbackVH interface which relies
on the key type being some Value subclass to establish the value handle
chain.

Ironically, the unittest is already in the right library.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202824 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-04 11:26:31 +00:00
Chandler Carruth
eb3d76da81 [Modules] Move ValueHandle into the IR library where Value itself lives.
Move the test for this class into the IR unittests as well.

This uncovers that ValueMap too is in the IR library. Ironically, the
unittest for ValueMap is useless in the Support library (honestly, so
was the ValueHandle test) and so it already lives in the IR unittests.
Mmmm, tasty layering.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202821 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-04 11:17:44 +00:00
Chandler Carruth
df3d8e8b4d [Modules] Move the LLVM IR pattern match header into the IR library, it
obviously is coupled to the IR.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202818 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-04 11:08:18 +00:00
Chandler Carruth
4bbfbdf7d7 [Modules] Move CallSite into the IR library where it belogs. It is
abstracting between a CallInst and an InvokeInst, both of which are IR
concepts.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202816 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-04 11:01:28 +00:00
Chandler Carruth
bd7cba0d81 [Modules] Move GetElementPtrTypeIterator into the IR library. As its
name might indicate, it is an iterator over the types in an instruction
in the IR.... You see where this is going.

Another step of modularizing the support library.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202815 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-04 10:40:04 +00:00
Chandler Carruth
876ac60880 [Modules] Move InstIterator out of the Support library, where it had no
business.

This header includes Function and BasicBlock and directly uses the
interfaces of both classes. It has to do with the IR, it even has that
in the name. =] Put it in the library it belongs to.

This is one step toward making LLVM's Support library survive a C++
modules bootstrap.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202814 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-04 10:30:26 +00:00
Chandler Carruth
1decd56b8d [cleanup] Re-sort all the includes with utils/sort_includes.py.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202811 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-04 10:07:28 +00:00
Diego Novillo
f05b45fdb2 Pass to emit DWARF path discriminators.
DWARF discriminators are used to distinguish multiple control flow paths
on the same source location. When this happens, instructions across
basic block boundaries will share the same debug location.

This pass detects this situation and creates a new lexical scope to one
of the two instructions. This lexical scope is a child scope of the
original and contains a new discriminator value. This discriminator is
then picked up from MCObjectStreamer::EmitDwarfLocDirective to be
written on the object file.

This fixes http://llvm.org/bugs/show_bug.cgi?id=18270.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202752 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-03 20:06:11 +00:00
Benjamin Kramer
9efaf2f2da [C++11] Use std::tie to simplify compare operators.
No functionality change.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202751 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-03 19:58:30 +00:00
Benjamin Kramer
dcf23ddbcb [C++11] Remove a leftover std::function instance.
It's not needed anymore.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202748 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-03 19:49:02 +00:00
Chandler Carruth
0550e93e89 [C++11] Remove the completely unnecessary requirement on SetVector's
remove_if that its predicate is adaptable. We don't actually need this,
we can write a generic adapter for any predicate.

This lets us remove some very wrong std::function usages. We should
never be using std::function for predicates to algorithms. This incurs
an *indirect* call overhead for every evaluation of the predicate, and
makes it very hard to inline through.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202742 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-03 19:28:52 +00:00
Evgeniy Stepanov
8b77d958fd [msan] Handle X86 SIMD bitshift intrinsics.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202712 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-03 13:47:42 +00:00
Tobias Grosser
75c86c9e9a [C++11] Add a basic block range view for RegionInfo
This also switches the users in LLVM to ensure this functionality is tested.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202705 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-03 13:00:39 +00:00
Chandler Carruth
3dfabcb249 [C++11] Add two range adaptor views to User: operands and
operand_values. The first provides a range view over operand Use
objects, and the second provides a range view over the Value*s being
used by those operands.

The naming is "STL-style" rather than "LLVM-style" because we have
historically named iterator methods STL-style, and range methods seem to
have far more in common with their iterator counterparts than with
"normal" APIs. Feel free to bikeshed on this one if you want, I'm happy
to change these around if people feel strongly.

I've switched code in SROA and LCG to exercise these mostly to ensure
they work correctly -- we don't really have an easy way to unittest this
and they're trivial.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202687 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-03 10:42:58 +00:00
Benjamin Kramer
a4f0aad951 [C++11] Replace llvm::tie with std::tie.
The old implementation is no longer needed in C++11.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202644 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-02 13:30:33 +00:00
Benjamin Kramer
d628f19f5d [C++11] Replace llvm::next and llvm::prior with std::next and std::prev.
Remove the old functions.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202636 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-02 12:27:27 +00:00
Craig Topper
629b96cb4f Switch all uses of LLVM_OVERRIDE to just use 'override' directly.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202621 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-02 09:09:27 +00:00
Chandler Carruth
0a3eef53d7 [C++11] Switch all uses of the llvm_move macro to use std::move
directly, and remove the macro.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202612 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-02 04:08:41 +00:00
Benjamin Kramer
ee5e607355 Now that we have C++11, turn simple functors into lambdas and remove a ton of boilerplate.
No intended functionality change.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202588 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-01 11:47:00 +00:00
Reid Kleckner
d95e8b43d3 Reflow isProfitableToMakeFastCC
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202555 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-28 22:50:08 +00:00
Kostya Serebryany
9aef4445a3 [asan] fix a pair of silly typos
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202391 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-27 13:13:59 +00:00
Kostya Serebryany
124bbea6cd [asan] disable asan-detect-invalid-pointer-pair (was enabled by mistake)
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202390 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-27 12:56:20 +00:00
Kostya Serebryany
9b3d3a96a8 [asan] *experimental* implementation of invalid-pointer-pair detector (finds when two unrelated pointers are compared or subtracted). This implementation has both false positives and false negatives and is not tuned for performance. A bug report for a proper implementation will follow.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202389 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-27 12:45:36 +00:00
Reid Kleckner
726bae9a66 GlobalOpt: Apply fastcc to internal x86_thiscallcc functions
We should apply fastcc whenever profitable.  We can expand this list,
but there are lots of conventions with performance implications that we
don't want to change.

Differential Revision: http://llvm-reviews.chandlerc.com/D2705

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202293 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-26 19:57:30 +00:00
Andrew Trick
401d35bedb Fix PR18165: LSR must avoid scaling factors that exceed the limit on truncated use.
Patch by Michael Zolotukhin!

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202273 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-26 16:31:56 +00:00
Chandler Carruth
2b442bffcb [SROA] Use the correct index integer size in GEPs through non-default
address spaces.

This isn't really a correctness issue (the values are truncated) but its
much cleaner.

Patch by Matt Arsenault!

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202252 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-26 10:08:16 +00:00
Chandler Carruth
5b95cec37c [SROA] Teach SROA how to handle pointers from address spaces other than
the default.

Based on the patch by Matt Arsenault, D1764!

I switched one place to use the more direct pointer type to compute the
desired address space, and I reworked the memcpy rewriting section to
reflect significant refactorings that this patch helped inspire.

Thanks to several of the folks who helped review and improve the patch
as well.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202247 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-26 08:25:02 +00:00
Chandler Carruth
38e90e3de1 [SROA] Split the alignment computation complete for the memcpy rewriting
to work independently for the slice side and the other side.

This allows us to only compute the minimum of the two when we actually
rewrite to a memcpy that needs to take the minimum, and preserve higher
alignment for one side or the other when rewriting to loads and stores.

This fix was inspired by seeing the result of some refactoring that
makes addrspace handling better.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202242 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-26 07:29:54 +00:00
Chandler Carruth
c1c37734ad [SROA] The original refactoring inspired by the addrspace patch in
D1764, which in turn set off the other refactorings to make
'getSliceAlign()' a sensible thing.

There are two possible inputs to the required alignment of a memory
transfer intrinsic: the alignment constraints of the source and the
destination. If we are *only* introducing a (potentially new) offset
onto one side of the transfer, we don't need to consider the alignment
constraints of the other side. Use this to simplify the logic feeding
into alignment computation for unsplit transfers.

Also, hoist the clamp of the magical zero alignment for these intrinsics
to the more customary one alignment early. This lets several other
conditions melt away.

No functionality changed. There is a further improvement this exposes
which *will* change functionality, but that's arriving in a separate
patch.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202232 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-26 05:33:36 +00:00
Chandler Carruth
dd080790eb [SROA] Yet another slight refactoring that simplifies an API in the
rewriting logic: don't pass custom offsets for the adjusted pointer to
the new alloca.

We always passed NewBeginOffset here. Sometimes we spelled it
BeginOffset, but only when they were in fact equal. Whats worse, the API
is set up so that you can't reasonably call it with anything else -- it
assumes that you're passing it an offset relative to the *original*
alloca that happens to fall within the new one. That's the whole point
of NewBeginOffset, it's the clamped beginning offset.

No functionality changed.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202231 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-26 05:12:43 +00:00
Chandler Carruth
c9166f098c [SROA] Simplify the computing of alignment: we only ever need the
alignment of the slice being rewritten, not any arbitrary offset.

Every caller is really just trying to compute the alignment for the
whole slice, never for some arbitrary alignment. They are also just
passing a type when they have one to see if we can skip an explicit
alignment in the IR by using the type's alignment. This makes for a much
simpler interface.

Another refactoring inspired by the addrspace patch for SROA, although
only loosely related.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202230 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-26 05:02:19 +00:00
Chandler Carruth
f28c057c42 [SROA] Use NewOffsetBegin in the unsplit case for memset merely for
consistency with memcpy rewriting, and fix a latent bug in the alignment
management for memset.

The alignment issue is that getAdjustedAllocaPtr is computing the
*relative* offset into the new alloca, but the alignment isn't being set
to the relative offset, it was using the the absolute offset which is
into the old alloca.

I don't think its possible to write a test case that actually reaches
this code where the resulting alignment would be observably different,
but the intent was clearly to use the relative offset within the new
alloca.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202229 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-26 04:45:24 +00:00
Chandler Carruth
f11f7a49b9 [SROA] Use the members for New{Begin,End}Offset in the rewrite helpers
rather than passing them as arguments.

While I generally prefer actual arguments, in this case the readability
loss is substantial. By using members we avoid repeatedly calculating
the offsets, and once we're using members it is useful to ensure that
those names *always* refer to the original-alloca-relative new offset
for a rewritten slice.

No functionality changed. Follow-up refactoring, all toward getting the
address space patch merged.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202228 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-26 04:25:04 +00:00
Chandler Carruth
abd2555e36 [SROA] Compute the New{Begin,End}Offset values once for each alloca
slice being rewritten.

We had the same code scattered across most of the visits. Instead,
compute the new offsets and the slice size once when we start to visit
a particular slice, and use the member variables from then on. This
reduces quite a bit of code duplication.

No functionality changed. Refactoring inspired to make it easier to
apply the address space patch to SROA.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202227 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-26 04:20:00 +00:00
Chandler Carruth
50bc165c54 [SROA] Fix PR18615 with some long overdue simplifications to the bounds
checking in SROA.

The primary change is to just rely on uge for checking that the offset
is within the allocation size. This removes the explicit checks against
isNegative which were terribly error prone (including the reversed logic
that led to PR18615) and prevented us from supporting stack allocations
larger than half the address space.... Ok, so maybe the latter isn't
*common* but it's a silly restriction to have.

Also, we used to try to support a PHI node which loaded from before the
start of the allocation if any of the loaded bytes were within the
allocation. This doesn't make any sense, we have never really supported
loading or storing *before* the allocation starts. The simplified logic
just doesn't care.

We continue to allow loading past the end of the allocation in part to
support cases where there is a PHI and some loads are larger than others
and the larger ones reach past the end of the allocation. We could solve
this a different and more conservative way, but I'm still somewhat
paranoid about this.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202224 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-26 03:14:14 +00:00
Chandler Carruth
9c256eccb3 [reassociate] Switch two std::sort calls into std::stable_sort calls as
their inputs come from std::stable_sort and they are not total orders.

I'm not a huge fan of this, but the really bad std::stable_sort is right
at the beginning of Reassociate. After we commit to stable-sort based
consistent respect of source order, the downstream sorts shouldn't undo
that unless they have a total order or they are used in an
order-insensitive way. Neither appears to be true for these cases.
I don't have particularly good test cases, but this jumped out by
inspection when looking for output instability in this pass due to
changes in the ordering of std::sort.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202196 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-25 21:54:50 +00:00
Chandler Carruth
495e40121b [SROA] Add an off-by-default *strict* inbounds check to SROA. I had SROA
implemented this way a long time ago and due to the overwhelming bugs
that surfaced, moved to a much more relaxed variant. Richard Smith would
like to understand the magnitude of this problem and it seems fairly
harmless to keep some flag-controlled logic to get the extremely strict
behavior here. I'll remove it if it doesn't prove useful.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202193 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-25 21:24:45 +00:00
Rafael Espindola
57edc9d4ff Make DataLayout a plain object, not a pass.
Instead, have a DataLayoutPass that holds one. This will allow parts of LLVM
don't don't handle passes to also use DataLayout.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202168 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-25 17:30:31 +00:00
Rafael Espindola
53eeb678ea Factor out calls to AA.getDataLayout().
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202157 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-25 15:52:19 +00:00
Rafael Espindola
37ecf69cbf Make a few more DataLayout variables const.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202155 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-25 14:24:11 +00:00
Chandler Carruth
a2ff58121a [SROA] Use the original load name with the SROA-prefixed IRB rather than
just "load". This helps avoid pointless de-duping with order-sensitive
numbers as we already have unique names from the original load. It also
makes the resulting IR quite a bit easier to read.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202140 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-25 11:21:48 +00:00
Chandler Carruth
f5701e4282 [SROA] Thread the ability to add a pointer-specific name prefix through
the pointer adjustment code. This is the primary code path that creates
totally new instructions in SROA and being able to lump them based on
the pointer value's name for which they were created causes
*significantly* fewer name collisions and general noise in the debug
output. This is particularly significant because it is making it much
harder to track down instability in the output of SROA, as name
de-duplication is a totally harmless form of instability that gets in
the way of seeing real problems.

The new fancy naming scheme tries to dig out the root "pre-SROA" name
for pointer values and associate that all the way through the pointer
formation instructions. Digging out the root is important to prevent the
multiple iterative rounds of SROA from just layering too much cruft on
top of cruft here. We already track the layers of SROAs iteration in the
alloca name prefix. We don't need to duplicate it here.

Should have no functionality change, and shouldn't have any really
measurable impact on NDEBUG builds, as most of the complex logic is
debug-only.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202139 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-25 11:19:56 +00:00
Chandler Carruth
19899632f7 [SROA] Rather than copying the logic for building a name prefix into the
PHI-pointer builder, just copy the builder and clobber the obvious
fields.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202136 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-25 11:12:04 +00:00
Chandler Carruth
b59c39e520 [SROA] Simplify some of the logic to dig out the old pointer value by
using OldPtr more heavily. Lots of this code was written before the
rewriter had an OldPtr member setup ahead of time. There are already
asserts in place that should ensure this doesn't change any
functionality.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202135 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-25 11:08:02 +00:00
Chandler Carruth
10b2920851 [SROA] Adjust to new clang-format style.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202134 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-25 11:07:58 +00:00
Chandler Carruth
64d5fd932b [SROA] Fix a *glaring* bug in r202091: you have to actually *write*
the break statement, not just think it to yourself....

No idea how this worked at all, much less survived most bots, my
bootstrap, and some bot bootstraps!

The Polly one didn't survive, and this was filed as PR18959. I don't
have a reduced test case and honestly I'm not seeing the need. What we
probably need here are better asserts / debug-build behavior in
SmallPtrSet so that this madness doesn't make it so far.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202129 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-25 09:45:27 +00:00
Alexey Samsonov
31c756b374 Silence GCC warning
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202119 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-25 07:56:00 +00:00
Alp Toker
bf930d5c1f Fix typos
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202107 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-25 04:21:15 +00:00
Chandler Carruth
eb1b5ba880 [SROA] Add a debugging tool which shuffles the slices sequence prior to
sorting it. This helps uncover latent reliance on the original ordering
which aren't guaranteed to be preserved by std::sort (but often are),
and which are based on the use-def chain orderings which also aren't
(technically) guaranteed.

Only available in C++11 debug builds, and behind a flag to prevent noise
at the moment, but this is generally useful so figured I'd put it in the
tree rather than keeping it out-of-tree.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202106 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-25 03:59:29 +00:00
Chandler Carruth
2516e7fb22 [SROA] Use a more direct way of determining whether we are processing
the destination operand or source operand of a memmove.

It so happens that it was impossible for SROA to try to rewrite
self-memmove where the operands are *identical*, because either such
a think is volatile (and we don't rewrite) or it is non-volatile, and we
don't even register it as a use of the alloca.

However, making the 'IsDest' test *rely* on this subtle fact is... Very
confusing for the reader. We should use the direct and readily available
test of the Use* which gives us concrete information about which operand
is being rewritten.

No functionality changed, I hope! ;]

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202103 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-25 03:50:14 +00:00
Chandler Carruth
c537759a7f [SROA] Fix another instability in SROA with respect to the slice
ordering.

The fundamental problem that we're hitting here is that the use-def
chain ordering is *itself* not a stable thing to be relying on in the
rewriting for SROA. Further, we use a non-stable sort over the slices to
arrange them based on the section of the alloca they're operating on.
With a debugging STL implementation (or different implementations in
stage2 and stage3) this can cause stage2 != stage3.

The specific aspect of this problem fixed in this commit deals with the
rewriting and load-speculation around PHIs and Selects. This, like many
other aspects of the use-rewriting in SROA, is really part of the
"strong SSA-formation" that is doen by SROA where it works very hard to
canonicalize loads and stores in *just* the right way to satisfy the
needs of mem2reg[1]. When we have a select (or a PHI) with 2 uses of the
same alloca, we test that loads downstream of the select are
speculatable around it twice. If only one of the operands to the select
needs to be rewritten, then if we get lucky we rewrite that one first
and the select is immediately speculatable. This can cause the order of
operand visitation, and thus the order of slices to be rewritten, to
change an alloca from promotable to non-promotable and vice versa.

The fix is to defer all of the speculation until *after* the rewrite
phase is done. Once we've rewritten everything, we can accurately test
for whether speculation will work (once, instead of twice!) and the
order ceases to matter.

This also happens to simplify the other subtlety of speculation -- we
need to *not* speculate anything unless the result of speculating will
make the alloca fully promotable by mem2reg. I had a previous attempt at
simplifying this, but it was still pretty horrible.

There is actually already a *really* nice test case for this in
basictest.ll, but on multiple STL implementations and inputs, we just
got "lucky". Fortunately, the test case is very small and we can
essentially build it in exactly the opposite way to get reasonable
coverage in both directions even from normal STL implementations.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202092 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-25 00:07:09 +00:00
Rafael Espindola
ec89b9fb9e Make some DataLayout pointers const.
No functionality change. Just reduces the noise of an upcoming patch.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202087 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-24 23:12:18 +00:00
Arnold Schwaighofer
68085c7bef SLPVectorizer: Try vectorizing 'splat' stores
Vectorize sequential stores of a broadcasted value.
5% on eon.

radar://16124699

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202067 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-24 19:52:29 +00:00
Rafael Espindola
ac69459e0f Replace the F_Binary flag with a F_Text one.
After this I will set the default back to F_None. The advantage is that
before this patch forgetting to set F_Binary would corrupt a file on windows.
Forgetting to set F_Text produces one that cannot be read in notepad, which
is a better failure mode :-)

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202052 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-24 18:20:12 +00:00
Arnold Schwaighofer
137342f198 LTO: Add the loop vectorizer to the LTO pipeline.
During the LTO phase LICM will move loop invariant global variables out of loops
(informed by GlobalModRef). This makes more loops countable presenting
opportunity for the loop vectorizer.

Adding the loop vectorizer improves some TSVC benchmarks and twolf/ref dataset
(5%) on x86-64.

radar://15970632

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202051 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-24 18:19:31 +00:00
Rafael Espindola
843efd49b7 Don't make F_None the default.
This will make it easier to switch the default to being binary files.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202042 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-24 15:07:20 +00:00
Kostya Serebryany
740a00491b [asan] simplify the code that compute the shadow offset; get rid of two internal flags that allowed to override it. The tests pass, but still this change might break asan on some platform not covered by tests. If you see this, please submit a fix with a test.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@202033 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-24 13:40:24 +00:00
Logan Chien
5e89e33e78 Include <cctype> for isdigit().
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201930 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-22 06:34:10 +00:00
Quentin Colombet
8048c44580 [CodeGenPrepare] Move CodeGenPrepare into lib/CodeGen.
CodeGenPrepare uses extensively TargetLowering which is part of libLLVMCodeGen.
This is a layer violation which would introduce eventually a dependence on
CodeGen in ScalarOpts.

Move CodeGenPrepare into libLLVMCodeGen to avoid that.

Follow-up of <rdar://problem/15519855>


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201912 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-22 00:07:45 +00:00
Rafael Espindola
75ece7a355 Rename a few more DataLayout variables from TD to DL.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201870 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-21 18:34:28 +00:00
Rafael Espindola
eb6e1d3165 Rename a few more DataLayout variables.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201833 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-21 01:53:35 +00:00
Rafael Espindola
f116e5308d Rename many DataLayout variables from TD to DL.
I am really sorry for the noise, but the current state where some parts of the
code use TD (from the old name: TargetData) and other parts use DL makes it
hard to write a patch that changes where those variables come from and how
they are passed along.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201827 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-21 00:06:31 +00:00
Nick Lewycky
0aabe661a4 Make sure that value handle users see the transformation of an indirect call to a direct call. This is important for the CallGraph iteration. Patch by Björn Steinbrink!
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201822 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-20 23:00:15 +00:00
Rafael Espindola
737c9f6005 Add back r201608, r201622, r201624 and r201625
r201608 made llvm corretly handle private globals with MachO. r201622 fixed
a bug in it and r201624 and r201625 were changes for using private linkage,
assuming that llvm would do the right thing.

They all got reverted because r201608 introduced a crash in LTO. This patch
includes a fix for that. The issue was that TargetLoweringObjectFile now has
to be initialized before we can mangle names of private globals. This is
trivially true during the normal codegen pipeline (the asm printer does it),
but LTO has to do it manually.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201700 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-19 17:23:20 +00:00
Rafael Espindola
3e0d3a45e5 This reverts commit r201625 and r201624.
Since r201608 got reverted, it is not safe to use private linkage in these cases
until it is committed back.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201688 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-19 15:49:46 +00:00
Tim Northover
44697f3fc1 X86 CodeGenPrep: sink shufflevectors before shifts
On x86, shifting a vector by a scalar is significantly cheaper than shifting a
vector by another fully general vector. Unfortunately, because SelectionDAG
operates on just one basic block at a time, the shufflevector instruction that
reveals whether the right-hand side of a shift *is* really a scalar is often
not visible to CodeGen when it's needed.

This adds another handler to CodeGenPrepare, to sink any useful shufflevector
instructions down to the basic block where they're used, predicated on a target
hook (since on other architectures, doing so will often just introduce extra
real work).

rdar://problem/16063505

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201655 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-19 10:02:43 +00:00
Rafael Espindola
070b5745ae Now that llvm always does the right thing with private, use it.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201625 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-19 02:08:39 +00:00
Rafael Espindola
39d8dcb53b Rename some member variables from TD to DL.
TargetData was renamed DataLayout back in r165242.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201581 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-18 15:33:12 +00:00
Tim Northover
4bcb985295 GlobalMerge: move "-global-merge" option to the pass itself.
It's rather odd to have the flag enabling and disabling this pass only affect a
single target.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201559 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-18 11:17:29 +00:00
Gerolf Hoflehner
3bc859466b fix for null VectorizedValue assertion in the SLP Vectorizer (in function vectorizeTree()). radar://16064178
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201501 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-17 03:06:16 +00:00
Gerolf Hoflehner
8e810aeec3 fixed typo in comment as my test commit
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201486 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-16 10:43:25 +00:00
Quentin Colombet
038600b42e [CodeGenPrepare][AddressingModeMatcher] Give up on type promotion if the
transformation does not bring any immediate benefits and introduce an illegal
operation. 


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201439 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-14 22:23:22 +00:00
Rafael Espindola
034b8f9d31 Trivial cleanup: reuse existing variable.
Extracted while trying to understand http://llvm-reviews.chandlerc.com/D1764.

Patch by Matt Arsenault.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201425 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-14 19:02:01 +00:00
Matt Arsenault
f222ebe86c Do more addrspacecast transforms that happen for bitcast.
Makes addrspacecast (gep) do addrspacecast (gep) instead.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201376 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-14 00:49:12 +00:00
Benjamin Kramer
62356fc694 InstCombine: Replace custom constant folding code with ConstantExpr.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201352 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-13 18:23:24 +00:00
Benjamin Kramer
8ed2c8e99f Reduce code duplication resulting from the ConstantVector/ConstantDataVector split.
No intended functionality change.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201344 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-13 16:48:38 +00:00
Reid Kleckner
2798b77586 GlobalOpt: Aliases don't have sections, don't copy them when replacing
As defined in LangRef, aliases do not have sections.  However, LLVM's
GlobalAlias class inherits from GlobalValue, which means we can read and
set its section.  We should probably ban that as a separate change,
since it doesn't make much sense for an alias to have a section that
differs from its aliasee.

Fixes PR18757, where the section was being lost on the global in code
from Clang like:

extern "C" {
__attribute__((used, section("CUSTOM"))) static int in_custom_section;
}

Reviewers: rafael.espindola

Differential Revision: http://llvm-reviews.chandlerc.com/D2758

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201286 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-13 02:18:36 +00:00
Owen Anderson
3042a65e5f Remove a very old instcombine where we would turn sequences of selects into
logical operations on the i1's driving them.  This is a bad idea for every
target I can think of (confirmed with micro tests on all of: x86-64, ARM,
AArch64, Mips, and PowerPC) because it forces the i1 to be materialized into
a general purpose register, whereas consuming it directly into a select generally
allows it to exist only transiently in a predicate or flags register.

Chandler ran a set of performance tests with this change, and reported no
measurable change on x86-64.


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201275 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-12 23:54:07 +00:00
Andrea Di Biagio
029a76b0a2 [Vectorizer] Add a new 'OperandValueKind' in TargetTransformInfo called
'OK_NonUniformConstValue' to identify operands which are constants but
not constant splats.

The cost model now allows returning 'OK_NonUniformConstValue'
for non splat operands that are instances of ConstantVector or
ConstantDataVector.

With this change, targets are now able to compute different costs
for instructions with non-uniform constant operands.
For example, On X86 the cost of a vector shift may vary depending on whether
the second operand is a uniform or non-uniform constant.

This patch applies the following changes:
 - The cost model computation now takes into account non-uniform constants;
 - The cost of vector shift instructions has been improved in
   X86TargetTransformInfo analysis pass;
 - BBVectorize, SLPVectorizer and LoopVectorize now know how to distinguish
   between non-uniform and uniform constant operands.

Added a new test to verify that the output of opt
'-cost-model -analyze' is valid in the following configurations: SSE2,
SSE4.1, AVX, AVX2.



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201272 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-12 23:43:47 +00:00
Benjamin Kramer
1e6240a85d InstCombine: Teach icmp merging about the equivalence of bit tests and UGE/ULT with a power of 2.
This happens in bitfield code. While there reorganize the existing code
a bit.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201176 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-11 21:09:03 +00:00
Chandler Carruth
8615ab4a4a [LPM] Switch LICM to actively use LCSSA in addition to preserving it.
Fixes PR18753 and PR18782.

This is necessary for LICM to preserve LCSSA correctly and efficiently.
There is still some active discussion about whether we should be using
LCSSA, but we can't just immediately stop using it and we *need* LICM to
preserve it while we are using it. We can restore the old SSAUpdater
driven code if and when there is a serious effort to remove the reliance
on LCSSA from all of the loop passes.

However, this also serves as a great example of why LCSSA is very nice
to have. This change significantly simplifies the process of sinking
instructions for LICM, and makes it quite a bit less expensive.

It wouldn't even be as complex as it is except that I had to start the
process of removing the big recursive LCSSA formation hammer in order to
switch even this much of the re-forming code to asserting that LCSSA was
preserved. I'll fully remove that next just to tidy things up until the
LCSSA debate settles one way or the other.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201148 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-11 12:52:27 +00:00
Quentin Colombet
921f0b1d66 [CodeGenPrepare] Undo changes that happened for the profitability check.
The addressing mode matcher checks at some point the profitability of folding an
instruction into the addressing mode. When the instruction to be folded has
several uses, it checks that the instruction can be folded in each use.
To do so, it creates a new matcher for each use and check if the instruction is
in the list of the matched instructions of this new matcher.

The new matchers may promote some instructions and this has to be undone to keep
the state of the original matcher consistent.

A test case will follow.

<rdar://problem/16020230>


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201121 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-11 01:59:02 +00:00
Chandler Carruth
d5a9ea8afe [LPM] A terribly simple fix to a terribly complex bug: PR18773.
The crux of the issue is that LCSSA doesn't preserve stateful alias
analyses. Before r200067, LICM didn't cause LCSSA to run in the LTO pass
manager, where LICM runs essentially without any of the other loop
passes. As a consequence the globalmodref-aa pass run before that loop
pass manager was able to survive the loop pass manager and be used by
DSE to eliminate stores in the function called from the loop body in
Adobe-C++/loop_unroll (and similar patterns in other benchmarks).

When LICM was taught to preserve LCSSA it had to require it as well.
This caused it to be run in the loop pass manager and because it did not
preserve AA, the stateful AA was lost. Most of LLVM's AA isn't stateful
and so this didn't manifest in most cases. Also, in most cases LCSSA was
already running, and so there was no interesting change.

The real kicker is that LCSSA by its definition (injecting PHI nodes
only) trivially preserves AA! All we need to do is mark it, and then
everything goes back to working as intended. It probably was blocking
some other weird cases of stateful AA but the only one I have is
a 1000-line IR test case from loop_unroll, so I don't really have a good
test case here.

Hopefully this fixes the regressions on performance that have been seen
since that revision.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201104 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-10 19:39:35 +00:00
Benjamin Kramer
299918ad48 Make succ_iterator a real random access iterator and clean up a couple of users.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201088 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-10 14:17:42 +00:00
Kostya Serebryany
8cab11a9f8 [asan] support for FreeBSD, LLVM part. patch by Viktor Kutuzov
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201067 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-10 07:37:04 +00:00
Arnold Schwaighofer
846acbeef1 LoopVectorizer: Keep track of conditional store basic blocks
Before conditional store vectorization/unrolling we had only one
vectorized/unrolled basic block. After adding support for conditional store
vectorization this will not only be one block but multiple basic blocks. The
last block would have the back-edge. I updated the code to use a vector of basic
blocks instead of a single basic block and fixed the users to use the last entry
in this vector. But, I forgot to add the basic blocks to this vector!

Fixes PR18724.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201028 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-08 20:41:13 +00:00
Juergen Ributzka
6f1819f2e6 [Constant Hoisting] Fix insertion point for constant materialization.
The bitcast instruction during constant materialization was not placed correcly
in the presence of phi nodes. This commit fixes the insertion point to be in the
idom instead.

This fixes PR18768

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201009 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-08 00:20:49 +00:00
Juergen Ributzka
1368e659d7 [Constant Hoisting] Don't update the use list while traversing it - DOH!
This fix first traverses the whole use list of the constant expression and
keeps track of the instructions that need to be updated. Then perform the
fixup afterwards.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@201008 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-08 00:20:45 +00:00
Quentin Colombet
30c0f72237 [CodeGenPrepare] Move away sign extensions that get in the way of addressing
mode.

Basically the idea is to transform code like this:
%idx = add nsw i32 %a, 1
%sextidx = sext i32 %idx to i64
%gep = gep i8* %myArray, i64 %sextidx
load i8* %gep

Into:
%sexta = sext i32 %a to i64
%idx = add nsw i64 %sexta, 1
%gep = gep i8* %myArray, i64 %idx
load i8* %gep

That way the computation can be folded into the addressing mode.

This transformation is done as part of the addressing mode matcher.
If the matching fails (not profitable, addressing mode not legal, etc.), the
matcher will revert the related promotions.

<rdar://problem/15519855>


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200947 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-06 21:44:56 +00:00
Nick Lewycky
44e40408ee A memcpy out of an fresh alloca is a no-op, delete it. Patch by Patrick Walton!
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200907 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-06 06:29:19 +00:00
Manman Ren
c7ac256d52 Set default of inlinecold-threshold to 225.
225 is the default value of inline-threshold. This change will make sure
we have the same inlining behavior as prior to r200886.

As Chandler points out, even though we don't have code in our testing
suite that uses cold attribute, there are larger applications that do
use cold attribute.

r200886 + this commit intend to keep the same behavior as prior to r200886.
We can later on tune the inlinecold-threshold.

The main purpose of r200886 is to help performance of instrumentation based
PGO before we actually hook up inliner with analysis passes such as BPI and BFI.
For instrumentation based PGO, we try to increase inlining of hot functions and
reduce inlining of cold functions by setting inlinecold-threshold.

Another option suggested by Chandler is to use a boolean flag that controls
if we should use OptSizeThreshold for cold functions. The default value
of the boolean flag should not change the current behavior. But it gives us
less freedom in controlling inlining of cold functions.


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200898 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-06 01:59:22 +00:00
Paul Robinson
2684ddd72e Disable most IR-level transform passes on functions marked 'optnone'.
Ideally only those transform passes that run at -O0 remain enabled,
in reality we get as close as we reasonably can.
Passes are responsible for disabling themselves, it's not the job of
the pass manager to do it for them.


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200892 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-06 00:07:05 +00:00
Manman Ren
df7da79db6 Inliner uses a smaller inline threshold for callees with cold attribute.
Added command line option inlinecold-threshold to set threshold for inlining
functions with cold attribute. Listen to the cold attribute when it would
decrease the inline threshold.


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200886 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-05 22:53:44 +00:00
Benjamin Kramer
fb0ad6bd15 SimplifyLibCalls: Push TLI through the exp2->ldexp transform.
For the odd case of platforms with exp2 available but not ldexp.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200795 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-04 20:27:23 +00:00
Duncan P. N. Exon Smith
483727da48 cleanup: scc_iterator consumers should use isAtEnd
No functional change.  Updated loops from:

    for (I = scc_begin(), E = scc_end(); I != E; ++I)

to:

    for (I = scc_begin(); !I.isAtEnd(); ++I)

for teh win.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200789 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-04 19:19:07 +00:00
Tim Northover
8f0354c973 OS X: the correct function is __sincospif_stret, not __sincospi_stretf
rdar://problem/13729466

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200771 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-04 16:28:20 +00:00
Kai Nacke
6840e895c1 Add strchr(p, 0) -> p + strlen(p) to SimplifyLibCalls
Add the missing transformation strchr(p, 0) -> p + strlen(p) to SimplifyLibCalls
and remove the ToDo comment.

Reviewer: Duncan P.N. Exan Smith


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200736 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-04 05:55:16 +00:00
Nick Lewycky
e4d1a3e352 Self-memcpy-elision and memcpy of constant byte to memset transforms don't care how many bytes you were trying to transfer. Sink that safety test after those transforms. Noticed by inspection.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200726 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-04 00:18:54 +00:00
Reid Kleckner
81558937d7 inalloca: Don't remove dead arguments in the presence of inalloca args
It disturbs the layout of the parameters in memory and registers,
leading to problems in the backend.

The plan for optimizing internal inalloca functions going forward is to
essentially SROA the argument memory and demote any captured arguments
(things that aren't trivially written by a load or store) to an indirect
pointer to a static alloca.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200717 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-03 20:42:49 +00:00
Duncan P. N. Exon Smith
e6562c5088 Lower llvm.expect intrinsic correctly for i1
LowerExpectIntrinsic previously only understood the idiom of an expect
intrinsic followed by a comparison with zero. For llvm.expect.i1, the
comparison would be stripped by the early-cse pass.

Patch by Daniel Micay.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200664 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-02 22:43:55 +00:00
Arnold Schwaighofer
a16c1b55e2 LoopVectorizer: Enable unrolling of conditional stores and the load/store
unrolling heuristic per default

Benchmarking on x86_64 (thanks Chandler!) and ARM has shown those options speed
up some benchmarks while not causing any interesting regressions.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200621 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-02 03:12:34 +00:00
Chandler Carruth
115fd30b24 [LPM] Apply a really big hammer to fix PR18688 by recursively reforming
LCSSA when we promote to SSA registers inside of LICM.

Currently, this is actually necessary. The promotion logic in LICM uses
SSAUpdater which doesn't understand how to place LCSSA PHI nodes.
Teaching it to do so would be a very significant undertaking. It may be
worthwhile and I've left a FIXME about this in the code as well as
starting a thread on llvmdev to try to figure out the right long-term
solution.

For now, the PR needs to be fixed. Short of using the promition
SSAUpdater to place both the LCSSA PHI nodes and the promoted PHI nodes,
I don't see a cleaner or cheaper way of achieving this. Fortunately,
LCSSA is relatively lazy and sparse -- it should only update
instructions which need it. We can also skip the recursive variant when
we don't promote to SSA values.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200612 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-01 13:35:14 +00:00
Eli Bendersky
a7bc25e34c Remove some unused #includes
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200611 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-01 13:12:54 +00:00
Reid Kleckner
86cb795388 Revert "[SLPV] Recognize vectorizable intrinsics during SLP vectorization ..."
This reverts commit r200576.  It broke 32-bit self-host builds by
vectorizing two calls to @llvm.bswap.i64, which we then fail to expand.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200602 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-01 01:37:30 +00:00
Chandler Carruth
093b0413fe [SLPV] Recognize vectorizable intrinsics during SLP vectorization and
transform accordingly. Based on similar code from Loop vectorization.
Subsequent commits will include vectorization of function calls to
vector intrinsics and form function calls to vector library calls.

Patch by Raul Silvera! (Much delayed due to my not running dcommit)

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200576 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-31 21:14:40 +00:00
Chandler Carruth
93228f6199 [vectorizer] Tweak the way we do small loop runtime unrolling in the
loop vectorizer to not do so when runtime pointer checks are needed and
share code with the new (not yet enabled) load/store saturation runtime
unrolling. Also ensure that we only consider the runtime checks when the
loop hasn't already been vectorized. If it has, the runtime check cost
has already been paid.

I've fleshed out a test case to cover the scalar unrolling as well as
the vector unrolling and comment clearly why we are or aren't following
the pattern.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200530 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-31 10:51:08 +00:00
Bob Wilson
8498c08404 Fix a bug in gcov instrumentation introduced by r195513. <rdar://15930350>
The entry block of a function starts with all the static allocas. The change
in r195513 splits the block before those allocas, which has the effect of
turning them into dynamic allocas. That breaks all sorts of things. Change to
split after the initial allocas, and also add a comment explaining why the
block is split.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200515 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-31 05:24:01 +00:00
Chandler Carruth
a403ceb205 [LPM] Fix PR18643, another scary place where loop transforms failed to
preserve loop simplify of enclosing loops.

The problem here starts with LoopRotation which ends up cloning code out
of the latch into the new preheader it is buidling. This can create
a new edge from the preheader into the exit block of the loop which
breaks LoopSimplify form. The code tries to fix this by splitting the
critical edge between the latch and the exit block to get a new exit
block that only the latch dominates. This sadly isn't sufficient.

The exit block may be an exit block for multiple nested loops. When we
clone an edge from the latch of the inner loop to the new preheader
being built in the outer loop, we create an exiting edge from the outer
loop to this exit block. Despite breaking the LoopSimplify form for the
inner loop, this is fine for the outer loop. However, when we split the
edge from the inner loop to the exit block, we create a new block which
is in neither the inner nor outer loop as the new exit block. This is
a predecessor to the old exit block, and so the split itself takes the
outer loop out of LoopSimplify form. We need to split every edge
entering the exit block from inside a loop nested more deeply than the
exit block in order to preserve all of the loop simplify constraints.

Once we try to do that, a problem with splitting critical edges
surfaces. Previously, we tried a very brute force to update LoopSimplify
form by re-computing it for all exit blocks. We don't need to do this,
and doing this much will sometimes but not always overlap with the
LoopRotate bug fix. Instead, the code needs to specifically handle the
cases which can start to violate LoopSimplify -- they aren't that
common. We need to see if the destination of the split edge was a loop
exit block in simplified form for the loop of the source of the edge.
For this to be true, all the predecessors need to be in the exact same
loop as the source of the edge being split. If the dest block was
originally in this form, we have to split all of the deges back into
this loop to recover it. The old mechanism of doing this was
conservatively correct because at least *one* of the exiting blocks it
rewrote was the DestBB and so the DestBB's predecessors were fixed. But
this is a much more targeted way of doing it. Making it targeted is
important, because ballooning the set of edges touched prevents
LoopRotate from being able to split edges *it* needs to split to
preserve loop simplify in a coherent way -- the critical edge splitting
would sometimes find the other edges in need of splitting but not
others.

Many, *many* thanks for help from Nick reducing these test cases
mightily. And helping lots with the analysis here as this one was quite
tricky to track down.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200393 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-29 13:16:53 +00:00
Chandler Carruth
6a67a3f3ec [LPM] Fix PR18642, a pretty nasty bug in IndVars that "never mattered"
because of the inside-out run of LoopSimplify in the LoopPassManager and
the fact that LoopSimplify couldn't be "preserved" across two
independent LoopPassManagers.

Anyways, in that case, IndVars wasn't correctly preserving an LCSSA PHI
node because it thought it was rewriting (via SCEV) the incoming value
to a loop invariant value. While it may well be invariant for the
current loop, it may be rewritten in terms of an enclosing loop's
values. This in and of itself is fine, as the LCSSA PHI node in the
enclosing loop for the inner loop value we're rewriting will have its
own LCSSA PHI node if used outside of the enclosing loop. With me so
far?

Well, the current loop and the enclosing loop may share an exiting
block and exit block, and when they do they also share LCSSA PHI nodes.
In this case, its not valid to RAUW through the LCSSA PHI node.

Expected crazy test included.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200372 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-29 04:40:19 +00:00
Arnold Schwaighofer
8dc253e97b LoopVectorizer: Don't count the induction variable multiple times
When estimating register pressure, don't count the induction variable mulitple
times. It is unlikely to be unrolled. This is currently disabled and hidden
behind a flag ("enable-ind-var-reg-heur").

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200371 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-29 04:36:12 +00:00
Rafael Espindola
f611ae40fd Fix pr14893.
When simplifycfg moves an instruction, it must drop metadata it doesn't know
is still valid with the preconditions changes. In particular, it must drop
the range and tbaa metadata.

The patch implements this with an utility function to drop all metadata not
in a white list.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200322 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-28 16:56:46 +00:00
Chandler Carruth
05d43d8b6f [vectorizer] Completely disable the block frequency guidance of the loop
vectorizer, placing it behind an off-by-default flag.

It turns out that block frequency isn't what we want at all, here or
elsewhere. This has been I think a nagging feeling for several of us
working with it, but Arnold has given some really nice simple examples
where the results are so comprehensively wrong that they aren't useful.

I'm planning to email the dev list with a summary of why its not really
useful and a couple of ideas about how to better structure these types
of heuristics.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200294 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-28 09:10:41 +00:00
Reid Kleckner
59bec0e3c0 Update optimization passes to handle inalloca arguments
Summary:
I searched Transforms/ and Analysis/ for 'ByVal' and updated those call
sites to check for inalloca if appropriate.

I added tests for any change that would allow an optimization to fire on
inalloca.

Reviewers: nlewycky

Differential Revision: http://llvm-reviews.chandlerc.com/D2449

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200281 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-28 02:38:36 +00:00
Chandler Carruth
5ea4a54551 [LPM] Fix PR18616 where the shifts to the loop pass manager to extract
LCSSA from it caused a crasher with the LoopUnroll pass.

This crasher is really nasty. We destroy LCSSA form in a suprising way.
When unrolling a loop into an outer loop, we not only need to restore
LCSSA form for the outer loop, but for all children of the outer loop.
This is somewhat obvious in retrospect, but hey!

While this seems pretty heavy-handed, it's not that bad. Fundamentally,
we only do this when we unroll a loop, which is already a heavyweight
operation. We're unrolling all of these hypothetical inner loops as
well, so their size and complexity is already on the critical path. This
is just adding another pass over them to re-canonicalize.

I have a test case from PR18616 that is great for reproducing this, but
pretty useless to check in as it relies on many 10s of nested empty
loops that get unrolled and deleted in just the right order. =/ What's
worse is that investigating this has exposed another source of failure
that is likely to be even harder to test. I'll try to come up with test
cases for these fixes, but I want to get the fixes into the tree first
as they're causing crashes in the wild.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200273 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-28 01:25:38 +00:00
Arnold Schwaighofer
a47aa4b4ef LoopVectorize: Support conditional stores by scalarizing
The vectorizer takes a loop like this and widens all instructions except for the
store. The stores are scalarized/unrolled and hidden behind an "if" block.

  for (i = 0; i < 128; ++i) {
    if (a[i] < 10)
      a[i] += val;
  }

  for (i = 0; i < 128; i+=2) {
    v = a[i:i+1];
    v0 = (extract v, 0) + 10;
    v1 = (extract v, 1) + 10;
    if (v0 < 10)
      a[i] = v0;
    if (v1 < 10)
      a[i] = v1;
  }

The vectorizer relies on subsequent optimizations to sink instructions into the
conditional block where they are anticipated.

The flag "vectorize-num-stores-pred" controls whether and how many stores to
handle this way. Vectorization of conditional stores is disabled per default for
now.

This patch also adds a change to the heuristic when the flag
"enable-loadstore-runtime-unroll" is enabled (off by default). It unrolls small
loops until load/store ports are saturated. This heuristic uses TTI's
getMaxUnrollFactor as a measure for load/store ports.

I also added a second flag -enable-cond-stores-vec. It will enable vectorization
of conditional stores. But there is no cost model for vectorization of
conditional stores in place yet so this will not do good at the moment.

rdar://15892953

Results for x86-64 -O3 -mavx +/- -mllvm -enable-loadstore-runtime-unroll
-vectorize-num-stores-pred=1 (before the BFI change):

 Performance Regressions:
   Benchmarks/Ptrdist/yacr2/yacr2 7.35% (maze3() is identical but 10% slower)
   Applications/siod/siod         2.18%
 Performance improvements:
   mesa                          -4.42%
   libquantum                    -4.15%

 With a patch that slightly changes the register heuristics (by subtracting the
 induction variable on both sides of the register pressure equation, as the
 induction variable is probably not really unrolled):

 Performance Regressions:
   Benchmarks/Ptrdist/yacr2/yacr2  7.73%
   Applications/siod/siod          1.97%

 Performance Improvements:
   libquantum                    -13.05% (we now also unroll quantum_toffoli)
   mesa                           -4.27%

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200270 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-28 01:01:53 +00:00
Manman Ren
aa6627016f PGO branch weight: keep halving the weights until they can fit into
uint32.

When folding branches to common destination, the updated branch weights
can exceed uint32 by more than factor of 2. We should keep halving the
weights until they can fit into uint32.


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200262 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-27 23:39:03 +00:00
Chandler Carruth
5f61e70eac [vectorize] Initial version of respecting PGO in the vectorizer: treat
cold loops as-if they were being optimized for size.

Nothing fancy here. Simply test case included. The nice thing is that we
can now incrementally build on top of this to drive other heuristics.
All of the infrastructure work is done to get the profile information
into this layer.

The remaining work necessary to make this a fully general purpose loop
unroller for very hot loops is to make it a fully general purpose loop
unroller. Things I know of but am not going to have time to benchmark
and fix in the immediate future:

1) Don't disable the entire pass when the target is lacking vector
   registers. This really doesn't make any sense any more.
2) Teach the unroller at least and the vectorizer potentially to handle
   non-if-converted loops. This is trivial for the unroller but hard for
   the vectorizer.
3) Compute the relative hotness of the loop and thread that down to the
   various places that make cost tradeoffs (very likely only the
   unroller makes sense here, and then only when dealing with loops that
   are small enough for unrolling to not completely blow out the LSD).

I'm still dubious how useful hotness information will be. So far, my
experiments show that if we can get the correct logic for determining
when unrolling actually helps performance, the code size impact is
completely unimportant and we can unroll in all cases. But at least
we'll no longer burn code size on cold code.

One somewhat unrelated idea that I've had forever but not had time to
implement: mark all functions which are only reachable via the global
constructors rigging in the module as optsize. This would also decrease
the impact of any more aggressive heuristics here on code size.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200219 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-27 13:11:50 +00:00
Benjamin Kramer
08aa38d39b ConstantHoisting: We can't insert instructions directly in front of a PHI node.
Insert before the terminating instruction of the dominating block instead.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200218 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-27 13:11:43 +00:00
Chandler Carruth
1c4746ed70 [vectorizer] Add an override for the target instruction cost and use it
to stabilize a test that really is trying to test generic behavior and
not a specific target's behavior.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200215 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-27 11:41:50 +00:00
Chandler Carruth
91a3f1dc8e [vectorizer] Simplify code to use existing helpers on the Function
object and fewer pointless variables.

Also, add a clarifying comment and a FIXME because the code which
disables *all* vectorization if we can't use implicit floating point
instructions just makes no sense at all.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200214 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-27 11:27:37 +00:00
Chandler Carruth
424b2b0093 [vectorizer] Teach the loop vectorizer's unroller to only unroll by
powers of two. This is essentially always the correct thing given the
impact on alignment, scaling factors that can be used in addressing
modes, etc. Also, fix the management of the unroll vs. small loop cost
to more accurately model things with this world.

Enhance a test case to actually exercise more of the unroll machinery if
using synthetic constants rather than a specific target model. Before
this change, with the added flags this test will unroll 3 times instead
of either 2 or 4 (the two sensible answers).

While I don't expect this to make a huge difference, if there are lots
of loops sitting right on the edge of hitting the 'small unroll' factor,
they might change behavior. However, I've benchmarked moving the small
loop cost up and down in many various ways and by a huge factor (2x)
without seeing more than 0.2% code size growth. Small adjustments such
as the series that led up here have led to about 1% improvement on some
benchmarks, but it is very close to the noise floor so I mostly checked
that nothing regressed. Let me know if you see bad behavior on other
targets but I don't expect this to be a sufficiently dramatic change to
trigger anything.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200213 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-27 11:12:24 +00:00
Chandler Carruth
9f22a8788f [vectorizer] Add some flags which are useful for conducting experiments
with the unrolling behavior in the loop vectorizer. No functionality
changed at this point.

These are a bit hack-y, but talking with Hal, there doesn't seem to be
a cleaner way to easily experiment with different thresholds here and he
was also interested in them so I wanted to commit them. Suggestions for
improvement are very welcome here.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200212 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-27 11:12:19 +00:00
Chandler Carruth
3fa842d791 [vectorizer] Fix a trivial oversight where we always requested the
number of vector registers rather than toggling between vector and
scalar register number based on VF. I don't have a test case as
I spotted this by inspection and on X86 it only makes a difference if
your target is lacking SSE and thus has *no* vector registers.

If someone wants to add a test case for this for ARM or somewhere else
where this is more significant, that would be awesome.

Also made the variable name a bit more sensible while I'm here.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200211 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-27 11:12:14 +00:00
Chandler Carruth
0afd0bc5fa [vectorizer] Clean up the handling of unvectorized loop unrolling in the
LoopVectorize pass.

The logic here doesn't make much sense. We *only* unrolled if the
unvectorized loop was a reduction loop with a single basic block *and*
small loop body. The reduction part in particular doesn't make much
sense. Instead, if we just fall through to the vectorized unroll logic
it makes more sense of unrolling if there is a vectorized reduction that
could be hacked on by the SLP vectorizer *or* if the loop is small.

This is mostly a cleanup and nothing in the test suite really exercises
this, but I did run benchmarks across this change and saw no really
significant changes.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200198 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-27 08:17:58 +00:00
Chandler Carruth
754b83a51a [LPM] Conclude my immediate work by making the LoopVectorizer
a FunctionPass. With this change the loop vectorizer no longer is a loop
pass and can readily depend on function analyses. In particular, with
this change we no longer have to form a loop pass manager to run the
loop vectorizer which simplifies the entire pass management of LLVM.

The next step here is to teach the loop vectorizer to leverage profile
information through the profile information providing analysis passes.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200074 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-25 10:01:55 +00:00
Chandler Carruth
3d69cf57e1 [LPM] Make LCSSA a utility with a FunctionPass that applies it to all
the loops in a function, and teach LICM to work in the presance of
LCSSA.

Previously, LCSSA was a loop pass. That made passes requiring it also be
loop passes and unable to depend on function analysis passes easily. It
also caused outer loops to have a different "canonical" form from inner
loops during analysis. Instead, we go into LCSSA form and preserve it
through the loop pass manager run.

Note that this has the same problem as LoopSimplify that prevents
enabling its verification -- loop passes which run at the end of the loop
pass manager and don't preserve these are valid, but the subsequent loop
pass runs of outer loops that do preserve this pass trigger too much
verification and fail because the inner loop no longer verifies.

The other problem this exposed is that LICM was completely unable to
handle LCSSA form. It didn't preserve it and it actually would give up
on moving instructions in many cases when they were used by an LCSSA phi
node. I've taught LICM to support detecting LCSSA-form PHI nodes and to
hoist and sink around them. This may actually let LICM fire
significantly more because we put everything into LCSSA form to rotate
the loop before running LICM. =/ Now LICM should handle that fine and
preserve it correctly. The down side is that LICM has to require LCSSA
in order to preserve it. This is just a fact of life for LCSSA. It's
entirely possible we should completely remove LCSSA from the optimizer.

The test updates are essentially accomodating LCSSA phi nodes in the
output of LICM, and the fact that we now completely sink every
instruction in ashr-crash below the loop bodies prior to unrolling.

With this change, LCSSA is computed only three times in the pass
pipeline. One of them could be removed (and potentially a SCEV run and
a separate LoopPassManager entirely!) if we had a LoopPass variant of
InstCombine that ran InstCombine on the loop body but refused to combine
away LCSSA PHI nodes. Currently, this also prevents loop unrolling from
being in the same loop pass manager is rotate, LICM, and unswitch.

There is one thing that I *really* don't like -- preserving LCSSA in
LICM is quite expensive. We end up having to re-run LCSSA twice for some
loops after LICM runs because LICM can undo LCSSA both in the current
loop and the parent loop. I don't really see good solutions to this
other than to completely move away from LCSSA and using tools like
SSAUpdater instead.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200067 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-25 04:07:24 +00:00
Juergen Ributzka
943ce55f39 Revert "Revert "Add Constant Hoisting Pass" (r200034)"
This reverts commit r200058 and adds the using directive for
ARMTargetTransformInfo to silence two g++ overload warnings.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200062 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-25 02:02:55 +00:00
Hans Wennborg
503793e834 Revert "Add Constant Hoisting Pass" (r200034)
This commit caused -Woverloaded-virtual warnings. The two new
TargetTransformInfo::getIntImmCost functions were only added to the superclass,
and to the X86 subclass. The other targets were not updated, and the
warning highlighted this by pointing out that e.g. ARMTTI::getIntImmCost was
hiding the two new getIntImmCost variants.

We could pacify the warning by adding "using TargetTransformInfo::getIntImmCost"
to the various subclasses, or turning it off, but I suspect that it's wrong to
leave the functions unimplemnted in those targets. The default implementations
return TCC_Free, which I don't think is right e.g. for ARM.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200058 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-25 01:18:18 +00:00
Juergen Ributzka
96172cb4a4 Add Constant Hoisting Pass
Retry commit r200022 with a fix for the build bot errors. Constant expressions
have (unlike instructions) module scope use lists and therefore may have users
in different functions. The fix is to simply ignore these out-of-function uses.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200034 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-24 20:18:00 +00:00
Benjamin Kramer
d7053be532 InstCombine: Don't try to use aggregate elements of ConstantExprs.
PR18600.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200028 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-24 19:02:37 +00:00
Juergen Ributzka
dc6f9b9a4f Revert "Add Constant Hoisting Pass"
This reverts commit r200022 to unbreak the build bots.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200024 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-24 18:40:30 +00:00
Juergen Ributzka
fb282c68b7 Add Constant Hoisting Pass
This pass identifies expensive constants to hoist and coalesces them to
better prepare it for SelectionDAG-based code generation. This works around the
limitations of the basic-block-at-a-time approach.

First it scans all instructions for integer constants and calculates its
cost. If the constant can be folded into the instruction (the cost is
TCC_Free) or the cost is just a simple operation (TCC_BASIC), then we don't
consider it expensive and leave it alone. This is the default behavior and
the default implementation of getIntImmCost will always return TCC_Free.

If the cost is more than TCC_BASIC, then the integer constant can't be folded
into the instruction and it might be beneficial to hoist the constant.
Similar constants are coalesced to reduce register pressure and
materialization code.

When a constant is hoisted, it is also hidden behind a bitcast to force it to
be live-out of the basic block. Otherwise the constant would be just
duplicated and each basic block would have its own copy in the SelectionDAG.
The SelectionDAG recognizes such constants as opaque and doesn't perform
certain transformations on them, which would create a new expensive constant.

This optimization is only applied to integer constants in instructions and
simple (this means not nested) constant cast experessions. For example:
%0 = load i64* inttoptr (i64 big_constant to i64*)

Reviewed by Eric

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200022 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-24 18:23:08 +00:00
Alp Toker
ae43cab6ba Fix known typos
Sweep the codebase for common typos. Includes some changes to visible function
names that were misspelt.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200018 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-24 17:20:08 +00:00
Chandler Carruth
4296ce5662 [LPM] Fix a logic error in LICM spotted by inspection.
We completely skipped promotion in LICM if the loop has a preheader or
dedicated exits, but not *both*. We hoist if there is a preheader, and
sink if there are dedicated exits, but either hoisting or sinking can
move loop invariant code out of the loop!

I have no idea if this has a practical consequence. If anyone has ideas
for a test case, let me know.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199966 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-24 02:24:47 +00:00
Chandler Carruth
42e23de4db [cleanup] Use the type-based preservation method rather than a string
literal that bakes a pass name and forces parsing it in the pass
manager.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199963 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-24 01:59:49 +00:00
Rafael Espindola
35b78fd04c Remove tail marker when changing an argument to an alloca.
Argument promotion can replace an argument of a call with an alloca. This
requires clearing the tail marker as it is very likely that the callee is now
using an alloca in the caller.

This fixes pr14710.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199909 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-23 17:19:42 +00:00
Chandler Carruth
aaf44af769 [LPM] Make LoopSimplify no longer a LoopPass and instead both a utility
function and a FunctionPass.

This has many benefits. The motivating use case was to be able to
compute function analysis passes *after* running LoopSimplify (to avoid
invalidating them) and then to run other passes which require
LoopSimplify. Specifically passes like unrolling and vectorization are
critical to wire up to BranchProbabilityInfo and BlockFrequencyInfo so
that they can be profile aware. For the LoopVectorize pass the only
things in the way are LoopSimplify and LCSSA. This fixes LoopSimplify
and LCSSA is next on my list.

There are also a bunch of other benefits of doing this:
- It is now very feasible to make more passes *preserve* LoopSimplify
  because they can simply run it after changing a loop. Because
  subsequence passes can assume LoopSimplify is preserved we can reduce
  the runs of this pass to the times when we actually mutate a loop
  structure.
- The new pass manager should be able to more easily support loop passes
  factored in this way.
- We can at long, long last observe that LoopSimplify is preserved
  across SCEV. This *halves* the number of times we run LoopSimplify!!!

Now, getting here wasn't trivial. First off, the interfaces used by
LoopSimplify are all over the map regarding how analysis are updated. We
end up with weird "pass" parameters as a consequence. I'll try to clean
at least some of this up later -- I'll have to have it all clean for the
new pass manager.

Next up I discovered a really frustrating bug. LoopUnroll *claims* to
preserve LoopSimplify. That's actually a lie. But the way the
LoopPassManager ends up running the passes, it always ran LoopSimplify
on the unrolled-into loop, rectifying this oversight before any
verification could kick in and point out that in fact nothing was
preserved. So I've added code to the unroller to *actually* simplify the
surrounding loop when it succeeds at unrolling.

The only functional change in the test suite is that we now catch a case
that was previously missed because SCEV and other loop transforms see
their containing loops as simplified and thus don't miss some
opportunities. One test case has been converted to check that we catch
this case rather than checking that we miss it but at least don't get
the wrong answer.

Note that I have #if-ed out all of the verification logic in
LoopSimplify! This is a temporary workaround while extracting these bits
from the LoopPassManager. Currently, there is no way to have a pass in
the LoopPassManager which preserves LoopSimplify along with one which
does not. The LPM will try to verify on each loop in the nest that
LoopSimplify holds but the now-Function-pass cannot distinguish what
loop is being verified and so must try to verify all of them. The inner
most loop is clearly no longer simplified as there is a pass which
didn't even *attempt* to preserve it. =/ Once I get LCSSA out (and maybe
LoopVectorize and some other fixes) I'll be able to re-enable this check
and catch any places where we are still failing to preserve
LoopSimplify. If this causes problems I can back this out and try to
commit *all* of this at once, but so far this seems to work and allow
much more incremental progress.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199884 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-23 11:23:19 +00:00
Matt Arsenault
88a9f0476c Handle an addrspacecast case in memcpyopt
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199836 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-22 21:53:19 +00:00
Tim Northover
b9b629cbaa Loop strength reduce: fix function name.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199801 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-22 13:27:00 +00:00
Chandler Carruth
c04f2c99ab [SROA] Fix a bug which could cause the common type finding to return
inconsistent results for different orderings of alloca slices. The
fundamental issue is that it is just always a mistake to return early
from this function. There is no effective early exit to leverage. This
patch stops trynig to do so and simplifies the code a bit as
a consequence.

Original diagnosis and patch by James Molloy with some name tweaks by me
in part reflecting feedback from Duncan Smith on the mailing list.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199771 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-21 23:16:05 +00:00
Owen Anderson
1e1446bf84 Fix all the remaining lost-fast-math-flags bugs I've been able to find. The most important of these are cases in the generic logic for combining BinaryOperators.
This logic hadn't been updated to handle FastMathFlags, and it took me a while to detect it because it doesn't show up in a simple search for CreateFAdd.


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199629 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-20 07:44:53 +00:00
Benjamin Kramer
b45edea9b3 InstCombine: Modernize a bunch of cast combines.
Also make them vector-aware.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199608 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-19 20:05:13 +00:00
Benjamin Kramer
2b03d0051f InstCombine: Hoist 3 copies of AddOne/SubOne into a header.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199605 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-19 16:56:10 +00:00
Benjamin Kramer
c7645e860a InstCombine: Replace a hand-rolled version of isKnownToBeAPowerOfTwo with the real thing.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199604 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-19 16:48:41 +00:00
Benjamin Kramer
0487faa97b InstCombine: Teach most integer add/sub/mul/div combines how to deal with vectors.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199602 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-19 15:24:22 +00:00
Benjamin Kramer
3f6a9d705a InstCombine: Refactor fmul/fdiv combines to handle vectors.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199598 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-19 13:36:27 +00:00
Chandler Carruth
e1a5243053 Fix a really nasty SROA bug with how we handled out-of-bounds memcpy
intrinsics.

Reported on the list by Evan with a couple of attempts to fix, but it
took a while to dig down to the root cause. There are two overlapping
bugs here, both centering around the circumstance of discovering
a memcpy operand which is known to be completely outside the bounds of
the alloca.

First, we need to kill the *other* side of the memcpy if it was added to
this alloca. Otherwise we'll factor it into our slicing and try to
rewrite it even though we know for a fact that it is dead. This is made
more tricky because we can visit the sides in either order. So we have
to both kill the other side and skip instructions marked as dead. The
latter really should be goodness in every case, but here is a matter of
correctness.

Second, we need to actually remove the *uses* of the alloca by the
memcpy when queuing it for later deletion. Otherwise it may still be
using the alloca when we go to promote it (if the rewrite re-uses the
existing alloca instruction). Do this by factoring out the
use-clobbering used when for nixing a Phi argument and re-using it
across the operands of a to-be-deleted instruction.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199590 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-19 12:16:54 +00:00
Arnold Schwaighofer
2becaaf3a1 LoopVectorizer: A reduction that has multiple uses of the reduction value is not
a reduction.

Really. Under certain circumstances (the use list of an instruction has to be
set up right - hence the extra pass in the test case) we would not recognize
when a value in a potential reduction cycle was used multiple times by the
reduction cycle.

Fixes PR18526.
radar://15851149

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199570 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-19 03:18:31 +00:00
Nick Lewycky
6d2bd95ff1 Don't refuse to transform constexpr(call(arg, ...)) to call(constexpr(arg), ...)) just because the function has multiple return values even if their return types are the same. Patch by Eduard Burtescu!
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199564 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-18 22:47:12 +00:00
Benjamin Kramer
8e937c39bb InstCombine: Make the (fmul X, -1.0) -> (fsub -0.0, X) transform handle vectors too.
PR18532.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199553 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-18 16:43:14 +00:00
Owen Anderson
774cec5748 Fix more instances of dropped fast math flags when optimizing FADD instructions. All found by inspection (aka grep).
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199528 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-18 00:48:14 +00:00
Kostya Serebryany
bd2c711cdd [asan] extend asan-coverage (still experimental).
- add a mode for collecting per-block coverage (-asan-coverage=2).
   So far the implementation is naive (all blocks are instrumented),
   the performance overhead on top of asan could be as high as 30%.
 - Make sure the one-time calls to __sanitizer_cov are moved to function buttom,
   which in turn required to copy the original debug info into the call insn.

Here is the performance data on SPEC 2006
(train data, comparing asan with asan-coverage={0,1,2}):

                             asan+cov0     asan+cov1      diff 0-1    asan+cov2       diff 0-2      diff 1-2
       400.perlbench,        65.60,        65.80,         1.00,        76.20,         1.16,         1.16
           401.bzip2,        65.10,        65.50,         1.01,        75.90,         1.17,         1.16
             403.gcc,         1.64,         1.69,         1.03,         2.04,         1.24,         1.21
             429.mcf,        21.90,        22.60,         1.03,        23.20,         1.06,         1.03
           445.gobmk,       166.00,       169.00,         1.02,       205.00,         1.23,         1.21
           456.hmmer,        88.30,        87.90,         1.00,        91.00,         1.03,         1.04
           458.sjeng,       210.00,       222.00,         1.06,       258.00,         1.23,         1.16
      462.libquantum,         1.73,         1.75,         1.01,         2.11,         1.22,         1.21
         464.h264ref,       147.00,       152.00,         1.03,       160.00,         1.09,         1.05
         471.omnetpp,       115.00,       116.00,         1.01,       140.00,         1.22,         1.21
           473.astar,       133.00,       131.00,         0.98,       142.00,         1.07,         1.08
       483.xalancbmk,       118.00,       120.00,         1.02,       154.00,         1.31,         1.28
            433.milc,        19.80,        20.00,         1.01,        20.10,         1.02,         1.01
            444.namd,        16.20,        16.20,         1.00,        17.60,         1.09,         1.09
          447.dealII,        41.80,        42.20,         1.01,        43.50,         1.04,         1.03
          450.soplex,         7.51,         7.82,         1.04,         8.25,         1.10,         1.05
          453.povray,        14.00,        14.40,         1.03,        15.80,         1.13,         1.10
             470.lbm,        33.30,        34.10,         1.02,        34.10,         1.02,         1.00
         482.sphinx3,        12.40,        12.30,         0.99,        13.00,         1.05,         1.06


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199488 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-17 11:00:30 +00:00
Quentin Colombet
9b24eeee01 [opt][PassInfo] Allow opt to run passes that need target machine.
When registering a pass, a pass can now specify a second construct that takes as
argument a pointer to TargetMachine.
The PassInfo class has been updated to reflect that possibility.
If such a constructor exists opt will use it instead of the default constructor
when instantiating the pass.

Since such IR passes are supposed to be rare, no specific support has been
added to this commit to allow an easy registration of such a pass.
In other words, for such pass, the initialization function has to be
hand-written (see CodeGenPrepare for instance).

Now, codegenprepare can be tested using opt:
opt -codegenprepare -mtriple=mytriple input.ll


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199430 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-16 21:44:34 +00:00
Owen Anderson
5ee5e0c430 Fix two cases where we could lose fast math flags when optimizing FADD expressions.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199427 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-16 21:26:02 +00:00
Owen Anderson
5d9450f92f Fix an instance where we would drop fast math flags when performing an fdiv to reciprocal multiply transformation.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199425 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-16 21:07:52 +00:00
Owen Anderson
da5e148474 Fix a bug in InstCombine where we failed to preserve fast math flags when optimizing an FMUL expression.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199424 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-16 20:59:41 +00:00
Owen Anderson
a2a8bbb30f Teach InstCombine that (fmul X, -1.0) can be simplified to (fneg X), which LLVM expresses as (fsub -0.0, X).
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199420 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-16 20:36:42 +00:00
Evgeniy Stepanov
6d49eafb00 [asan] Remove -fsanitize-address-zero-base-shadow command line
flag from clang, and disable zero-base shadow support on all platforms
where it is not the default behavior.

- It is completely unused, as far as we know.
- It is ABI-incompatible with non-zero-base shadow, which means all
objects in a process must be built with the same setting. Failing to
do so results in a segmentation fault at runtime.
- It introduces a backward dependency of compiler-rt on user code,
which is uncommon and complicates testing.

This is the LLVM part of a larger change.



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199371 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-16 10:19:12 +00:00
Hans Wennborg
89fa06ba0f Switch-to-lookup tables: set threshold to 3 cases
There has been an old FIXME to find the right cut-off for when it's worth
analyzing and potentially transforming a switch to a lookup table.

The switches always have two or more cases. I could not measure any speed-up
by transforming a switch with two cases. A switch with three cases gets a nice
speed-up, and I couldn't measure any compile-time regression, so I think this
is the right threshold.

In a Clang self-host, this causes 480 new switches to be transformed,
and reduces the final binary size with 8 KB.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199294 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-15 05:00:27 +00:00
Arnold Schwaighofer
e96fec2e43 LoopVectorize: Only strip casts from integer types when replacing symbolic
strides

Fixes PR18480.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199291 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-15 03:35:46 +00:00
Matt Arsenault
0445dc203b Do pointer cast simplifications on addrspacecast
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199254 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-14 20:00:45 +00:00
Matt Arsenault
e16fa983bf Remove a check for an illegal condition.
Bitcasts can't be between address spaces anymore.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199253 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-14 19:56:57 +00:00
Matt Arsenault
60ecc44266 Make nocapture analysis work with addrspacecast
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199246 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-14 19:11:52 +00:00
Duncan P. N. Exon Smith
3e4542b2ca Reapply "LTO: add API to set strategy for -internalize"
Reapply r199191, reverted in r199197 because it carelessly broke
Other/link-opts.ll.  The problem was that calling
createInternalizePass("main") would select
createInternalizePass(bool("main")) instead of
createInternalizePass(ArrayRef<const char *>("main")).  This commit
fixes the bug.

The original commit message follows.

Add API to LTOCodeGenerator to specify a strategy for the -internalize
pass.

This is a new attempt at Bill's change in r185882, which he reverted in
r188029 due to problems with the gold linker.  This puts the onus on the
linker to decide whether (and what) to internalize.

In particular, running internalize before outputting an object file may
change a 'weak' symbol into an internal one, even though that symbol
could be needed by an external object file --- e.g., with arclite.

This patch enables three strategies:

- LTO_INTERNALIZE_FULL: the default (and the old behaviour).
- LTO_INTERNALIZE_NONE: skip -internalize.
- LTO_INTERNALIZE_HIDDEN: only -internalize symbols with hidden
  visibility.

LTO_INTERNALIZE_FULL should be used when linking an executable.

Outputting an object file (e.g., via ld -r) is more complicated, and
depends on whether hidden symbols should be internalized.  E.g., for
ld -r, LTO_INTERNALIZE_NONE can be used when -keep_private_externs, and
LTO_INTERNALIZE_HIDDEN can be used otherwise.  However,
LTO_INTERNALIZE_FULL is inappropriate, since the output object file will
eventually need to link with others.

lto_codegen_set_internalize_strategy() sets the strategy for subsequent
calls to lto_codegen_write_merged_modules() and lto_codegen_compile*().

<rdar://problem/14334895>

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199244 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-14 18:52:17 +00:00
Nico Rieck
38f68c5a2e Decouple dllexport/dllimport from linkage
Representing dllexport/dllimport as distinct linkage types prevents using
these attributes on templates and inline functions.

Instead of introducing further mixed linkage types to include linkonce and
weak ODR, the old import/export linkage types are replaced with a new
separate visibility-like specifier:

  define available_externally dllimport void @f() {}
  @Var = dllexport global i32 1, align 4

Linkage for dllexported globals and functions is now equal to their linkage
without dllexport. Imported globals and functions must be either
declarations with external linkage, or definitions with
AvailableExternallyLinkage.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199218 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-14 15:22:47 +00:00
Nico Rieck
55463f4ec1 Revert "Decouple dllexport/dllimport from linkage"
Revert this for now until I fix an issue in Clang with it.

This reverts commit r199204.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199207 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-14 12:38:32 +00:00
Nico Rieck
bce07a0c3b Decouple dllexport/dllimport from linkage
Representing dllexport/dllimport as distinct linkage types prevents using
these attributes on templates and inline functions.

Instead of introducing further mixed linkage types to include linkonce and
weak ODR, the old import/export linkage types are replaced with a new
separate visibility-like specifier:

  define available_externally dllimport void @f() {}
  @Var = dllexport global i32 1, align 4

Linkage for dllexported globals and functions is now equal to their linkage
without dllexport. Imported globals and functions must be either
declarations with external linkage, or definitions with
AvailableExternallyLinkage.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199204 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-14 11:55:03 +00:00
NAKAMURA Takumi
d13b9da1b6 Revert r199191, "LTO: add API to set strategy for -internalize"
Please update also Other/link-opts.ll, in next time.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199197 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-14 09:40:18 +00:00
Duncan P. N. Exon Smith
67af0456bc LTO: add API to set strategy for -internalize
Add API to LTOCodeGenerator to specify a strategy for the -internalize
pass.

This is a new attempt at Bill's change in r185882, which he reverted in
r188029 due to problems with the gold linker.  This puts the onus on the
linker to decide whether (and what) to internalize.

In particular, running internalize before outputting an object file may
change a 'weak' symbol into an internal one, even though that symbol
could be needed by an external object file --- e.g., with arclite.

This patch enables three strategies:

- LTO_INTERNALIZE_FULL: the default (and the old behaviour).
- LTO_INTERNALIZE_NONE: skip -internalize.
- LTO_INTERNALIZE_HIDDEN: only -internalize symbols with hidden
  visibility.

LTO_INTERNALIZE_FULL should be used when linking an executable.

Outputting an object file (e.g., via ld -r) is more complicated, and
depends on whether hidden symbols should be internalized.  E.g., for
ld -r, LTO_INTERNALIZE_NONE can be used when -keep_private_externs, and
LTO_INTERNALIZE_HIDDEN can be used otherwise.  However,
LTO_INTERNALIZE_FULL is inappropriate, since the output object file will
eventually need to link with others.

lto_codegen_set_internalize_strategy() sets the strategy for subsequent
calls to lto_codegen_write_merged_modules() and lto_codegen_compile*().

<rdar://problem/14334895>

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199191 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-14 06:37:26 +00:00
Chandler Carruth
7f2eff792a [PM] Split DominatorTree into a concrete analysis result object which
can be used by both the new pass manager and the old.

This removes it from any of the virtual mess of the pass interfaces and
lets it derive cleanly from the DominatorTreeBase<> template. In turn,
tons of boilerplate interface can be nuked and it turns into a very
straightforward extension of the base DominatorTree interface.

The old analysis pass is now a simple wrapper. The names and style of
this split should match the split between CallGraph and
CallGraphWrapperPass. All of the users of DominatorTree have been
updated to match using many of the same tricks as with CallGraph. The
goal is that the common type remains the resulting DominatorTree rather
than the pass. This will make subsequent work toward the new pass
manager significantly easier.

Also in numerous places things became cleaner because I switched from
re-running the pass (!!! mid way through some other passes run!!!) to
directly recomputing the domtree.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199104 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-13 13:07:17 +00:00
Chandler Carruth
2073b0a63c [PM] Pull the generic graph algorithms and data structures for dominator
trees into the Support library.

These are all expressed in terms of the generic GraphTraits and CFG,
with no reliance on any concrete IR types. Putting them in support
clarifies that and makes the fact that the static analyzer in Clang uses
them much more sane. When moving the Dominators.h file into the IR
library I claimed that this was the right home for it but not something
I planned to work on. Oops.

So why am I doing this? It happens to be one step toward breaking the
requirement that IR verification can only be performed from inside of
a pass context, which completely blocks the implementation of
verification for the new pass manager infrastructure. Fixing it will
also allow removing the concept of the "preverify" step (WTF???) and
allow the verifier to cleanly flag functions which fail verification in
a way that precludes even computing dominance information. Currently,
that results in a fatal error even when you ask the verifier to not
fatally error. It's awesome like that.

The yak shaving will continue...

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199095 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-13 10:52:56 +00:00
Chandler Carruth
56e1394c88 [cleanup] Move the Dominators.h and Verifier.h headers into the IR
directory. These passes are already defined in the IR library, and it
doesn't make any sense to have the headers in Analysis.

Long term, I think there is going to be a much better way to divide
these matters. The dominators code should be fully separated into the
abstract graph algorithm and have that put in Support where it becomes
obvious that evn Clang's CFGBlock's can use it. Then the verifier can
manually construct dominance information from the Support-driven
interface while the Analysis library can provide a pass which both
caches, reconstructs, and supports a nice update API.

But those are very long term, and so I don't want to leave the really
confusing structure until that day arrives.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199082 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-13 09:26:24 +00:00
Chandler Carruth
9f20a4c6ce Re-sort #include lines again, prior to moving headers around.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199080 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-13 08:04:33 +00:00
Hans Wennborg
19236d53eb Switch-to-lookup tables: Don't require a result for the default
case when the lookup table doesn't have any holes.

This means we can build a lookup table for switches like this:

  switch (x) {
    case 0: return 1;
    case 1: return 2;
    case 2: return 3;
    case 3: return 4;
    default: exit(1);
  }

The default case doesn't yield a constant result here, but that doesn't matter,
since a default result is only necessary for filling holes in the lookup table,
and this table doesn't have any holes.

This makes us transform 505 more switches in a clang bootstrap, and shaves 164 KB
off the resulting clang binary.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199025 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-12 00:44:41 +00:00
Arnold Schwaighofer
73c9559237 LoopVectorizer: Enable strided memory accesses versioning per default
I saw no compile or execution time regressions on x86_64 -mavx -O3.

radar://13075509

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199015 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-11 20:40:34 +00:00
NAKAMURA Takumi
55da404566 LoopVectorize.cpp: Appease MSC16.
Excuse me, I hope msc16 builders would be fine till its end day.
Introduce nullptr then. ;)

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@199001 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-11 09:59:27 +00:00
Diego Novillo
4b2b2da9c7 Extend and simplify the sample profile input file.
1- Use the line_iterator class to read profile files.

2- Allow comments in profile file. Lines starting with '#'
   are completely ignored while reading the profile.

3- Add parsing support for discriminators and indirect call samples.

   Our external profiler can emit more profile information that we are
   currently not handling. This patch does not add new functionality to
   support this information, but it allows profile files to provide it.

   I will add actual support later on (for at least one of these
   features, I need support for DWARF discriminators in Clang).

   A sample line may contain the following additional information:

   Discriminator. This is used if the sampled program was compiled with
   DWARF discriminator support
   (http://wiki.dwarfstd.org/index.php?title=Path_Discriminators). This
   is currently only emitted by GCC and we just ignore it.

   Potential call targets and samples. If present, this line contains a
   call instruction. This models both direct and indirect calls. Each
   called target is listed together with the number of samples. For
   example,

                    130: 7  foo:3  bar:2  baz:7

   The above means that at relative line offset 130 there is a call
   instruction that calls one of foo(), bar() and baz(). With baz()
   being the relatively more frequent call target.

   Differential Revision: http://llvm-reviews.chandlerc.com/D2355

4- Simplify format of profile input file.

   This implements earlier suggestions to simplify the format of the
   sample profile file. The symbol table is not necessary and function
   profiles do not need to know the number of samples in advance.

   Differential Revision: http://llvm-reviews.chandlerc.com/D2419

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@198973 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-10 23:23:51 +00:00
Diego Novillo
0de8cecb84 Propagation of profile samples through the CFG.
This adds a propagation heuristic to convert instruction samples
into branch weights. It implements a similar heuristic to the one
implemented by Dehao Chen on GCC.

The propagation proceeds in 3 phases:

1- Assignment of block weights. All the basic blocks in the function
   are initial assigned the same weight as their most frequently
   executed instruction.

2- Creation of equivalence classes. Since samples may be missing from
   blocks, we can fill in the gaps by setting the weights of all the
   blocks in the same equivalence class to the same weight. To compute
   the concept of equivalence, we use dominance and loop information.
   Two blocks B1 and B2 are in the same equivalence class if B1
   dominates B2, B2 post-dominates B1 and both are in the same loop.

3- Propagation of block weights into edges. This uses a simple
   propagation heuristic. The following rules are applied to every
   block B in the CFG:

   - If B has a single predecessor/successor, then the weight
     of that edge is the weight of the block.

   - If all the edges are known except one, and the weight of the
     block is already known, the weight of the unknown edge will
     be the weight of the block minus the sum of all the known
     edges. If the sum of all the known edges is larger than B's weight,
     we set the unknown edge weight to zero.

   - If there is a self-referential edge, and the weight of the block is
     known, the weight for that edge is set to the weight of the block
     minus the weight of the other incoming edges to that block (if
     known).

Since this propagation is not guaranteed to finalize for every CFG, we
only allow it to proceed for a limited number of iterations (controlled
by -sample-profile-max-propagate-iterations). It currently uses the same
GCC default of 100.

Before propagation starts, the pass builds (for each block) a list of
unique predecessors and successors. This is necessary to handle
identical edges in multiway branches. Since we visit all blocks and all
edges of the CFG, it is cleaner to build these lists once at the start
of the pass.

Finally, the patch fixes the computation of relative line locations.
The profiler emits lines relative to the function header. To discover
it, we traverse the compilation unit looking for the subprogram
corresponding to the function. The line number of that subprogram is the
line where the function begins. That becomes line zero for all the
relative locations.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@198972 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-10 23:23:46 +00:00
Arnold Schwaighofer
ee3f7de62e LoopVectorizer: Handle strided memory accesses by versioning
for (i = 0; i < N; ++i)
   A[i * Stride1] += B[i * Stride2];

We take loops like this and check that the symbolic strides 'Strided1/2' are one
and drop to the scalar loop if they are not.

This is currently disabled by default and hidden behind the flag
'enable-mem-access-versioning'.

radar://13075509

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@198950 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-10 18:20:32 +00:00
Chandler Carruth
560e3955c3 Put the functionality for printing a value to a raw_ostream as an
operand into the Value interface just like the core print method is.
That gives a more conistent organization to the IR printing interfaces
-- they are all attached to the IR objects themselves. Also, update all
the users.

This removes the 'Writer.h' header which contained only a single function
declaration.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@198836 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-09 02:29:41 +00:00
Hao Liu
9e0fd27ce7 Fix a bug about generating undef operand when optimising shuffle vector and insert element in instruction combine.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@198730 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-08 03:06:15 +00:00
Chandler Carruth
bc65a8d518 Move the LLVM IR asm writer header files into the IR directory, as they
are part of the core IR library in order to support dumping and other
basic functionality.

Rename the 'Assembly' include directory to 'AsmParser' to match the
library name and the only functionality left their -- printing has been
in the core IR library for quite some time.

Update all of the #includes to match.

All of this started because I wanted to have the layering in good shape
before I started adding support for printing LLVM IR using the new pass
infrastructure, and commandline support for the new pass infrastructure.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@198688 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-07 12:34:26 +00:00
Chandler Carruth
974a445bd9 Re-sort all of the includes with ./utils/sort_includes.py so that
subsequent changes are easier to review. About to fix some layering
issues, and wanted to separate out the necessary churn.

Also comment and sink the include of "Windows.h" in three .inc files to
match the usage in Memory.inc.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@198685 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-07 11:48:04 +00:00
Andrew Trick
b4e0c9b85d Reapply r198654 "indvars: sink truncates outside the loop."
This doesn't seem to have actually broken anything. It was paranoia
on my part. Trying again now that bots are more stable.

This is a follow up of the r198338 commit that added truncates for
lcssa phi nodes. Sinking the truncates below the phis cleans up the
loop and simplifies subsequent analysis within the indvars pass.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@198678 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-07 06:59:12 +00:00
Andrew Trick
2352abb7c6 Revert "indvars: sink truncates outside the loop."
This reverts commit r198654.

One of the bots reported a SciMark failure.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@198659 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-07 01:50:58 +00:00
Andrew Trick
ced88c5918 indvars: sink truncates outside the loop.
This is a follow up of the r198338 commit that added truncates for
lcssa phi nodes. Sinking the truncates below the phis cleans up the
loop and simplifies subsequent analysis within the indvars pass.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@198654 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-07 01:02:55 +00:00
Andrew Trick
f86063e7ec 80 col. comment.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@198653 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-07 01:02:52 +00:00
Andrew Trick
a55aaf7fe6 Reapply r198478 "Fix PR18361: Invalidate LoopDispositions after LoopSimplify hoists things."
Now with a fix for PR18384: ValueHandleBase::ValueIsDeleted.

We need to invalidate SCEV's loop info when we delete a block, even if no values are hoisted.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@198631 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-06 19:43:14 +00:00
Alp Toker
395f7c2505 Add missed cleanup from r198456
All other uses of this macro in LLVM/clang have been moved to the function
definition so follow suite (and the usage advice) here too for consistency.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@198516 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-04 22:47:48 +00:00
Alp Toker
ccd36c4c54 Revert "Fix PR18361: Invalidate LoopDispositions after LoopSimplify hoists things."
This commit was the source of crasher PR18384:

While deleting: label %for.cond127
An asserting value handle still pointed to this value!
UNREACHABLE executed at llvm/lib/IR/Value.cpp:671!

Reverting to get the builders green, feel free to re-land after fixing up.
(Renato has a handy isolated repro if you need it.)

This reverts commit r198478.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@198503 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-04 17:00:45 +00:00
Andrew Trick
c846d65f99 Fix PR18361: Invalidate LoopDispositions after LoopSimplify hoists things.
getSCEV for an ashr instruction creates an intermediate zext
expression when it truncates its operand.

The operand is initially inside the loop, so the narrow zext
expression has a non-loop-invariant loop disposition.

LoopSimplify then runs on an outer loop, hoists the ashr operand, and
properly invalidate the SCEVs that are mapped to value.

The SCEV expression for the ashr is now an AddRec with the hoisted
value as the now loop-invariant start value.

The LoopDisposition of this wide value was properly invalidated during
LoopSimplify.

However, if we later get the ashr SCEV again, we again try to create
the intermediate zext expression. We get the same SCEV that we did
earlier, and it is still cached because it was never mapped to a
Value. When we try to create a new AddRec we abort because we're using
the old non-loop-invariant LoopDisposition.

I don't have a solution for this other than to clear LoopDisposition
when LoopSimplify hoists things.

I think the long-term strategy should be to perform LoopSimplify on
all loops before computing SCEV and before running any loop opts on
individual loops. It's possible we may want to rerun LoopSimplify on
individual loops, but it should rarely do anything, so rarely require
invalidating SCEV.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@198478 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-04 05:52:49 +00:00
Nico Weber
c3d3f0c696 Add a LLVM_DUMP_METHOD macro.
The motivation is to mark dump methods as used in debug builds so that they can
be called from lldb, but to not do so in release builds so that they can be
dead-stripped.

There's lots of potential follow-up work suggested in the thread
"Should dump methods be LLVM_ATTRIBUTE_USED only in debug builds?" on cfe-dev,
but everyone seems to agreen on this subset.

Macro name chosen by fair coin toss.



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@198456 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-03 22:53:37 +00:00
David Peixotto
dace98d805 Fix loop rerolling pass failure with non-consant loop lower bound
The loop rerolling pass was failing with an assertion failure from a
failed cast on loops like this:

  void foo(int *A, int *B, int m, int n) {
    for (int i = m; i < n; i+=4) {
      A[i+0] = B[i+0] * 4;
      A[i+1] = B[i+1] * 4;
      A[i+2] = B[i+2] * 4;
      A[i+3] = B[i+3] * 4;
    }
  }

The code was casting the SCEV-expanded code for the new
induction variable to a phi-node. When the loop had a non-constant
lower bound, the SCEV expander would end the code expansion with an
add insted of a phi node and the cast would fail.

It looks like the cast to a phi node was only needed to get the
induction variable value coming from the backedge to compute the end
of loop condition. This patch changes the loop reroller to compare
the induction variable to the number of times the backedge is taken
instead of the iteration count of the loop. In other words, we stop
the loop when the current value of the induction variable ==
IterationCount-1. Previously, the comparison was comparing the
induction variable value from the next iteration == IterationCount.

This problem only seems to occur on 32-bit targets. For some reason,
the loop is not rerolled on 64-bit targets.

PR18290


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@198425 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-03 17:20:01 +00:00
Hal Finkel
ac8ba0c0fd Disable compare sinking in CodeGenPrepare when multiple condition registers are available
As noted in the comment above CodeGenPrepare::OptimizeInst, which aggressively
sinks compares to reduce pressure on the condition register(s), for targets
such as PowerPC with multiple condition registers, this may not be the right
thing to do. This adds an HasMultipleConditionRegisters boolean to TLI, and
CodeGenPrepare::OptimizeInst is skipped when HasMultipleConditionRegisters is
true.

This functionality will be used by the PowerPC backend in an upcoming commit.
Especially when the PowerPC backend starts tracking individual condition
register bits as separate allocatable entities (which will happen in this
upcoming commit), this sinking from CodeGenPrepare::OptimizeInst is
significantly suboptimial.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@198354 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-02 21:13:43 +00:00
Andrew Trick
5f8e79e6d2 indvars: cleanup the IV visitor. It does more than gather sext/zext info.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@198353 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-02 21:12:11 +00:00
Matt Arsenault
e13db1008b Delete unread globals through addrspacecast
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@198346 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-02 20:01:43 +00:00
Matt Arsenault
1085cc1262 Fix addrspacecast with metadata globals
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@198345 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-02 19:53:49 +00:00
Andrew Trick
fcbe3d9501 indvars: insert truncate at loop boundary to avoid redundant IVs.
When widening an IV to remove s/zext, we generally try to eliminate
the original narrow IV. However, LCSSA phi nodes outside the loop were
still using the original IV. Clean this up more aggressively to avoid
redundancy in generated code.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@198338 91177308-0d34-0410-b5e6-96231b3b80d8
2014-01-02 19:29:38 +00:00
Nico Weber
15deec7a8d Set LLVM_EXPORTED_SYMBOL_FILE in CMakeLists whose corresponding Makefiles do so.
(unittests/ExecutionEngine/JIT/CMakeLists.txt is still missing for now, since
it handles export files in a strange way: It generates a .exports file from a
.def file instead of the other way round.)


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@198183 91177308-0d34-0410-b5e6-96231b3b80d8
2013-12-29 23:06:49 +00:00
Alexander Potapenko
b09f505445 [ASan] Fix the test for __asan_gen_ globals and actually fix http://llvm.org/bugs/show_bug.cgi?id=17976
by setting the correct linkage (as stated in the bug).


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@198018 91177308-0d34-0410-b5e6-96231b3b80d8
2013-12-25 16:46:27 +00:00
Alexander Potapenko
c58d98777c [ASan] Make sure none of the __asan_gen_ global strings end up in the symbol table, add a test.
This should fix http://llvm.org/bugs/show_bug.cgi?id=17976
Another test checking for the global variables' locations and prefixes on Darwin will be committed separately.



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@198017 91177308-0d34-0410-b5e6-96231b3b80d8
2013-12-25 14:22:15 +00:00
Andrew Trick
c7b0b7dc8f Add support to indvars for optimizing sadd.with.overflow.
Split sadd.with.overflow into add + sadd.with.overflow to allow
analysis and optimization. This should ideally be done after
InstCombine, which can perform code motion (eventually indvars should
run after all canonical instcombines). We want ISEL to recombine the
add and the check, at least on x86.

This is currently under an option for reducing live induction
variables: -liv-reduce. The next step is reducing liveness of IVs that
are live out of the overflow check paths. Once the related
optimizations are fully developed, reviewed and tested, I do expect
this to become default.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@197926 91177308-0d34-0410-b5e6-96231b3b80d8
2013-12-23 23:31:49 +00:00
Richard Sandiford
166acc9489 Fix Scalarizer insertion point when replacing PHIs with insertelements
If the Scalarizer scalarized a vector PHI but could not scalarize
all uses of it, it would insert a series of insertelements to reconstruct
the vector PHI value from the scalar ones.  The problem was that it would
emit these insertelements immediately after the PHI, even if there were
other PHIs after it.


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@197909 91177308-0d34-0410-b5e6-96231b3b80d8
2013-12-23 14:51:56 +00:00
Richard Sandiford
b09beed540 Fix Scalarizer handling of vector GEPs with multiple index operands
The old code only worked for one index operand.  Also handle "inbounds".


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@197908 91177308-0d34-0410-b5e6-96231b3b80d8
2013-12-23 14:45:00 +00:00
Kostya Serebryany
26f1d1cbbc [asan] don't unpoison redzones on function exit in use-after-return mode.
Summary:
Before this change the instrumented code before Ret instructions looked like:
  <Unpoison Frame Redzones>
  if (Frame != OriginalFrame) // I.e. Frame is fake
     <Poison Complete Frame>

Now the instrumented code looks like:
  if (Frame != OriginalFrame) // I.e. Frame is fake
     <Poison Complete Frame>
  else
     <Unpoison Frame Redzones>

Reviewers: eugenis

Reviewed By: eugenis

CC: llvm-commits

Differential Revision: http://llvm-reviews.chandlerc.com/D2458

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@197907 91177308-0d34-0410-b5e6-96231b3b80d8
2013-12-23 14:15:08 +00:00
Kostya Serebryany
1fe014e5fc [asan] produce fewer stores when poisoning stack shadow
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@197904 91177308-0d34-0410-b5e6-96231b3b80d8
2013-12-23 09:24:36 +00:00
Justin Bogner
bcb726dc8a Transforms: Don't create bad weights when eliminating dead cases
If we happen to eliminate every case in a switch that has branch
weights, we currently try to create metadata for the one remaining
branch, triggering an assert. Instead, we need to check that the
metadata we're trying to create is sensible.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@197791 91177308-0d34-0410-b5e6-96231b3b80d8
2013-12-20 08:21:30 +00:00
Kay Tiong Khoo
75c84130c1 Stay classy (and legal) LLVM. Remove links to 3rd party SMT solver whose links may not be permanent.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@197713 91177308-0d34-0410-b5e6-96231b3b80d8
2013-12-19 18:35:54 +00:00
Kay Tiong Khoo
79b37835f9 Improved fix for PR17827 (instcombine of shift/and/compare).
This change fixes the case of arithmetic shift right - do not attempt to fold that case.
This change also relaxes the conditions when attempting to fold the logical shift right and shift left cases.

No additional IR-level test cases included at this time. See http://llvm.org/bugs/show_bug.cgi?id=17827 for proofs that these are correct transformations.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@197705 91177308-0d34-0410-b5e6-96231b3b80d8
2013-12-19 18:07:17 +00:00
Evgeniy Stepanov
de011a5f90 [dfsan] Simplify code after r197677.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@197679 91177308-0d34-0410-b5e6-96231b3b80d8
2013-12-19 14:37:03 +00:00
Evgeniy Stepanov
a7eb2b83ba Add an explicit insert point argument to SplitBlockAndInsertIfThen.
Currently SplitBlockAndInsertIfThen requires that branch condition is an
Instruction itself, which is very inconvenient, because it is sometimes an
Operator, or even a Constant.


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@197677 91177308-0d34-0410-b5e6-96231b3b80d8
2013-12-19 13:29:56 +00:00
Arnold Schwaighofer
83196a9fcb LoopVectorizer: Don't if-convert constant expressions that can trap
A phi node operand or an instruction operand could be a constant expression that
can trap (division). Check that we don't vectorize such cases.

PR16729
radar://15653590

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@197449 91177308-0d34-0410-b5e6-96231b3b80d8
2013-12-17 01:11:01 +00:00
Yi Jiang
4faf2ba0c7 Enable double to float shrinking optimizations for binary functions like 'fmin/fmax'. Fix radar:15283121
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@197434 91177308-0d34-0410-b5e6-96231b3b80d8
2013-12-16 22:42:40 +00:00
Hal Finkel
1e3d96622a Fix a use-after-free error in GlobalOpt CleanupConstantGlobalUsers
GlobalOpt's CleanupConstantGlobalUsers function uses a worklist array to manage
constant users to be visited. The pointers in this array need to be weak
handles because when we delete a constant array, we may also be holding a
pointer to one of its elements (or an element of one of its elements if we're
dealing with an array of arrays) in the worklist.

Fixes PR17347.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@197178 91177308-0d34-0410-b5e6-96231b3b80d8
2013-12-12 20:45:24 +00:00
Hal Finkel
8cd46c06f2 Initialize the barrier pass llvm::initializeIPO
The barrier pass is a temporary hack, and should go away soon. Nevertheless, if
we don't initialize it, then opt will not understand -barrier, and this will
break bugpoint (because when it dumps the passes from the default pass manager
-barrier will be there).

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@197177 91177308-0d34-0410-b5e6-96231b3b80d8
2013-12-12 20:45:08 +00:00
Yi Jiang
41a3007571 Resubmit r196544: Apply transformation on OS X 10.9+ and iOS 7.0+: pow(10, x) ―> __exp10(x)
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@197109 91177308-0d34-0410-b5e6-96231b3b80d8
2013-12-12 01:55:04 +00:00