Commit Graph

111 Commits

Author SHA1 Message Date
Nadav Rotem
7557e521e5 LoopVectorizer: Calculate the number of pointers to disambiguate at runtime based on the numbers of reads and writes.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@180593 91177308-0d34-0410-b5e6-96231b3b80d8
2013-04-26 05:08:59 +00:00
Nadav Rotem
975b1ddf60 LoopVectorizer: No need to generate pointer disambiguation checks between readonly pointers.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@180570 91177308-0d34-0410-b5e6-96231b3b80d8
2013-04-25 19:55:03 +00:00
Arnold Schwaighofer
a4b8b4ccc9 LoopVectorize: Scalarize padded types
This patch disables memory-instruction vectorization for types that need padding
bytes, e.g., x86_fp80 has 10 bytes store size with 6 bytes padding in darwin on
x86_64. Because the load/store vectorization is performed by the bit casting to
a packed vector, which has incompatible memory layout due to the lack of padding
bytes, the present vectorizer produces inconsistent result for memory
instructions of those types.
This patch checks an equality of the AllocSize of a scalar type and allocated
size for each vector element, to ensure that there is no padding bytes and the
array can be read/written using vector operations.

Patch by Daisuke Takahashi!

Fixes PR15758.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@180196 91177308-0d34-0410-b5e6-96231b3b80d8
2013-04-24 16:16:01 +00:00
Arnold Schwaighofer
b03ad17536 LoopVectorizer: Bail out if we don't have datalayout we need it
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@180195 91177308-0d34-0410-b5e6-96231b3b80d8
2013-04-24 16:15:58 +00:00
Nadav Rotem
a7d9a6ee63 LoopVectorizer: Fix 15830. When scalarizing and unrolling stores make sure that the order in which the elements are scalarized is the same as the original order.
This fixes a miscompilation in FreeBSD's regex library.



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@180121 91177308-0d34-0410-b5e6-96231b3b80d8
2013-04-23 17:12:42 +00:00
Pekka Jaaskelainen
2e59a125fc Call the potentially costly isAnnotatedParallel() only once.
Made the uniform write test's checks a bit stricter.



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@180119 91177308-0d34-0410-b5e6-96231b3b80d8
2013-04-23 16:44:43 +00:00
Pekka Jaaskelainen
a8958769ea Refuse to (even try to) vectorize loops which have uniform writes,
even if erroneously annotated with the parallel loop metadata.

Fixes Bug 15794: 
"Loop Vectorizer: Crashes with the use of llvm.loop.parallel metadata"



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@180081 91177308-0d34-0410-b5e6-96231b3b80d8
2013-04-23 08:08:51 +00:00
Arnold Schwaighofer
a3fb330d05 LoopVectorizer: Recognize min/max reductions
A min/max operation is represented by a select(cmp(lt/le/gt/ge, X, Y), X, Y)
sequence in LLVM. If we see such a sequence we can treat it just as any other
commutative binary instruction and reduce it.

This appears to help bzip2 by about 1.5% on an imac12,2.

radar://12960601

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@179773 91177308-0d34-0410-b5e6-96231b3b80d8
2013-04-18 17:22:34 +00:00
Benjamin Kramer
403fc14370 LoopVectorize: Use a set to avoid longer cycles in the reduction chain too.
Fixes PR15748.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@179757 91177308-0d34-0410-b5e6-96231b3b80d8
2013-04-18 14:29:13 +00:00
Arnold Schwaighofer
08a0e8f8db LoopVectorizer: integer division is not a reduction operation
Don't classify idiv/udiv as a reduction operation. Integer division is lossy.
For example : (1 / 2) * 4 != 4/2.

Example:

int a[] = { 2, 5, 2, 2}
int x = 80;

for()
  x /= a[i];

Scalar:
  x /= 2 // = 40
  x /= 5 // = 8
  x /= 2 // = 4
  x /= 2 // = 2

Vectorized:

 <80, 1> / <2,5> //= <40,0>
 <40, 0> / <2,2> //= <20,0>

 20*0 = 0

radar://13640654

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@179381 91177308-0d34-0410-b5e6-96231b3b80d8
2013-04-12 15:15:19 +00:00
Arnold Schwaighofer
ac2cc0170f LoopVectorizer: Pass OperandValueKind information to the cost model
Pass down the fact that an operand is going to be a vector of constants.

This should bring the performance of MultiSource/Benchmarks/PAQ8p/paq8p on x86
back. It had degraded to scalar performance due to my pervious shift cost change
that made all shifts expensive on x86.

radar://13576547

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@178809 91177308-0d34-0410-b5e6-96231b3b80d8
2013-04-04 23:26:27 +00:00
Benjamin Kramer
13497b3aa7 X86TTI: Add accurate costs for itofp operations, based on the actual instruction counts.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@178459 91177308-0d34-0410-b5e6-96231b3b80d8
2013-04-01 10:23:49 +00:00
Arnold Schwaighofer
c184a5f4ca LoopVectorizer: Insert some white space to make test case more readable
Also remove some unneeded function attributes.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@177114 91177308-0d34-0410-b5e6-96231b3b80d8
2013-03-14 21:31:09 +00:00
Arnold Schwaighofer
e2188d9c43 Add missing asserts flag to test - it uses debug flags
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@177102 91177308-0d34-0410-b5e6-96231b3b80d8
2013-03-14 19:01:58 +00:00
Arnold Schwaighofer
d517da33b7 LoopVectorize: Invert case when we use a vector cmp value to query select cost
We generate a select with a vectorized condition argument when the condition is
NOT loop invariant. Not the other way around.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@177098 91177308-0d34-0410-b5e6-96231b3b80d8
2013-03-14 18:54:36 +00:00
Benjamin Kramer
1cb47b9afe Test case hygiene.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@176772 91177308-0d34-0410-b5e6-96231b3b80d8
2013-03-09 18:25:40 +00:00
Arnold Schwaighofer
56ee544a3a LoopVectorizer: Ignore dbg.value instructions
We want vectorization to happen at -g. Ignore calls to the dbg.value intrinsic
and don't transfer them to the vectorized code.

radar://13378964

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@176768 91177308-0d34-0410-b5e6-96231b3b80d8
2013-03-09 15:56:34 +00:00
Benjamin Kramer
35a4a0ca51 Force cpu in test.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@176702 91177308-0d34-0410-b5e6-96231b3b80d8
2013-03-08 17:01:18 +00:00
Benjamin Kramer
f22d9cfa6d Insert the reduction start value into the first bypass block to preserve domination.
Fixes PR15344.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@176701 91177308-0d34-0410-b5e6-96231b3b80d8
2013-03-08 16:58:37 +00:00
Arnold Schwaighofer
5f0d9dbdf4 X86 cost model: Adjust cost for custom lowered vector multiplies
This matters for example in following matrix multiply:

int **mmult(int rows, int cols, int **m1, int **m2, int **m3) {
  int i, j, k, val;
  for (i=0; i<rows; i++) {
    for (j=0; j<cols; j++) {
      val = 0;
      for (k=0; k<cols; k++) {
        val += m1[i][k] * m2[k][j];
      }
      m3[i][j] = val;
    }
  }
  return(m3);
}

Taken from the test-suite benchmark Shootout.

We estimate the cost of the multiply to be 2 while we generate 9 instructions
for it and end up being quite a bit slower than the scalar version (48% on my
machine).

Also, properly differentiate between avx1 and avx2. On avx-1 we still split the
vector into 2 128bits and handle the subvector muls like above with 9
instructions.
Only on avx-2 will we have a cost of 9 for v4i64.

I changed the test case in test/Transforms/LoopVectorize/X86/avx1.ll to use an
add instead of a mul because with a mul we now no longer vectorize. I did
verify that the mul would be indeed more expensive when vectorized with 3
kernels:

for (i ...)
   r += a[i] * 3;
for (i ...)
  m1[i] = m1[i] * 3; // This matches the test case in avx1.ll
and a matrix multiply.

In each case the vectorized version was considerably slower.

radar://13304919

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@176403 91177308-0d34-0410-b5e6-96231b3b80d8
2013-03-02 04:02:52 +00:00
Nadav Rotem
5290baacb8 PR14448 - prevent the loop vectorizer from vectorizing the same loop twice.
The LoopVectorizer often runs multiple times on the same function due to inlining.
When this happens the loop vectorizer often vectorizes the same loops multiple times, increasing code size and adding unneeded branches.
With this patch, the vectorizer during vectorization puts metadata on scalar loops and marks them as 'already vectorized' so that it knows to ignore them when it sees them a second time.

PR14448.



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@176399 91177308-0d34-0410-b5e6-96231b3b80d8
2013-03-02 01:33:49 +00:00
Benjamin Kramer
5d79bb8770 LoopVectorize: Don't hang forever if a PHI only has skipped PHI uses.
Fixes PR15384.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@176366 91177308-0d34-0410-b5e6-96231b3b80d8
2013-03-01 19:07:31 +00:00
Benjamin Kramer
2fe71f8f7e LoopVectorize: Vectorize math builtin calls.
This properly asks TargetLibraryInfo if a call is available and if it is, it
can be translated into the corresponding LLVM builtin. We don't vectorize sqrt()
yet because I'm not sure about the semantics for negative numbers. The other
intrinsic should be exact equivalents to the libm functions.

Differential Revision: http://llvm-reviews.chandlerc.com/D465

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@176188 91177308-0d34-0410-b5e6-96231b3b80d8
2013-02-27 15:24:19 +00:00
Renato Golin
4ff470e608 Some more tests for the global structure vectorizer
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@175964 91177308-0d34-0410-b5e6-96231b3b80d8
2013-02-23 12:48:30 +00:00
Renato Golin
fd4af7d334 More tests to global struct vectorizer
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@175898 91177308-0d34-0410-b5e6-96231b3b80d8
2013-02-22 16:18:31 +00:00
Renato Golin
e18bce5317 Allow GlobalValues to vectorize with AliasAnalysis
Storing the load/store instructions with the values
and inspect them using Alias Analysis to make sure
they don't alias, since the GEP pointer operand doesn't
take the offset into account.

Trying hard to not add any extra cost to loads and stores
that don't overlap on global values, AA is *only* calculated
if all of the previous attempts failed.

Using biggest vector register size as the stride for the
vectorization access, as we're being conservative and
the cost model (which calculates the real vectorization
factor) is only run after the legalization phase.

We might re-think this relationship in the future, but
for now, I'd rather be safe than sorry.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@175818 91177308-0d34-0410-b5e6-96231b3b80d8
2013-02-21 22:39:03 +00:00
Pekka Jaaskelainen
b3dedc2ef2 Forgot to 'svn add' the LoopVectorizer tests for the new parallel loop metadata, sorry.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@175311 91177308-0d34-0410-b5e6-96231b3b80d8
2013-02-15 21:50:19 +00:00
NAKAMURA Takumi
f5b39cd8de Formatting.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@174380 91177308-0d34-0410-b5e6-96231b3b80d8
2013-02-05 15:32:16 +00:00
NAKAMURA Takumi
eec44ce834 llvm/test/Transforms/LoopVectorize/X86/vector_ptr_load_store.ll: "-debug" requires +Asserts.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@174379 91177308-0d34-0410-b5e6-96231b3b80d8
2013-02-05 15:32:10 +00:00
Arnold Schwaighofer
935645b765 Loop Vectorizer: Handle pointer stores/loads in getWidestType()
In the loop vectorizer cost model, we used to ignore stores/loads of a pointer
type when computing the widest type within a loop. This meant that if we had
only stores/loads of pointers in a loop we would return a widest type of 8bits
(instead of 32 or 64 bit) and therefore a vector factor that was too big.

Now, if we see a consecutive store/load of pointers we use the size of a pointer
(from data layout).

This problem occured in SingleSource/Benchmarks/Shootout-C++/hash.cpp (reduced
test case is the first test in vector_ptr_load_store.ll).

radar://13139343

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@174377 91177308-0d34-0410-b5e6-96231b3b80d8
2013-02-05 15:08:02 +00:00
Pekka Jaaskelainen
d72b4d321e Made the min-trip-count-switch test X86-specific to avoid
breakage with builds without X86-support.


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@174052 91177308-0d34-0410-b5e6-96231b3b80d8
2013-01-31 10:33:22 +00:00
Renato Golin
0261cea689 Adding simple cast cost to ARM
Changing ARMBaseTargetMachine to return ARMTargetLowering intead of
the generic one (similar to x86 code).

Tests showing which instructions were added to cast when necessary
or cost zero when not. Downcast to 16 bits are not lowered in NEON,
so costs are not there yet.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@173849 91177308-0d34-0410-b5e6-96231b3b80d8
2013-01-29 23:31:38 +00:00
Pekka Jaaskelainen
d855049576 LoopVectorize: convert TinyTripCountVectorThreshold constant
to a command line switch.



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@173837 91177308-0d34-0410-b5e6-96231b3b80d8
2013-01-29 21:42:08 +00:00
Nadav Rotem
f148c66ce4 Add support for reverse pointer induction variables. These are loops that contain pointers that count backwards.
For example, this is the hot loop in BZIP:

  do {
    m = *--p;
    *p = ( ... );
  } while (--n);



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@173219 91177308-0d34-0410-b5e6-96231b3b80d8
2013-01-23 01:35:00 +00:00
Nadav Rotem
0bbbc52dc8 LoopVectorizer: Implement a new heuristics for selecting the unroll factor.
We ignore the cpu frontend and focus on pipeline utilization. We do this because we
don't have a good way to estimate the loop body size at the IR level.



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@172964 91177308-0d34-0410-b5e6-96231b3b80d8
2013-01-20 05:24:29 +00:00
Nadav Rotem
bcdabadaf4 Change the cpu type in the test.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@172963 91177308-0d34-0410-b5e6-96231b3b80d8
2013-01-20 05:20:56 +00:00
Benjamin Kramer
1af132dcf3 LoopVectorizer: Emit memory checks into their own basic block.
This separates the check for "too few elements to run the vector loop" from the
"memory overlap" check, giving a lot nicer code and allowing to skip the memory
checks when we're not going to execute the vector code anyways. We still leave
the decision of whether to emit the memory checks as branches or setccs, but it
seems to be doing a good job. If ugly code pops up we may want to emit them as
separate blocks too. Small speedup on MultiSource/Benchmarks/MallocBench/espresso.

Most of this is legwork to allow multiple bypass blocks while updating PHIs,
dominators and loop info.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@172902 91177308-0d34-0410-b5e6-96231b3b80d8
2013-01-19 13:57:58 +00:00
Benjamin Kramer
c759dd5f83 Move test that depends on the x86 target into a target-specific directory.
Should fix the arm buildbot (which only builds the arm target).

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@172611 91177308-0d34-0410-b5e6-96231b3b80d8
2013-01-16 13:25:56 +00:00
Nadav Rotem
b6db95f42b Fix PR14547. Handle induction variables of small sizes smaller than i32 (i8 and i16).
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@172348 91177308-0d34-0410-b5e6-96231b3b80d8
2013-01-13 07:56:29 +00:00
Nadav Rotem
3e40d927a7 ARM Cost Model: Modify the target independent cost model to ask
the target if it supports the different CAST types. We didn't do this
on X86 because of the different register sizes and types, but on ARM
this makes sense.



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@172245 91177308-0d34-0410-b5e6-96231b3b80d8
2013-01-11 19:54:13 +00:00
Nadav Rotem
a675c74208 ARM Cost Model: We need to detect the max bitwidth of types in the loop in order to select the max vectorization factor.
We don't have a detailed analysis on which values are vectorized and which stay scalars in the vectorized loop so we use
another method. We look at reduction variables, loads and stores, which are the only ways to get information in and out
of loop iterations. If the data types are extended and truncated then the cost model will catch the cost of the vector
zext/sext/trunc operations.



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@172178 91177308-0d34-0410-b5e6-96231b3b80d8
2013-01-11 07:11:59 +00:00
Nadav Rotem
c560bf638b LoopVectorizer: Fix a bug in the vectorization of BinaryOperators. The BinaryOperator can be folded to an Undef, and we don't want to set NSW flags to undef vals.
PR14878



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@172079 91177308-0d34-0410-b5e6-96231b3b80d8
2013-01-10 17:34:39 +00:00
Nadav Rotem
14925e6b88 ARM Cost model: Use the size of vector registers and widest vectorizable instruction to determine the max vectorization factor.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@172010 91177308-0d34-0410-b5e6-96231b3b80d8
2013-01-09 22:29:00 +00:00
Nadav Rotem
df8c22a104 ARM Cost Model: Add a basic vectorization unrolling test.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@171931 91177308-0d34-0410-b5e6-96231b3b80d8
2013-01-09 01:29:07 +00:00
Nadav Rotem
3c90b3d5fd Remove the -licm pass from the loop vectorizer test because the loop vectorizer does it now.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@171930 91177308-0d34-0410-b5e6-96231b3b80d8
2013-01-09 01:20:59 +00:00
Nadav Rotem
83be7b0dd3 Cost Model: Move the 'max unroll factor' variable to the TTI and add initial Cost Model support on ARM.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@171928 91177308-0d34-0410-b5e6-96231b3b80d8
2013-01-09 01:15:42 +00:00
Nadav Rotem
111e5fe7e0 LoopVectorizer: Add support for floating point reductions
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@171812 91177308-0d34-0410-b5e6-96231b3b80d8
2013-01-07 23:13:00 +00:00
Nadav Rotem
9a6c6a3736 LoopVectorizer: When we vectorizer and widen loops we process many elements at once. This is a good thing, except for
small loops. On small loops post-loop that handles scalars (and runs slower) can take more time to execute than the
rest of the loop. This patch disables widening of loops with a small static trip count.



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@171798 91177308-0d34-0410-b5e6-96231b3b80d8
2013-01-07 21:54:51 +00:00
Nadav Rotem
024328ea49 Fix a typo. Remove the duplicated test.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@171584 91177308-0d34-0410-b5e6-96231b3b80d8
2013-01-05 01:17:46 +00:00
Nadav Rotem
d5b92c3891 iLoopVectorize: Non commutative operators can be used as reduction variables as long as the reduction chain is used in the LHS.
PR14803.



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@171583 91177308-0d34-0410-b5e6-96231b3b80d8
2013-01-05 01:15:47 +00:00