Commit Graph

694 Commits

Author SHA1 Message Date
Matt Arsenault
6fc71a0cfc R600/SI: Make sure double vector fmul is tested
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@217787 91177308-0d34-0410-b5e6-96231b3b80d8
2014-09-15 17:04:54 +00:00
Matt Arsenault
e626ee51b6 R600/SI: Add some mubuf testcases.
I noticed some odd looking cases where addr64 wasn't set
when storing to a pointer in an SGPR. This seems to be intentional,
and partially tested already.

The documentation seems to describe addr64 in terms of which registers
addressing modifiers come from, but I would expect to always need
addr64 when using 64-bit pointers. If no offset is applied,
it makes sense to not need to worry about doing a 64-bit add
for the final address. A small immediate offset can be applied,
so is it OK to not have addr64 set if a carry is necessary when adding
the base pointer in the resource to the offset?

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@217785 91177308-0d34-0410-b5e6-96231b3b80d8
2014-09-15 16:48:01 +00:00
Matt Arsenault
d189a0407d R600/SI: Add preliminary support for flat address space
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@217777 91177308-0d34-0410-b5e6-96231b3b80d8
2014-09-15 15:41:53 +00:00
Matt Arsenault
a0ba49844c R600/SI: Fix broken check lines
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@217736 91177308-0d34-0410-b5e6-96231b3b80d8
2014-09-14 18:32:05 +00:00
Matt Arsenault
86ffcddf42 R600/SI: Fix off by 1 error in used register count
The register numbers start at 0, so if only 1 register
was used, this was reported as 0.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@217636 91177308-0d34-0410-b5e6-96231b3b80d8
2014-09-11 22:51:37 +00:00
Matt Arsenault
31b1bdbd95 Add DAG combine for shl + add of constants.
Do
 (shl (add x, c1), c2) -> (add (shl x, c2), c1 << c2)

This is already done for multiplies, but since multiplies
by powers of two are turned into shifts, we also need
to handle it here.

This might want checks for isLegalAddImmediate to avoid
transforming an add of a legal immediate with one that isn't.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@217610 91177308-0d34-0410-b5e6-96231b3b80d8
2014-09-11 17:34:19 +00:00
Aaron Watry
1ff44854f6 R600: Test local atomics for evergreen
Now that the operations are all implemented, we can test this sub-arch here.

Signed-off-by: Aaron Watry <awatry@gmail.com>
Reviewed-by: Matt Arsenault <matthew.arsenault@amd.com>

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@217595 91177308-0d34-0410-b5e6-96231b3b80d8
2014-09-11 15:02:52 +00:00
Matt Arsenault
5ee5d45e7e R600/SI: Fix losing chain when fixing reg class of loads.
The lost chain resulting in earlier side effecting nodes
being deleted.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@217561 91177308-0d34-0410-b5e6-96231b3b80d8
2014-09-10 23:26:19 +00:00
Matt Arsenault
257e85e7c2 R600: Custom lower frem
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@217553 91177308-0d34-0410-b5e6-96231b3b80d8
2014-09-10 21:44:27 +00:00
Matt Arsenault
ef4bb30475 R600/SI: Replace LDS atomics with no return versions
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@217379 91177308-0d34-0410-b5e6-96231b3b80d8
2014-09-08 15:07:31 +00:00
Matt Arsenault
6a712e709d R600/SI: Relax a few tests to help enable scheduler
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@217320 91177308-0d34-0410-b5e6-96231b3b80d8
2014-09-06 20:44:41 +00:00
Matt Arsenault
360ed46f68 R600/SI: Fix broken check lines.
Fix missing check, and hardcoded register numbers.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@217318 91177308-0d34-0410-b5e6-96231b3b80d8
2014-09-06 20:37:56 +00:00
Matt Arsenault
89a7e3ec3e R600/SI: Use same complex patterns for DS atomics
This fixes hitting the same negative base offset problem
that was already fixed for regular loads and stores.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@217256 91177308-0d34-0410-b5e6-96231b3b80d8
2014-09-05 16:24:58 +00:00
Jan Vesely
286f644bce R600: Fix FROUND
round halfway cases away from zero

Signed-off-by: Jan Vesely <jan.vesely@rutgers.edu>
Reviewed-by: Tom Stellard <tom@stellard.net>

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@217250 91177308-0d34-0410-b5e6-96231b3b80d8
2014-09-05 14:26:54 +00:00
Tom Stellard
7cda2d0666 R600/SI: Use S_ADD_U32 and S_SUB_U32 for low half of 64-bit operations
https://bugs.freedesktop.org/show_bug.cgi?id=83416

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@217248 91177308-0d34-0410-b5e6-96231b3b80d8
2014-09-05 14:07:59 +00:00
Matt Arsenault
c9cc488dfe R600/SI: Try to keep i32 mul on SALU
Also fix bug this exposed where when legalizing an immediate
operand, a v_mov_b32 would be created with a VSrc dest register.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@217108 91177308-0d34-0410-b5e6-96231b3b80d8
2014-09-03 23:24:35 +00:00
Tom Stellard
ce4caf146f R600/SI: Add a pattern for i64 and in a branch
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@217041 91177308-0d34-0410-b5e6-96231b3b80d8
2014-09-03 15:22:41 +00:00
Matt Arsenault
2aab51a118 R600/SI: Relax some ordering in tests.
This will help with enabling misched

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@216971 91177308-0d34-0410-b5e6-96231b3b80d8
2014-09-02 21:45:50 +00:00
Matt Arsenault
9c21df64a4 R600/SI: Fix hardcoded register numbers in test
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@216944 91177308-0d34-0410-b5e6-96231b3b80d8
2014-09-02 20:43:07 +00:00
Matt Arsenault
f471c483e6 R600/SI: Add failing testcase.
This is broken when 64-bit add is only partially
moved to the VALU.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@216933 91177308-0d34-0410-b5e6-96231b3b80d8
2014-09-02 19:12:31 +00:00
Matt Arsenault
f7a3c7e705 Fix interference caused by fmul 2, x -> fadd x, x
If an fmul was introduced by lowering, it wouldn't be folded
into a multiply by a constant since the earlier combine would
have replaced the fmul with the fadd.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@216932 91177308-0d34-0410-b5e6-96231b3b80d8
2014-09-02 19:02:53 +00:00
Matt Arsenault
f4d57e7874 R600/SI: Use mad for fsub + fmul
We can use a negate source modifier to match
this for fsub.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@216735 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-29 16:01:14 +00:00
Tom Stellard
f50f927d65 R600/SI: Use READ2/WRITE2 instructions for 64-bit mem ops with 32-bit alignment
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@216279 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-22 18:49:35 +00:00
Tom Stellard
ec4cb3346d R600/SI: Use a ComplexPattern for DS loads and stores
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@216278 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-22 18:49:33 +00:00
Tom Stellard
fdbf61d00d R600/SI: Teach moveToVALU how to handle more S_LOAD_* instructions
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@216220 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-21 20:41:00 +00:00
Tom Stellard
5f52739370 R600/SI: Make sure SCRATCH_WAVE_OFFSET is added as Live-In to the function
This fixes a crash in an ocl conformance test.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@216219 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-21 20:40:58 +00:00
Matt Arsenault
c86e55eb6e R600/SI: Move all fabs / fneg handling to patterns
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@215749 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-15 18:42:22 +00:00
Matt Arsenault
0498d07255 R600/SI: Use source modifiers for f64 fneg
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@215748 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-15 18:42:18 +00:00
Matt Arsenault
c882fc78fe R600/SI: Use source modifier for f64 fabs
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@215747 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-15 18:42:15 +00:00
Matt Arsenault
34ef4cd65b R600/SI: Fix offset folding in some cases with shifted pointers.
Ordinarily (shl (add x, c1), c2) -> (add (shl x, c2), c1 << c2)
is only done if the add has one use. If the resulting constant
add can be folded into an addressing mode, force this to happen
for the pointer operand.

This ends up happening a lot because of how LDS objects are allocated.
Since the globals are allocated next to each other, acessing the first
element of the second object is directly indexed by a shifted pointer.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@215739 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-15 17:49:05 +00:00
Matt Arsenault
5bc44c7603 R600/SI: Add intrinsic for ldexp
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@215734 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-15 17:30:25 +00:00
Matt Arsenault
ed76ca720b R600/SI: Implement isLegalAddressingMode
The default assumes that a 16-bit signed offset is used.
LDS instruction use a 16-bit unsigned offset, so it wasn't
being used in some cases where it was assumed a negative offset
could be used.

More should be done here, but first isLegalAddressingMode needs
to gain an addressing mode argument. For now, copy most of the rest
of the default implementation with the immediate offset change.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@215732 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-15 17:17:07 +00:00
Matt Arsenault
bd949eea85 R600: Correctly set the src value offset for scalarized kernel args
This for some reason fixes v1i64 kernel arguments on pre-SI. This
currently breaks some other cases in the kernel-args.ll test for R600,
but I'm not particularly confident in the new output. VTX_READ_* are not
used for some of the scalarized cases, and the code reading from the
constant buffer doesn't make much sense to me.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@215564 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-13 18:14:11 +00:00
Hal Finkel
695e914c03 Fix classof for ISD::INTRINSIC_W_CHAIN and INTRINSIC_VOID
Unfortunately, our use of the SDNode class hierarchy for INTRINSIC_W_CHAIN and
INTRINSIC_VOID nodes is somewhat broken right now. These nodes sometimes are
used for memory intrinsics (those with MachineMemOperands), and sometimes not.
When not, the nodes are not created as instances of MemIntrinsicSDNode, but
rather created as some other subclass of SDNode using DAG::getNode. When they
are memory intrinsics, they are created using DAG::getMemIntrinsicNode as
instances of MemIntrinsicSDNode. MemIntrinsicSDNode is a subclass of
MemSDNode, but prior to r214452, we had a non-self-consistent setup whereby
MemIntrinsicSDNode::classof on INTRINSIC_W_CHAIN and INTRINSIC_VOID would
return true but MemSDNode::classof on INTRINSIC_W_CHAIN and INTRINSIC_VOID
would return false. In r214452, MemSDNode::classof was changed to return true
for INTRINSIC_W_CHAIN and INTRINSIC_VOID, which is now self-consistent. The
problem is that neither the pre-r214452 logic and the post-r214452 logic are
really right. The truth is that not all INTRINSIC_W_CHAIN and INTRINSIC_VOID
nodes are instances of MemIntrinsicSDNode (or MemSDNode for that matter), and
the return value from classof needs to reflect that. This was broken before
r214452 (because MemIntrinsicSDNode::classof always returned true), and was
broken afterward (because MemSDNode::classof also always returned true), and
will now be correct.

The minimal solution is to grab one of the SubclassData bits (there is one left
for MemIntrinsicSDNode nodes) and use it to store whether or not a particular
INTRINSIC_W_CHAIN or INTRINSIC_VOID is really an instance of
MemIntrinsicSDNode or not. Doing this allows both MemIntrinsicSDNode::classof
and MemSDNode::classof to return the correct answer for the underlying object
for both the memory-intrinsic and non-memory-intrinsic cases.

This fixes the problem that r214452 created in the SelectionDAGDumper (thanks
to Matt Arsenault for pointing it out).

Because PowerPC does not implement getTgtMemIntrinsic, this change breaks
test/CodeGen/PowerPC/unal-altivec-wint.ll. I've XFAILed it for now, and will
fix it in a follow-up commit.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@215511 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-13 01:15:37 +00:00
Jan Vesely
3c57820bbb R600: Use optimized 24bit path in udivrem
v2: drop enum keyword
    use correct extension mode
    don't bother computing the sign in unsinged case

Signed-off-by: Jan Vesely <jan.vesely@rutgers.edu>

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@215462 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-12 17:31:20 +00:00
Jan Vesely
b40562c0ec R600: Use i24 optimized path for SREM
v2: add tests
    rename LowerSDIV24 to LowerSDIVREM24
    handle the rem part in this function

Signed-off-by: Jan Vesely <jan.vesely@rutgers.edu>

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@215460 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-12 17:31:17 +00:00
Tom Stellard
13f4476c55 R600/SI: Add a ComplexPattern for selecting MUBUF _OFFSET variant
This saves us from having to copy a 64-bit 0 value into VGPRs for
BUFFER_* instruction which only have a 12-bit immediate offset.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@215399 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-11 22:18:17 +00:00
Tom Stellard
68e9ebbe44 R600/SI: Add check for low 32 bits of encoding to mubuf tests
There are no variable values like registers encoded in the low 32 bits of MUBUF
instructions, so it is relatively easy to check these bits, and it will
help prevent us from introducing encoding bugs.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@215397 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-11 22:18:11 +00:00
Tom Stellard
728d0e4218 R600/SI: Clear lds bit on MUBUF instructions used for private stores
This bit was left uninitialized, which was causing some random failures
of piglit tests.

NOTE: This is a candidate for the 3.5 branch.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@215396 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-11 22:18:09 +00:00
Tom Stellard
0df264a0fd R600/SI: Fix broken test
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@215395 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-11 22:18:05 +00:00
Tom Stellard
4e8a136db8 R600/SI: Custom lower CONCAT_VECTORS
This will lower them using register copies rather than loads and stores
to the stack.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@215270 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-09 01:06:56 +00:00
Tom Stellard
f1ba587963 R600/SI: Update concat_vectors.ll to check for scratch usage
These tests were using SI-NOT: MOVREL to make sure concat vectors
weren't being lowered to stack loads and stores, but we are using
scratch buffers for the stack now instead of registers, so we need
to add an additional SI-NOT check for scratch buffers.

With this change I was able to uncover one broken test which will
be fixed in a future commit.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@215269 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-09 01:06:53 +00:00
Matt Arsenault
60178b180f R600: Cleanup fadd and fsub tests
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@214991 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-06 20:27:55 +00:00
Matt Arsenault
85dc7da6f3 R600: Increase nearby load scheduling threshold.
This partially fixes weird looking load scheduling
in memcpy test. The load clustering doesn't seem
particularly smart, but this method seems to be partially
deprecated so it might not be worth trying to fix.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@214943 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-06 00:29:49 +00:00
Matt Arsenault
c9c70b1651 R600/SI: Implement areLoadsFromSameBasePtr
This currently has a noticable effect on the kernel argument loads.
LDS and global loads are more problematic, I think because of how copies
are currently inserted to ensure that the address is a VGPR.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@214942 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-06 00:29:43 +00:00
Tom Stellard
94dfb8818d R600/SI: Update MUBUF assembly string to match AMD proprietary compiler
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@214866 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-05 14:48:12 +00:00
Tom Stellard
9a7e35aecc R600/SI: Avoid generating REGISTER_LOAD instructions.
SI doesn't use REGISTER_LOAD anymore, but it was still hitting this code
path for 8-bit and 16-bit private loads.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@214865 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-05 14:40:52 +00:00
Matt Arsenault
fc65cf649c R600/SI: Fix extra whitespace in asm str
This slipped in in r214467, so something like

V_MOV_B32_e32  v0, ... is now printed with 2 spaces
between the instruction name and first operand.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@214660 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-03 05:27:14 +00:00
Matt Arsenault
507c5af818 R600: Cleanup fneg tests
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@214612 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-02 02:26:51 +00:00
Tom Stellard
b2df20d015 Revert "R600: Move code for generating REGISTER_LOAD into R600ISelLowering.cpp"
This reverts commit r214566.

I did not mean to commit this yet.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@214572 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-01 21:55:50 +00:00
Tom Stellard
7f288b455e R600: Move code for generating REGISTER_LOAD into R600ISelLowering.cpp
SI doesn't use REGISTER_LOAD anymore, but it was still hitting this code
path for 8-bit and 16-bit private loads.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@214566 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-01 21:50:47 +00:00
Matt Arsenault
f8230857c0 R600: Cleanup test
Remove -CHECKs, use multiple prefixes, name values,
also test the @llvm.fabs version

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@214525 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-01 17:00:29 +00:00
Tom Stellard
17c8fefc9f R600/SI: Do abs/neg folding with ComplexPatterns
Abs/neg folding has moved out of foldOperands and into the instruction
selection phase using complex patterns.  As a consequence of this
change, we now prefer to select the 64-bit encoding for most
instructions and the modifier operands have been dropped from
integer VOP3 instructions.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@214467 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-01 00:32:39 +00:00
Tom Stellard
635fb9fb59 R600/SI: Fold immediates when shrinking instructions
This will prevent us from using extra MOV instructions once we prefer
selecting 64-bit instructions.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@214464 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-01 00:32:33 +00:00
Tom Stellard
e5fc4208eb R600/SI: Fix incorrect commute operation in shrink instructions pass
We were commuting the instruction by still shrinking it using the
original opcode.

NOTE: This is a candidate for the 3.5 branch.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@214463 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-01 00:32:28 +00:00
Jan Vesely
22c99ee827 R600: Modernize work item intrinsics test
Signed-off-by: Jan Vesely <jan.vesely@rutgers.edu>
Reviewed-by: Matt Arsenault <Matthew.Arsenault@amd.com>

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@214451 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-31 22:11:03 +00:00
Matt Arsenault
40a0707ee7 R600: Modernize test
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@214108 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-28 18:06:08 +00:00
Matt Arsenault
3a5e9cb146 R600/SI: Implement getOptimalMemOpType
The default guess uses i32. This needs an address space argument
to really do the right thing in all cases.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@214104 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-28 17:49:26 +00:00
Matt Arsenault
2dd264c8a3 Add alignment value to allowsUnalignedMemoryAccess
Rename to allowsMisalignedMemoryAccess.

On R600, 8 and 16 byte accesses are mostly OK with 4-byte alignment,
and don't need to be split into multiple accesses. Vector loads with
an alignment of the element type are not uncommon in OpenCL code.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@214055 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-27 17:46:40 +00:00
Matt Arsenault
86ac3829af R600/SI: Fix broken test.
There was no check prefix for the instruction lines.
Match what is emitted though, although I'm pretty sure it is
incorrect.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@214035 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-26 21:21:42 +00:00
Chandler Carruth
69359ed45b [SDAG] When performing post-legalize DAG combining, run the legalizer
over each node in the worklist prior to combining.

This allows the combiner to produce new nodes which need to go back
through legalization. This is particularly useful when generating
operands to target specific nodes in a post-legalize DAG combine where
the operands are significantly easier to express as pre-legalized
operations. My immediate use case will be PSHUFB formation where we need
to build a constant shuffle mask with a build_vector node.

This also refactors the relevant functionality in the legalizer to
support this, and updates relevant tests. I've spoken to the R600 folks
and these changes look like improvements to them. The avx512 change
needs to be investigated, I suspect there is a disagreement between the
legalizer and the DAG combiner there, but it seems a minor issue so
leaving it to be re-evaluated after this patch.

Differential Revision: http://reviews.llvm.org/D4564

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@214020 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-26 05:49:40 +00:00
Chandler Carruth
d24d326705 [SDAG] Introduce a combined set to the DAG combiner which tracks nodes
which have successfully round-tripped through the combine phase, and use
this to ensure all operands to DAG nodes are visited by the combiner,
even if they are only added during the combine phase.

This is critical to have the combiner reach nodes that are *introduced*
during combining. Previously these would sometimes be visited and
sometimes not be visited based on whether they happened to end up on the
worklist or not. Now we always run them through the combiner.

This fixes quite a few bad codegen test cases lurking in the suite while
also being more principled. Among these, the TLS codegeneration is
particularly exciting for programs that have this in the critical path
like TSan-instrumented binaries (although I think they engineer to use
a different TLS that is faster anyways).

I've tried to check for compile-time regressions here by running llc
over a merged (but not LTO-ed) clang bitcode file and observed at most
a 3% slowdown in llc. Given that this is essentially a worst case (none
of opt or clang are running at this phase) I think this is tolerable.
The actual LTO case should be even less costly, and the cost in normal
compilation should be negligible.

With this combining logic, it is possible to re-legalize as we combine
which is necessary to implement PSHUFB formation on x86 as
a post-legalize DAG combine (my ultimate goal).

Differential Revision: http://reviews.llvm.org/D4638

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213898 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-24 22:15:28 +00:00
Matt Arsenault
855a7e6eff R600: Add FMA instructions for Evergreen
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213882 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-24 17:41:01 +00:00
Matt Arsenault
de929f8b7d R600: Match rcp node on pre-SI
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213844 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-24 06:59:24 +00:00
Matt Arsenault
f303d037f2 R600: Fix LowerSDIV24
Use ComputeNumSignBits instead of checking for i8 / i16 which only
worked when AMDIL was lying about having legal i8 / i16.

If an integer is known to fit in 24-bits, we can
do division faster with float ops.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213843 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-24 06:59:20 +00:00
Chandler Carruth
a6425604c2 [SDAG] Make the DAGCombine worklist not grow endlessly due to duplicate
insertions.

The old behavior could cause arbitrarily bad memory usage in the DAG
combiner if there was heavy traffic of adding nodes already on the
worklist to it. This commit switches the DAG combine worklist to work
the same way as the instcombine worklist where we null-out removed
entries and only add new entries to the worklist. My measurements of
codegen time shows slight improvement. The memory utilization is
unsurprisingly dominated by other factors (the IR and DAG itself
I suspect).

This change results in subtle, frustrating churn in the particular order
in which DAG combines are applied which causes a number of minor
regressions where we fail to match a pattern previously matched by
accident. AFAICT, all of these should be using AddToWorklist to directly
or should be written in a less brittle way. None of the changes seem
drastically bad, and a few of the changes seem distinctly better.

A major change required to make this work is to significantly harden the
way in which the DAG combiner handle nodes which become dead
(zero-uses). Previously, we relied on the ability to "priority-bump"
them on the combine worklist to achieve recursive deletion of these
nodes and ensure that the frontier of remaining live nodes all were
added to the worklist. Instead, I've introduced a routine to just
implement that precise logic with no indirection. It is a significantly
simpler operation than that of the combiner worklist proper. I suspect
this will also fix some other problems with the combiner.

I think the x86 changes are really minor and uninteresting, but the
avx512 change at least is hiding a "regression" (despite the test case
being just noise, not testing some performance invariant) that might be
looked into. Not sure if any of the others impact specific "important"
code paths, but they didn't look terribly interesting to me, or the
changes were really minor. The consensus in review is to fix any
regressions that show up after the fact here.

Thanks to the other reviewers for checking the output on other
architectures. There is a specific regression on ARM that Tim already
has a fix prepped to commit.

Differential Revision: http://reviews.llvm.org/D4616

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213727 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-23 07:08:53 +00:00
Tom Stellard
9787e8c76b R600/SI: Add instruction shrinking pass
This pass converts 64-bit instructions to 32-bit when possible.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213561 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-21 16:55:33 +00:00
Tom Stellard
05388f25d7 R600/SI: Clean up some of the unused REGISTER_{LOAD,STORE} code
There are a few more cleanups to do, but I ran into some problems
with ext loads and trunc stores, when I tried to change some of the
vector loads and stores from custom to legal, so I wasn't able to
get rid of everything.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213552 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-21 15:45:06 +00:00
Tom Stellard
3280804237 R600/SI: Use scratch memory for large private arrays
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213551 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-21 15:45:01 +00:00
Tom Stellard
b664d47cb0 R600/SI: Store constant initializer data in constant memory
This implements a solution for constant initializers suggested
by Vadim Girlin, where we store the data after the shader code
and then use the S_GETPC instruction to compute its address.

This saves use the trouble of creating a new buffer for constant data
and then having to pass the pointer to the kernel via user SGPRs or the
input buffer.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213530 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-21 14:01:14 +00:00
Tom Stellard
54a2540fee R600/SI: Use VALU for i1 XOR
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213528 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-21 14:01:10 +00:00
Matt Arsenault
83a385a57b R600: Add missing test for concat_vectors
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213473 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-20 07:13:17 +00:00
Matt Arsenault
eb957bfe32 R600/SI: Remove dead code and add missing tests.
This probably was killed by some generic DAGCombiner
improvements in checking the TargetBooleanContents instead
of just 1.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213471 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-20 06:11:02 +00:00
Matt Arsenault
18ecf3fff3 R600/SI: implement range reduction for sin/cos
These instructions can only take a limited input range, and return
the constant value 1 out of range. We should do range reduction to
be able to process arbitrary values. Use a FRACT instruction after
normalization to achieve this. Also add a test for constant folding
with the lowered code with unsafe-fp-math enabled.

v2: use DAG lowering instead of intrinsic, adapt test
v3: calculate constant, fold pattern into instruction definition
v4: misc style fixes, add sin-fold testcase, cosmetics

Patch by Grigori Goronzy

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213458 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-19 18:44:39 +00:00
Tim Northover
7714a60ed1 R600: support fpext/fptrunc operations to and from f16.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213376 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-18 13:01:37 +00:00
Tim Northover
0afed03229 CodeGen: soften f16 type by default instead of marking legal.
Actual support for softening f16 operations is still limited, and can be added
when it's needed.  But Soften is much closer to being a useful thing to try
than keeping it Legal when no registers can actually hold such values.

Longer term, we probably want something between Soften and Promote semantics
for most targets, it'll be more efficient to promote the 4 basic operations to
f32 than libcall them.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213372 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-18 12:41:46 +00:00
Tim Northover
f811b2895f R600: rename misleading fp16 test.
This test is actually going in the opposite direction to what the
filename and function name suggested.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213358 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-18 08:43:30 +00:00
Tim Northover
cc03227446 R600: support f16 -> f64 conversion intrinsic.
Unfortunately, we don't seem to have a direct truncation, but the
extension can be legally split into two operations so we should
support that.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213357 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-18 08:43:24 +00:00
Tim Northover
3e61ccdded CodeGen: extend f16 conversions to permit types > float.
This makes the two intrinsics @llvm.convert.from.f16 and
@llvm.convert.to.f16 accept types other than simple "float". This is
only strictly needed for the truncate operation, since otherwise
double rounding occurs and there's no way to represent the strict IEEE
conversion. However, for symmetry we allow larger types in the extend
too.

During legalization, we can expand an "fp16_to_double" operation into
two extends for convenience, but abort when the truncate isn't legal. A new
libcall is probably needed here.

Even after this commit, various target tweaks are needed to actually use the
extended intrinsics. I've put these into separate commits for clarity, so there
are no actual tests of f64 conversion here.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213248 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-17 10:51:23 +00:00
Matt Arsenault
a27bf373d9 R600/SI: Allow using f32 rcp / rsq when denormals not handled.
These are precise enough to use for OpenCL unless denormals
are handled.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213107 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-15 23:50:10 +00:00
Matt Arsenault
7929c13df0 R600/SI: Fix select on i1
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213096 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-15 21:44:37 +00:00
Matt Arsenault
4b0a7f3946 R600/SI: Implement less wrong f32 fdiv
Assuming single precision denormals and accurate sqrt/div are not
reported, this passes the OpenCL conformance test.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213089 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-15 20:18:31 +00:00
Jan Vesely
7b5f8a525a R600: Implement zero undef variants of ctlz/cttz
v2: use ffbh/l if available
v3: Rebase on top of Matt's SI patches

Signed-off-by: Jan Vesely <jan.vesely@rutgers.edu>
Reviewed-by: Tom Stellard <tom@stellard.net>

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213072 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-15 15:51:09 +00:00
Matt Arsenault
5fbf09a69f R600: Add dag combine for copy of an illegal type.
This helps avoid redundant instructions to unpack, and repack
the vectors. Ideally we could recognize that pattern and eliminate
it. Currently v4i8 and other small element type vectors are scalarized,
so this has the added bonus of avoiding that.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213031 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-15 02:06:31 +00:00
Matt Arsenault
b7df516d1f R600: Add denormal handling subtarget features.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213018 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-14 23:40:49 +00:00
Matt Arsenault
5b70c8ac7e R600/SI: Default to no single precision denormals.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213017 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-14 23:40:43 +00:00
Matt Arsenault
98c2299532 R600: Run more tests with promote alloca disabled.
Re-run tests changed in r211110 to test both paths.
Also fix broken check line.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@212895 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-13 02:46:17 +00:00
Matt Arsenault
ff71f3dac1 R600: Run private-memory test with and without alloca promote
The unpromoted path still needs to be tested since we can't
always promote to using LDS.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@212894 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-13 02:18:06 +00:00
Matt Arsenault
cc6a418600 R600: Add missing tests for some intrinsics
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@212870 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-12 00:36:19 +00:00
Marek Olsak
438e1f2ad8 R600/SI: Use i32 vectors for resources and samplers
This affects new intrinsics only.

What surprises me is that v32i8 still works.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@212831 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-11 17:11:52 +00:00
Marek Olsak
5a35fdcafb R600/SI: add sample and image intrinsics exposing all instruction fields
We need the intrinsics with offsets, so why not just add them all.
The R128 parameter will also be useful for reducing SGPR usage.
GL_ARB_image_load_store also adds some image GLSL modifiers like "coherent",
so Mesa will probably translate those to slc, glc, etc.

When LLVM 3.5 is released, I'll switch Mesa to these new intrinsics.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@212830 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-11 17:11:46 +00:00
Jan Vesely
865527a09b R600: Implement float to long/ulong
Use alg. from LegalizeDAG.cpp
Move Expand setting to SIISellowering

v2: Extend existing tests instead of creating new ones
v3: use separate LowerFPTOSINT function
v4: use TargetLowering::expandFP_TO_SINT
    add comment about using FP_TO_SINT for uints

Signed-off-by: Jan Vesely <jan.vesely@rutgers.edu>
Reviewed-by: Tom Stellard <tom@stellard.net>

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@212773 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-10 22:40:21 +00:00
Matt Arsenault
b730c3d28d Revert "Revert r212640, "Add trunc (select c, a, b) -> select c (trunc a), (trunc b) combine.""
Don't try to convert the select condition type.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@212750 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-10 18:21:04 +00:00
NAKAMURA Takumi
5290734b8f Revert r212640, "Add trunc (select c, a, b) -> select c (trunc a), (trunc b) combine."
This caused miscompilation on, at least, x86-64. SExt(i1 cond) confused other optimizations.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@212708 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-10 11:37:28 +00:00
Matt Arsenault
425ef825a6 R600/SI: Add support for llvm.convert.{to|from}.fp16
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@212676 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-10 03:22:20 +00:00
Matt Arsenault
3e8ed89484 Add trunc (select c, a, b) -> select c (trunc a), (trunc b) combine.
Do this if the truncate is free and the select is legal.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@212640 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-09 19:12:07 +00:00
Matt Arsenault
0e1619e77c R600: Fix mishandling of load / store chains.
Fixes various bugs with reordering loads and stores.
Scalarized vector loads weren't collecting the chains
at all.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@212473 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-07 18:34:45 +00:00
Tom Stellard
1378871b1a R600: Promote i64 loads to v2i32
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@212216 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-02 20:53:54 +00:00
Matt Arsenault
cc0a279949 Revert "Temporary hack to try cleaning extra .s file from bots."
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211967 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-27 23:11:26 +00:00
Matt Arsenault
051cc062b2 Temporary hack to try cleaning extra .s file from bots.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211963 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-27 21:43:50 +00:00
David Blaikie
e70cdf9468 Fix test so it doesn't try to write out temporary files into the test tree.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211916 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-27 17:45:43 +00:00
Matt Arsenault
ee5d4a7b73 R600: Don't crash on unhandled instruction in promote alloca
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211906 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-27 16:52:49 +00:00
Matt Arsenault
88a3c72e25 R600: Add some testcases for promote alloca pass.
More complicated GEPs are skipped. Add some tests to
actually stress this skipping.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211859 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-27 03:55:55 +00:00
Matt Arsenault
3cd8cf6bbd R600/SI: Add FP mode bits to binary.
The default rounding mode to initialize the mode register needs
to be reported to the runtime. Fill in other bits a kernel
may be interested in setting for future use.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211791 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-26 17:22:30 +00:00
Matt Arsenault
b0f5a0e7e7 R600: Fix vector FMA
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211757 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-26 01:28:05 +00:00
Tom Stellard
78d1e95201 R600: Promote i64 stores to v2i32
Now we need only one 64-bit pattern for stores.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211643 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-24 23:33:04 +00:00
Matt Arsenault
95eb45c5d9 R600: Fix inconsistency in rsq instructions.
R600 was using a clamped version of rsq, but SI was not. Add a
new rsq_clamped intrinsic and use them consistently.

It's unclear to me from the documentation what behavior
the R600 instructions have, so I assume they have the legacy behavior
described by the SI documents. For R600, use RECIPSQRT_IEEE
for both llvm.AMDGPU.rsq.legacy and llvm.AMDGPU.rsq. R600 also
has RECIPSQRT_FF, which I'm not sure how it fits in here.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211637 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-24 22:13:39 +00:00
Matt Arsenault
ed143b7c0c R600/SI: Fix div_scale intrinsic.
The operand that must match one of the others does matter,
and implement selecting for it.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211523 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-23 18:28:28 +00:00
Matt Arsenault
9ad2c7ef92 R600: Move add/sub with overflow out of AMDILISelLowering
Add more tests for these.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211517 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-23 18:00:49 +00:00
Matt Arsenault
c4471e9248 R600/SI: Handle i64 sub.
We can handle it the same way as add

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211514 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-23 18:00:38 +00:00
Jan Vesely
728ea0c91b R600: Add udivrem test
v2: move < %s to the end of the line
    space after ;
    add v4i32 test

Signed-off-by: Jan Vesely <jan.vesely@rutgers.edu>

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211476 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-22 21:42:58 +00:00
Tom Stellard
c0bf939e80 R600/SI: Add patterns for ctpop inside a branch
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211378 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-20 17:06:11 +00:00
Tom Stellard
61d64acd0c R600/SI: Add a pattern for f32 ftrunc
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211377 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-20 17:06:09 +00:00
Tom Stellard
2cda6e8ca6 R600: Expand vector flog2
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211376 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-20 17:06:07 +00:00
Tom Stellard
2d245e2da4 R600: Expand vector fexp2
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211375 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-20 17:06:05 +00:00
Tom Stellard
538c95179c R600/SI: Add a VALU pattern for i64 xor
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211373 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-20 17:05:57 +00:00
Matt Arsenault
64429cefba R600: Add a few tests I forgot to add.
These belong with r210827

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211253 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-19 04:24:43 +00:00
Matt Arsenault
d9b35435b8 R600/SI: Add intrinsics for various math instructions.
These will be used for custom lowering and for library
implementations of various math functions, so it's useful
to expose these as builtins.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211247 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-19 01:19:19 +00:00
Matt Arsenault
ce09bda96e R600: Handle fnearbyint
The difference from rint isn't really relevant here,
so treat them as equivalent. OpenCL doesn't have nearbyint,
so this is sort of pointless other than for completeness.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211229 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-18 22:03:45 +00:00
Marek Olsak
f286d63757 R600/SI: add gather4 and getlod intrinsics (v3)
This contains all the previous patches + getlod support on top of it.
It doesn't use SDNodes anymore, so it's quite small.
It also adds v16i8 to SReg_128, which is used for the sampler descriptor.

Reviewed-by: Tom Stellard

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211228 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-18 22:00:29 +00:00
Jan Vesely
52b6c2d6ef R600: Expand vector fceil
Move fp64 fceil tests to fceil64.ll

v2: rebase

Signed-off-by: Jan Vesely <jan.vesely@rutgers.edu>

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211194 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-18 17:57:29 +00:00
Matt Arsenault
2b6e6fc1a8 R600/SI: Add intrinsics for brev instructions
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211187 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-18 17:13:57 +00:00
Matt Arsenault
795ae8615f R600/SI: Prettier operand printing for 64-bit ops.
Copy what is done for 32-bit already so the order is about the same.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211186 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-18 17:13:51 +00:00
Matt Arsenault
debd831223 R600: Implement f64 ftrunc, ffloor and fceil.
CI has instructions for these, so this fixes them for older hardware.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211183 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-18 17:05:30 +00:00
Matt Arsenault
a5395c03f0 R600: Custom lower f64 frint for pre-CI
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211182 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-18 17:05:26 +00:00
Jan Vesely
c32d52df24 R600: Implement 64bit SRA
v2: Use capitalized variable name

Signed-off-by: Jan Vesely <jan.vesely@rutgers.edu>

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211159 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-18 12:27:17 +00:00
Jan Vesely
2d06e73d88 R600: Implement 64bit SRL
v2: use C++ style comment

Signed-off-by: Jan Vesely <jan.vesely@rutgers.edu>

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211158 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-18 12:27:15 +00:00
Jan Vesely
a64058f3eb R600: Implement 64bit SHL
v2: Use c++ style comment

Signed-off-by: Jan Vesely <jan.vesely@rutgers.edu>

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211157 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-18 12:27:13 +00:00
Matt Arsenault
3f1f259c22 R600/SI: Match cttz_zero_undef
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211116 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-17 17:36:27 +00:00
Matt Arsenault
62e378b057 R600/SI: Match ctlz_zero_undef
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211115 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-17 17:36:24 +00:00
Tom Stellard
f56e7678d1 R600: Use LDS and vectors for private memory
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211110 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-17 16:53:14 +00:00
Tom Stellard
bae98b1b45 SelectionDAG: Expand i64 = FP_TO_SINT i32
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211108 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-17 16:53:07 +00:00
Matt Arsenault
3c11f69001 R600: Add a rotr testcase I forgot to add
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211002 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-15 21:09:00 +00:00
Matt Arsenault
fa848ccd09 R600: Remove a few more things from AMDILISelLowering
Try to keep all the setOperationActions for integer ops
together.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211001 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-15 21:08:58 +00:00
Matt Arsenault
e2480a202f R600: Fix assert on vector sdiv
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211000 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-15 21:08:54 +00:00
Matt Arsenault
61bfbc4d96 R600: Report that integer division is expensive.
Divides by weird constants now emit much better code.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@210995 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-15 19:48:16 +00:00
NAKAMURA Takumi
e12d893d7a Don't expect tests always crashing. Add "REQUIRES:asserts".
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@210983 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-15 01:01:11 +00:00
Matt Arsenault
7a3a020478 R600: Add failing testcases.
These are reduced from assert in the
OpenCV CvtColor8u.BGR5652GRAY test.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@210969 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-14 04:26:09 +00:00
Matt Arsenault
aac455af99 R600: Fix asserts related to constant initializers
This would assert if a constant address space was extern
and therefore didn't have an initializer. If the initializer
was undef, it would hit the unreachable unhandled initializer case.

An extern global should never really occur since we don't have
machine linking, but bugpoint likes to remove initializers.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@210967 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-14 04:26:05 +00:00
Tim Northover
8f2a85e099 IR: add "cmpxchg weak" variant to support permitted failure.
This commit adds a weak variant of the cmpxchg operation, as described
in C++11. A cmpxchg instruction with this modifier is permitted to
fail to store, even if the comparison indicated it should.

As a result, cmpxchg instructions must return a flag indicating
success in addition to their original iN value loaded. Thus, for
uniformity *all* cmpxchg instructions now return "{ iN, i1 }". The
second flag is 1 when the store succeeded.

At the DAG level, a new ATOMIC_CMP_SWAP_WITH_SUCCESS node has been
added as the natural representation for the new cmpxchg instructions.
It is a strong cmpxchg.

By default this gets Expanded to the existing ATOMIC_CMP_SWAP during
Legalization, so existing backends should see no change in behaviour.
If they wish to deal with the enhanced node instead, they can call
setOperationAction on it. Beware: as a node with 2 results, it cannot
be selected from TableGen.

Currently, no use is made of the extra information provided in this
patch. Test updates are almost entirely adapting the input IR to the
new scheme.

Summary for out of tree users:
------------------------------

+ Legacy Bitcode files are upgraded during read.
+ Legacy assembly IR files will be invalid.
+ Front-ends must adapt to different type for "cmpxchg".
+ Backends should be unaffected by default.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@210903 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-13 14:24:07 +00:00
Matt Arsenault
d344c6bcf9 R600/SI: Fix selection error on i64 rotl / rotr.
Evergreen is still broken due to missing shl_parts.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@210885 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-13 04:00:30 +00:00
Matt Arsenault
00c3986254 R600: Mostly remove remaining AMDIL intrinsics.
Delete all unused ones, and add new AMDGPU named intrinsics for
the ones that are. Handle the old AMDIL names for comptability (although
remove their GCCBuiltin names) and add tests since there weren't any
for these before.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@210827 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-12 21:15:44 +00:00
Tom Stellard
82a51defb6 Revert "SelectionDAG: Enable (and (setcc x), (setcc y)) -> (setcc (and x, y)) for vectors"
This reverts commit r210540, adds a testcase for the regression it
caused, and marks the R600 test it was supposed to fix as XFAIL.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@210792 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-12 16:04:47 +00:00
Matt Arsenault
0b87955888 R600/SI: Use a register set to -1 for data0 on ds_inc*/ds_dec*
There is not such thing as a 0-data ds instruction, and the data
operand needs to be a vgpr set to something meaningful.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@210756 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-12 08:21:54 +00:00
Matt Arsenault
7fa80b45eb R600/SI: Fix bitcast between v2i32 and f64
This is the same problem fixed in r210664 for more types.

The test passes without this fix. For some reason
I'm only hitting this when creating selects lowered
to v2i32 selects.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@210692 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-11 19:31:13 +00:00
Matt Arsenault
c9dbd0da7a R600/SI: Add common 64-bit LDS atomics
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@210680 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-11 18:08:54 +00:00
Matt Arsenault
6b19a3a474 R600/SI: Add 32-bit LDS atomic cmpxchg
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@210678 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-11 18:08:48 +00:00
Matt Arsenault
a396a70c1d R600/SI: Use LDS atomic inc / dec
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@210677 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-11 18:08:45 +00:00
Matt Arsenault
2da1a85cbb R600/SI: Add other LDS atomic operations
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@210676 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-11 18:08:42 +00:00
Matt Arsenault
4a19dd468d R600/SI: Fix backwards names for local atomic instructions.
The manual lists them as *_RTN_U32, not *_U32_RTN, which is more
consistent with how every other sized instruction is named.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@210674 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-11 18:08:37 +00:00