Commit Graph

112 Commits

Author SHA1 Message Date
Jingyue Wu
9830b459be [NVPTX] Fix bugs related to isSingleValueType
Summary:
With isSingleValueType starting to treat vector types as single-value types,
code that uses this interface needs to be updated.

Test Plan:
vector-global.ll
nvcl-param-align.ll

Reviewers: jholewinski

Reviewed By: jholewinski

Subscribers: llvm-commits, meheff, eliben, jholewinski

Differential Revision: http://reviews.llvm.org/D6573

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@224440 91177308-0d34-0410-b5e6-96231b3b80d8
2014-12-17 17:59:04 +00:00
Duncan P. N. Exon Smith
1ef70ff39b IR: Make metadata typeless in assembly
Now that `Metadata` is typeless, reflect that in the assembly.  These
are the matching assembly changes for the metadata/value split in
r223802.

  - Only use the `metadata` type when referencing metadata from a call
    intrinsic -- i.e., only when it's used as a `Value`.

  - Stop pretending that `ValueAsMetadata` is wrapped in an `MDNode`
    when referencing it from call intrinsics.

So, assembly like this:

    define @foo(i32 %v) {
      call void @llvm.foo(metadata !{i32 %v}, metadata !0)
      call void @llvm.foo(metadata !{i32 7}, metadata !0)
      call void @llvm.foo(metadata !1, metadata !0)
      call void @llvm.foo(metadata !3, metadata !0)
      call void @llvm.foo(metadata !{metadata !3}, metadata !0)
      ret void, !bar !2
    }
    !0 = metadata !{metadata !2}
    !1 = metadata !{i32* @global}
    !2 = metadata !{metadata !3}
    !3 = metadata !{}

turns into this:

    define @foo(i32 %v) {
      call void @llvm.foo(metadata i32 %v, metadata !0)
      call void @llvm.foo(metadata i32 7, metadata !0)
      call void @llvm.foo(metadata i32* @global, metadata !0)
      call void @llvm.foo(metadata !3, metadata !0)
      call void @llvm.foo(metadata !{!3}, metadata !0)
      ret void, !bar !2
    }
    !0 = !{!2}
    !1 = !{i32* @global}
    !2 = !{!3}
    !3 = !{}

I wrote an upgrade script that handled almost all of the tests in llvm
and many of the tests in cfe (even handling many `CHECK` lines).  I've
attached it (or will attach it in a moment if you're speedy) to PR21532
to help everyone update their out-of-tree testcases.

This is part of PR21532.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@224257 91177308-0d34-0410-b5e6-96231b3b80d8
2014-12-15 19:07:53 +00:00
Duncan P. N. Exon Smith
8f616165e4 IR: Canonicalize metadata formatting, NFC
Canonicalize formatting of metadata to make it easier to upgrade via
scripts -- in particular, one line per metadata definition makes it more
`sed`-able.

This is preparation for changing the assembly syntax for metadata [1].

[1]: http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20141208/248449.html

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@224002 91177308-0d34-0410-b5e6-96231b3b80d8
2014-12-11 06:32:29 +00:00
Jingyue Wu
b043278834 [NVPTX] Do not emit .weak symbols for NVPTX
Summary:
".weak" symbols cannot be consumed by ptxas (PR21685). This patch makes the
weak directive in MCAsmPrinter customizable, and disables emitting ".weak"
symbols for NVPTX.

Test Plan: weak-linkage.ll

Reviewers: jholewinski

Reviewed By: jholewinski

Subscribers: majnemer, jholewinski, llvm-commits

Differential Revision: http://reviews.llvm.org/D6455

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@223077 91177308-0d34-0410-b5e6-96231b3b80d8
2014-12-01 21:16:17 +00:00
Justin Holewinski
e459c0bf65 [NVPTX] Add NVPTXLowerStructArgs pass
This works around the limitation that PTX does not allow .param space
loads/stores with arbitrary pointers.

If a function has a by-val struct ptr arg, say foo(%struct.x *byval %d), then
add the following instructions to the first basic block :

%temp = alloca %struct.x, align 8
%tt1 = bitcast %struct.x * %d to i8 *
%tt2 = llvm.nvvm.cvt.gen.to.param %tt2
%tempd = bitcast i8 addrspace(101) * to %struct.x addrspace(101) *
%tv = load %struct.x addrspace(101) * %tempd
store %struct.x %tv, %struct.x * %temp, align 8

The above code allocates some space in the stack and copies the incoming
struct from param space to local space. Then replace all occurences of %d
by %temp.

Fixes PR21465.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@221377 91177308-0d34-0410-b5e6-96231b3b80d8
2014-11-05 18:19:30 +00:00
Jingyue Wu
1d1d705a95 [NVPTX] aligned byte-buffers for vector return types
Summary:
Fixes PR21100 which is caused by inconsistency between the declared return type
and the expected return type at the call site. The new behavior is consistent
with nvcc and the NVPTXTargetLowering::getPrototype function.

Test Plan: test/Codegen/NVPTX/vector-return.ll

Reviewers: jholewinski

Reviewed By: jholewinski

Subscribers: llvm-commits, meheff, eliben, jholewinski

Differential Revision: http://reviews.llvm.org/D5612

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@220607 91177308-0d34-0410-b5e6-96231b3b80d8
2014-10-25 03:46:16 +00:00
Jingyue Wu
75d77cd179 [MachineSink] Use the real post dominator tree
Summary:
Fixes a FIXME in MachineSinking. Instead of using the simple heuristics in
isPostDominatedBy, use the real MachinePostDominatorTree and MachineLoopInfo.
The old heuristics caused instructions to sink unnecessarily, and might create
register pressure.

This is the second try of the fix. The first one (D4814) caused a performance
regression due to failing to sink instructions out of loops (PR21115). This
patch fixes PR21115 by sinking an instruction from a deeper loop to a shallower
one regardless of whether the target block post-dominates the source.

Thanks Alexey Volkov for reporting PR21115! 

Test Plan:
Added a NVPTX codegen test to verify that our change prevents the backend from
over-sinking. It also shows the unnecessary register pressure caused by
over-sinking.

Added an X86 test to verify we can sink instructions out of loops regardless of
the dominance relationship. This test is reduced from Alexey's test in PR21115.

Updated an affected test in X86.

Also ran SPEC CINT2006 and llvm-test-suite for compilation time and runtime
performance. Results are attached separately in the review thread.

Reviewers: Jiangning, resistor, hfinkel

Reviewed By: hfinkel

Subscribers: hfinkel, bruno, volkalexey, llvm-commits, meheff, eliben, jholewinski

Differential Revision: http://reviews.llvm.org/D5633

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@219773 91177308-0d34-0410-b5e6-96231b3b80d8
2014-10-15 03:27:43 +00:00
Jingyue Wu
ccd995ab0c Revert r216862 due to a performance regression
Reported by Alexey Volkov in PR21115


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@218771 91177308-0d34-0410-b5e6-96231b3b80d8
2014-10-01 15:22:13 +00:00
Jingyue Wu
d43e6df10b [MachineSink] Use the real post dominator tree
Summary:
Fixes a FIXME in MachineSinking. Instead of using the simple heuristics
in isPostDominatedBy, use the real MachinePostDominatorTree. The old
heuristics caused instructions to sink unnecessarily, and might create
register pressure.

Test Plan:
Added a NVPTX codegen test to verify that our change is in effect. It also
shows the unnecessary register pressure caused by over-sinking. Updated
affected tests in AArch64 and X86.

Reviewers: eliben, meheff, Jiangning

Reviewed By: Jiangning

Subscribers: jholewinski, aemerson, mcrosier, llvm-commits

Differential Revision: http://reviews.llvm.org/D4814



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@216862 91177308-0d34-0410-b5e6-96231b3b80d8
2014-09-01 03:47:25 +00:00
Jingyue Wu
87a2b36cf6 [NVPTX] Make the alignment an explicit argument to ldu/ldg
Summary:
Instead of specifying the alignment as metadata which may be destroyed by
transformation passes, make the alignment the second argument to ldu/ldg
intrinsic calls.

Test Plan:
ldu-ldg.ll
ldu-i8.ll
ldu-reg-plus-offset.ll

Reviewers: eliben, meheff, jholewinski

Reviewed By: meheff, jholewinski

Subscribers: jholewinski, llvm-commits

Differential Revision: http://reviews.llvm.org/D5093

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@216731 91177308-0d34-0410-b5e6-96231b3b80d8
2014-08-29 15:30:20 +00:00
Justin Holewinski
2941802bad [NVPTX] Add some extra tests for mul.wide to test non-power-of-two source types
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213794 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-23 20:23:49 +00:00
Justin Holewinski
03f160f9d3 [NVPTX] mul.wide generation works for any smaller integer source types, not just the next smaller power of two
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213784 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-23 18:46:03 +00:00
Justin Holewinski
6b50a7d179 [NVPTX] Make sure we do not generate MULWIDE ISD nodes when optimizations are disabled
With optimizations disabled, we disable the isel patterns for mul.wide; but we
were still generating MULWIDE ISD nodes.  Now, we only try to generate MULWIDE
ISD nodes in DAGCombine if the optimization level is not zero.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213773 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-23 17:40:45 +00:00
Eli Bendersky
07efe87198 Add some tests for NVPTX lowering of cmpxchg
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213586 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-21 22:54:44 +00:00
Eli Bendersky
bd9c6c0773 Add tests for atomic adds on floats.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213406 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-18 20:11:26 +00:00
Eli Bendersky
6a33571ba9 Use CHECK-LABEL where appropriate in this test.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213398 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-18 19:32:09 +00:00
Tim Northover
b41b1d4bac NVPTX: support fpext/fptrunc to and from f16.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213377 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-18 13:01:43 +00:00
Tim Northover
0afed03229 CodeGen: soften f16 type by default instead of marking legal.
Actual support for softening f16 operations is still limited, and can be added
when it's needed.  But Soften is much closer to being a useful thing to try
than keeping it Legal when no registers can actually hold such values.

Longer term, we probably want something between Soften and Promote semantics
for most targets, it'll be more efficient to promote the 4 basic operations to
f32 than libcall them.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213372 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-18 12:41:46 +00:00
Tim Northover
7bbf5786d7 NVPTX: support direct f16 <-> f64 conversions via intrinsics.
Clang may well start emitting these soon, and while it may not be
directly relevant for OpenCL or GLSL, the instructions were just
sitting there waiting to be used.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213356 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-18 08:30:10 +00:00
Justin Holewinski
11ae250ec9 [NVPTX] Improve handling of FP fusion
We now consider the FPOpFusion flag when determining whether
to fuse ops.  We also explicitly emit add.rn when fusion is
disabled to prevent ptxas from fusing the operations on its
own.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213287 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-17 18:10:09 +00:00
Justin Holewinski
07bc0b6ae6 [NVPTX] Add missing .v4 qualifier on vector store instruction
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213276 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-17 16:58:56 +00:00
Justin Holewinski
b26ede07ad [NVPTX] Flag surface/texture query instructions with IsTexSurfQuery
Also, add some tests to make sure we can handle surface/texture
queries on both Fermi and Kepler+.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213268 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-17 14:51:33 +00:00
Justin Holewinski
d6663f565c [NVPTX] Add more surface/texture intrinsics, including CUDA unified texture fetch
This also uses TSFlags to mark machine instructions that are surface/texture
accesses, as well as the vector width for surface operations.  This is used
to simplify some of the switch statements that need to detect surface/texture
instructions

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213256 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-17 11:59:04 +00:00
Justin Holewinski
a1535e3b9b [NVPTX] Honor alignment on vector loads/stores
We were not considering the stated alignment on vector loads/stores,
leading us to generate vector instructions even when we do not have
sufficient alignment.

Now, for IR like:

  %1 = load <4 x float>, <4 x float>* %ptr, align 4

we will generate correct, conservative PTX like:

  ld.f32 ... [%ptr]
  ld.f32 ... [%ptr+4]
  ld.f32 ... [%ptr+8]
  ld.f32 ... [%ptr+12]

Or if we have an alignment of 8 (for example), we can
generate code like:

  ld.v2.f32 ... [%ptr]
  ld.v2.f32 ... [%ptr+8]

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213186 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-16 19:45:35 +00:00
Justin Holewinski
7e6565112b [NVPTX] Rename registers %fl -> %fd and %rl -> %rd
This matches the internal behavior of NVIDIA tools like libnvvm.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@213168 91177308-0d34-0410-b5e6-96231b3b80d8
2014-07-16 16:26:58 +00:00
Justin Holewinski
7a28de08f3 [NVPTX] Add reflect intrinsic (better than matching by function name)
Also clean up some of the logic in NVVMReflect.cpp while we're messing around in there.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211948 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-27 18:36:11 +00:00
Justin Holewinski
c95d327874 [NVPTX] Add 'b' asm constraint
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211946 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-27 18:36:06 +00:00
Justin Holewinski
3c81367a5d [NVPTX] Error out if initializer is given for variable in an address space that does not support initialization
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211943 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-27 18:36:01 +00:00
Justin Holewinski
0ded57ccc5 [NVPTX] Add support for .managed variables for UVM
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211942 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-27 18:35:58 +00:00
Justin Holewinski
2a8dc35cca [NVPTX] Emit .weak linkage for link_once, weak, available_externally, and common linkage
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211941 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-27 18:35:56 +00:00
Justin Holewinski
cb8f98382b [NVPTX] Fix handling of ldg/ldu intrinsics.
The address space of the pointer must be global (1) for these intrinsics.  There must also be alignment metadata attached to the intrinsic calls, e.g.

%val = tail call i32 @llvm.nvvm.ldu.i.global.i32.p1i32(i32 addrspace(1)* %ptr), !align !0

!0 = metadata !{i32 4}

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211939 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-27 18:35:51 +00:00
Justin Holewinski
8992274412 [NVPTX] Clean up argument lowering code and properly handle alignment for structs and vectors
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211938 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-27 18:35:44 +00:00
Justin Holewinski
863b0d45a5 [NVPTX] Add support for [SHL,SRA,SRL]_PARTS
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211936 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-27 18:35:40 +00:00
Justin Holewinski
10da1651ed [NVPTX] Implement fma and imad contraction as target DAGCombiner patterns
This also introduces DAGCombiner patterns for mul.wide to multiply two smaller integers and produce a larger integer

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211935 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-27 18:35:37 +00:00
Justin Holewinski
508c80f11f [NVPTX] Add support for efficient rotate instructions on SM 3.2+
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211934 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-27 18:35:33 +00:00
Justin Holewinski
1f75f4a0ee [NVPTX] Add missing isel patterns for 64-bit atomics
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211933 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-27 18:35:30 +00:00
Justin Holewinski
ef92cf50d6 [NVPTX] Add isel patterns for bit-field extract (bfe)
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211932 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-27 18:35:27 +00:00
Justin Holewinski
de7bbdff33 [NVPTX] Add support for isspacep instruction
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211931 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-27 18:35:24 +00:00
Justin Holewinski
1571d272c8 [NVPTX] Add support for envreg reads
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211930 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-27 18:35:21 +00:00
Justin Holewinski
a54609ed93 [NVPTX] Emit .weak when linkage is not external, internal, or private
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211926 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-27 18:35:10 +00:00
Jingyue Wu
f6eb7e3175 Canonicalize addrspacecast ConstExpr between different pointer types
As a follow-up to r210375 which canonicalizes addrspacecast
instructions, this patch canonicalizes addrspacecast constant
expressions.

Given clang uses ConstantExpr::getAddrSpaceCast to emit addrspacecast
cosntant expressions, this patch is also a step towards having the
frontend emit canonicalized addrspacecasts.

Piggyback a minor refactor in InstCombineCasts.cpp

Update three affected tests in addrspacecast-alias.ll,
access-non-generic.ll and constant-fold-gep.ll and added one new test in
constant-fold-address-space-pointer.ll


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211004 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-15 21:40:57 +00:00
Alp Toker
8aeca44558 Reduce verbiage of lit.local.cfg files
We can just split targets_to_build in one place and make it immutable.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@210496 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-09 22:42:55 +00:00
Rafael Espindola
6fd1b8ee48 Allow aliases to be unnamed_addr.
Alias with unnamed_addr were in a strange state. It is stored in GlobalValue,
the language reference talks about "unnamed_addr aliases" but the verifier
was rejecting them.

It seems natural to allow unnamed_addr in aliases:

* It is a property of how it is accessed, not of the data itself.
* It is perfectly possible to write code that depends on the address
of an alias.

This patch then makes unname_addr legal for aliases. One side effect is that
the syntax changes for a corner case: In globals, unnamed_addr is now printed
before the address space.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@210302 91177308-0d34-0410-b5e6-96231b3b80d8
2014-06-06 01:20:28 +00:00
Eli Bendersky
85af3e7445 Fix the test: DCE optimized away everything.
Use volatile store to protect the generated PTX from DCE.

Patch by Jingyue Wu.



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@206763 91177308-0d34-0410-b5e6-96231b3b80d8
2014-04-21 17:23:12 +00:00
Justin Holewinski
77f268945e [NVPTX] Add preliminary intrinsics and codegen support for textures/surfaces
This commit adds intrinsics and codegen support for the surface read/write and texture read instructions that take an explicit sampler parameter. Codegen operates on image handles at the PTX level, but falls back to direct replacement of handles with kernel arguments if image handles are not enabled. Note that image handles are explicitly disabled for all target architectures in this change (to be enabled later).

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@205907 91177308-0d34-0410-b5e6-96231b3b80d8
2014-04-09 15:39:15 +00:00
Justin Holewinski
ac4c131de6 [NVPTX] Add support for addrspacecast in global variable initializers, including emitting generic() when casting to address space 0.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@205906 91177308-0d34-0410-b5e6-96231b3b80d8
2014-04-09 15:39:11 +00:00
Eli Bendersky
25540a7f39 Optimize away unnecessary address casts.
Removes unnecessary casts from non-generic address spaces to the generic address
space for certain code patterns.

Patch by Jingyue Wu.



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@205571 91177308-0d34-0410-b5e6-96231b3b80d8
2014-04-03 21:18:25 +00:00
Eli Bendersky
416a0e993f Fix for PR19099 - NVPTX produces invalid symbol names.
This is a more thorough fix for the issue than r203483. An IR pass will run
before NVPTX codegen to make sure there are no invalid symbol names that can't
be consumed by the ptxas assembler.



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@205212 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-31 15:56:26 +00:00
Eli Bendersky
b6f02c13da Add test to test/CodeGen/NVPTX for "alloca buffer" arguments.
Make sure such IR gets properly lowered to PTX.


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@204624 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-24 16:52:30 +00:00
Justin Holewinski
b8cb709858 [NVPTX] Add isel patterns for addrspacecast
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@204600 91177308-0d34-0410-b5e6-96231b3b80d8
2014-03-24 11:17:53 +00:00