IR: Split Metadata from Value

Split `Metadata` away from the `Value` class hierarchy, as part of
PR21532.  Assembly and bitcode changes are in the wings, but this is the
bulk of the change for the IR C++ API.

I have a follow-up patch prepared for `clang`.  If this breaks other
sub-projects, I apologize in advance :(.  Help me compile it on Darwin
I'll try to fix it.  FWIW, the errors should be easy to fix, so it may
be simpler to just fix it yourself.

This breaks the build for all metadata-related code that's out-of-tree.
Rest assured the transition is mechanical and the compiler should catch
almost all of the problems.

Here's a quick guide for updating your code:

  - `Metadata` is the root of a class hierarchy with three main classes:
    `MDNode`, `MDString`, and `ValueAsMetadata`.  It is distinct from
    the `Value` class hierarchy.  It is typeless -- i.e., instances do
    *not* have a `Type`.

  - `MDNode`'s operands are all `Metadata *` (instead of `Value *`).

  - `TrackingVH<MDNode>` and `WeakVH` referring to metadata can be
    replaced with `TrackingMDNodeRef` and `TrackingMDRef`, respectively.

    If you're referring solely to resolved `MDNode`s -- post graph
    construction -- just use `MDNode*`.

  - `MDNode` (and the rest of `Metadata`) have only limited support for
    `replaceAllUsesWith()`.

    As long as an `MDNode` is pointing at a forward declaration -- the
    result of `MDNode::getTemporary()` -- it maintains a side map of its
    uses and can RAUW itself.  Once the forward declarations are fully
    resolved RAUW support is dropped on the ground.  This means that
    uniquing collisions on changing operands cause nodes to become
    "distinct".  (This already happened fairly commonly, whenever an
    operand went to null.)

    If you're constructing complex (non self-reference) `MDNode` cycles,
    you need to call `MDNode::resolveCycles()` on each node (or on a
    top-level node that somehow references all of the nodes).  Also,
    don't do that.  Metadata cycles (and the RAUW machinery needed to
    construct them) are expensive.

  - An `MDNode` can only refer to a `Constant` through a bridge called
    `ConstantAsMetadata` (one of the subclasses of `ValueAsMetadata`).

    As a side effect, accessing an operand of an `MDNode` that is known
    to be, e.g., `ConstantInt`, takes three steps: first, cast from
    `Metadata` to `ConstantAsMetadata`; second, extract the `Constant`;
    third, cast down to `ConstantInt`.

    The eventual goal is to introduce `MDInt`/`MDFloat`/etc. and have
    metadata schema owners transition away from using `Constant`s when
    the type isn't important (and they don't care about referring to
    `GlobalValue`s).

    In the meantime, I've added transitional API to the `mdconst`
    namespace that matches semantics with the old code, in order to
    avoid adding the error-prone three-step equivalent to every call
    site.  If your old code was:

        MDNode *N = foo();
        bar(isa             <ConstantInt>(N->getOperand(0)));
        baz(cast            <ConstantInt>(N->getOperand(1)));
        bak(cast_or_null    <ConstantInt>(N->getOperand(2)));
        bat(dyn_cast        <ConstantInt>(N->getOperand(3)));
        bay(dyn_cast_or_null<ConstantInt>(N->getOperand(4)));

    you can trivially match its semantics with:

        MDNode *N = foo();
        bar(mdconst::hasa               <ConstantInt>(N->getOperand(0)));
        baz(mdconst::extract            <ConstantInt>(N->getOperand(1)));
        bak(mdconst::extract_or_null    <ConstantInt>(N->getOperand(2)));
        bat(mdconst::dyn_extract        <ConstantInt>(N->getOperand(3)));
        bay(mdconst::dyn_extract_or_null<ConstantInt>(N->getOperand(4)));

    and when you transition your metadata schema to `MDInt`:

        MDNode *N = foo();
        bar(isa             <MDInt>(N->getOperand(0)));
        baz(cast            <MDInt>(N->getOperand(1)));
        bak(cast_or_null    <MDInt>(N->getOperand(2)));
        bat(dyn_cast        <MDInt>(N->getOperand(3)));
        bay(dyn_cast_or_null<MDInt>(N->getOperand(4)));

  - A `CallInst` -- specifically, intrinsic instructions -- can refer to
    metadata through a bridge called `MetadataAsValue`.  This is a
    subclass of `Value` where `getType()->isMetadataTy()`.

    `MetadataAsValue` is the *only* class that can legally refer to a
    `LocalAsMetadata`, which is a bridged form of non-`Constant` values
    like `Argument` and `Instruction`.  It can also refer to any other
    `Metadata` subclass.

(I'll break all your testcases in a follow-up commit, when I propagate
this change to assembly.)

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@223802 91177308-0d34-0410-b5e6-96231b3b80d8

include/llvm/IR/Metadata.h

@@ -18,11 +18,13 @@
 
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/DenseMap.h"
-#include "llvm/ADT/FoldingSet.h"
 #include "llvm/ADT/ilist_node.h"
 #include "llvm/ADT/iterator_range.h"
+#include "llvm/IR/Constant.h"
+#include "llvm/IR/MetadataTracking.h"
 #include "llvm/IR/Value.h"
 #include "llvm/Support/ErrorHandling.h"
+#include <type_traits>
 
 namespace llvm {
 class LLVMContext;
@@ -38,20 +40,388 @@ enum LLVMConstants : uint32_t {
 /// \brief Root of the metadata hierarchy.
 ///
 /// This is a root class for typeless data in the IR.
-///
-/// TODO: Detach from the Value hierarchy.
-class Metadata : public Value {
+class Metadata {
+  friend class ReplaceableMetadataImpl;
+
+  /// \brief RTTI.
+  const unsigned char SubclassID;
+
 protected:
-  Metadata(LLVMContext &Context, unsigned ID);
+  /// \brief Storage flag for non-uniqued, otherwise unowned, metadata.
+  bool IsDistinctInContext : 1;
+  // TODO: expose remaining bits to subclasses.
+
+  unsigned short SubclassData16;
+  unsigned SubclassData32;
 
 public:
+  enum MetadataKind {
+    GenericMDNodeKind,
+    MDNodeFwdDeclKind,
+    ConstantAsMetadataKind,
+    LocalAsMetadataKind,
+    MDStringKind
+  };
+
+protected:
+  Metadata(unsigned ID)
+      : SubclassID(ID), IsDistinctInContext(false), SubclassData16(0),
+        SubclassData32(0) {}
+  ~Metadata() {}
+
+  /// \brief Store this in a big non-uniqued untyped bucket.
+  bool isStoredDistinctInContext() const { return IsDistinctInContext; }
+
+  /// \brief Default handling of a changed operand, which asserts.
+  ///
+  /// If subclasses pass themselves in as owners to a tracking node reference,
+  /// they must provide an implementation of this method.
+  void handleChangedOperand(void *, Metadata *) {
+    llvm_unreachable("Unimplemented in Metadata subclass");
+  }
+
+public:
+  unsigned getMetadataID() const { return SubclassID; }
+
+  /// \brief User-friendly dump.
+  void dump() const;
+  void print(raw_ostream &OS) const;
+  void printAsOperand(raw_ostream &OS, bool PrintType = true,
+                      const Module *M = nullptr) const;
+};
+
+#define HANDLE_METADATA(CLASS) class CLASS;
+#include "llvm/IR/Metadata.def"
+
+inline raw_ostream &operator<<(raw_ostream &OS, const Metadata &MD) {
+  MD.print(OS);
+  return OS;
+}
+
+/// \brief Metadata wrapper in the Value hierarchy.
+///
+/// A member of the \a Value hierarchy to represent a reference to metadata.
+/// This allows, e.g., instrinsics to have metadata as operands.
+///
+/// Notably, this is the only thing in either hierarchy that is allowed to
+/// reference \a LocalAsMetadata.
+class MetadataAsValue : public Value {
+  friend class ReplaceableMetadataImpl;
+  friend class LLVMContextImpl;
+
+  Metadata *MD;
+
+  MetadataAsValue(Type *Ty, Metadata *MD);
+  ~MetadataAsValue();
+
+public:
+  static MetadataAsValue *get(LLVMContext &Context, Metadata *MD);
+  static MetadataAsValue *getIfExists(LLVMContext &Context, Metadata *MD);
+  Metadata *getMetadata() const { return MD; }
+
   static bool classof(const Value *V) {
-    return V->getValueID() == GenericMDNodeVal ||
-           V->getValueID() == MDNodeFwdDeclVal ||
-           V->getValueID() == MDStringVal;
+    return V->getValueID() == MetadataAsValueVal;
+  }
+
+private:
+  void handleChangedMetadata(Metadata *MD);
+  void track();
+  void untrack();
+};
+
+/// \brief Shared implementation of use-lists for replaceable metadata.
+///
+/// Most metadata cannot be RAUW'ed.  This is a shared implementation of
+/// use-lists and associated API for the two that support it (\a ValueAsMetadata
+/// and \a TempMDNode).
+class ReplaceableMetadataImpl {
+  friend class MetadataTracking;
+
+public:
+  typedef MetadataTracking::OwnerTy OwnerTy;
+
+private:
+  SmallDenseMap<void *, OwnerTy, 4> UseMap;
+
+public:
+  ~ReplaceableMetadataImpl() {
+    assert(UseMap.empty() && "Cannot destroy in-use replaceable metadata");
+  }
+
+  /// \brief Replace all uses of this with MD.
+  ///
+  /// Replace all uses of this with \c MD, which is allowed to be null.
+  void replaceAllUsesWith(Metadata *MD);
+
+  /// \brief Resolve all uses of this.
+  ///
+  /// Resolve all uses of this, turning off RAUW permanently.  If \c
+  /// ResolveUsers, call \a GenericMDNode::resolve() on any users whose last
+  /// operand is resolved.
+  void resolveAllUses(bool ResolveUsers = true);
+
+private:
+  void addRef(void *Ref, OwnerTy Owner);
+  void dropRef(void *Ref);
+  void moveRef(void *Ref, void *New, const Metadata &MD);
+
+  static ReplaceableMetadataImpl *get(Metadata &MD);
+};
+
+/// \brief Value wrapper in the Metadata hierarchy.
+///
+/// This is a custom value handle that allows other metadata to refer to
+/// classes in the Value hierarchy.
+///
+/// Because of full uniquing support, each value is only wrapped by a single \a
+/// ValueAsMetadata object, so the lookup maps are far more efficient than
+/// those using ValueHandleBase.
+class ValueAsMetadata : public Metadata, ReplaceableMetadataImpl {
+  friend class ReplaceableMetadataImpl;
+  friend class LLVMContextImpl;
+
+  Value *V;
+
+protected:
+  ValueAsMetadata(LLVMContext &Context, unsigned ID, Value *V)
+      : Metadata(ID), V(V) {
+    assert(V && "Expected valid value");
+  }
+  ~ValueAsMetadata() {}
+
+public:
+  static ValueAsMetadata *get(Value *V);
+  static ConstantAsMetadata *getConstant(Value *C) {
+    return cast<ConstantAsMetadata>(get(C));
+  }
+  static LocalAsMetadata *getLocal(Value *Local) {
+    return cast<LocalAsMetadata>(get(Local));
+  }
+
+  static ValueAsMetadata *getIfExists(Value *V);
+  static ConstantAsMetadata *getConstantIfExists(Value *C) {
+    return cast_or_null<ConstantAsMetadata>(getIfExists(C));
+  }
+  static LocalAsMetadata *getLocalIfExists(Value *Local) {
+    return cast_or_null<LocalAsMetadata>(getIfExists(Local));
+  }
+
+  Value *getValue() const { return V; }
+  Type *getType() const { return V->getType(); }
+  LLVMContext &getContext() const { return V->getContext(); }
+
+  static void handleDeletion(Value *V);
+  static void handleRAUW(Value *From, Value *To);
+
+protected:
+  /// \brief Handle collisions after \a Value::replaceAllUsesWith().
+  ///
+  /// RAUW isn't supported directly for \a ValueAsMetadata, but if the wrapped
+  /// \a Value gets RAUW'ed and the target already exists, this is used to
+  /// merge the two metadata nodes.
+  void replaceAllUsesWith(Metadata *MD) {
+    ReplaceableMetadataImpl::replaceAllUsesWith(MD);
+  }
+
+public:
+  static bool classof(const Metadata *MD) {
+    return MD->getMetadataID() == LocalAsMetadataKind ||
+           MD->getMetadataID() == ConstantAsMetadataKind;
   }
 };
 
+class ConstantAsMetadata : public ValueAsMetadata {
+  friend class ValueAsMetadata;
+
+  ConstantAsMetadata(LLVMContext &Context, Constant *C)
+      : ValueAsMetadata(Context, ConstantAsMetadataKind, C) {}
+
+public:
+  static ConstantAsMetadata *get(Constant *C) {
+    return ValueAsMetadata::getConstant(C);
+  }
+  static ConstantAsMetadata *getIfExists(Constant *C) {
+    return ValueAsMetadata::getConstantIfExists(C);
+  }
+
+  Constant *getValue() const {
+    return cast<Constant>(ValueAsMetadata::getValue());
+  }
+
+  static bool classof(const Metadata *MD) {
+    return MD->getMetadataID() == ConstantAsMetadataKind;
+  }
+};
+
+class LocalAsMetadata : public ValueAsMetadata {
+  friend class ValueAsMetadata;
+
+  LocalAsMetadata(LLVMContext &Context, Value *Local)
+      : ValueAsMetadata(Context, LocalAsMetadataKind, Local) {
+    assert(!isa<Constant>(Local) && "Expected local value");
+  }
+
+public:
+  static LocalAsMetadata *get(Value *Local) {
+    return ValueAsMetadata::getLocal(Local);
+  }
+  static LocalAsMetadata *getIfExists(Value *Local) {
+    return ValueAsMetadata::getLocalIfExists(Local);
+  }
+
+  static bool classof(const Metadata *MD) {
+    return MD->getMetadataID() == LocalAsMetadataKind;
+  }
+};
+
+/// \brief Transitional API for extracting constants from Metadata.
+///
+/// This namespace contains transitional functions for metadata that points to
+/// \a Constants.
+///
+/// In prehistory -- when metadata was a subclass of \a Value -- \a MDNode
+/// operands could refer to any \a Value.  There's was a lot of code like this:
+///
+/// \code
+///     MDNode *N = ...;
+///     auto *CI = dyn_cast<ConstantInt>(N->getOperand(2));
+/// \endcode
+///
+/// Now that \a Value and \a Metadata are in separate hierarchies, maintaining
+/// the semantics for \a isa(), \a cast(), \a dyn_cast() (etc.) requires three
+/// steps: cast in the \a Metadata hierarchy, extraction of the \a Value, and
+/// cast in the \a Value hierarchy.  Besides creating boiler-plate, this
+/// requires subtle control flow changes.
+///
+/// The end-goal is to create a new type of metadata, called (e.g.) \a MDInt,
+/// so that metadata can refer to numbers without traversing a bridge to the \a
+/// Value hierarchy.  In this final state, the code above would look like this:
+///
+/// \code
+///     MDNode *N = ...;
+///     auto *MI = dyn_cast<MDInt>(N->getOperand(2));
+/// \endcode
+///
+/// The API in this namespace supports the transition.  \a MDInt doesn't exist
+/// yet, and even once it does, changing each metadata schema to use it is its
+/// own mini-project.  In the meantime this API prevents us from introducing
+/// complex and bug-prone control flow that will disappear in the end.  In
+/// particular, the above code looks like this:
+///
+/// \code
+///     MDNode *N = ...;
+///     auto *CI = mdconst::dyn_extract<ConstantInt>(N->getOperand(2));
+/// \endcode
+///
+/// The full set of provided functions includes:
+///
+///   mdconst::hasa                <=> isa
+///   mdconst::extract             <=> cast
+///   mdconst::extract_or_null     <=> cast_or_null
+///   mdconst::dyn_extract         <=> dyn_cast
+///   mdconst::dyn_extract_or_null <=> dyn_cast_or_null
+///
+/// The target of the cast must be a subclass of \a Constant.
+namespace mdconst {
+
+namespace detail {
+template <class T> T &make();
+template <class T, class Result> struct HasDereference {
+  typedef char Yes[1];
+  typedef char No[2];
+  template <size_t N> struct SFINAE {};
+
+  template <class U, class V>
+  static Yes &hasDereference(SFINAE<sizeof(static_cast<V>(*make<U>()))> * = 0);
+  template <class U, class V> static No &hasDereference(...);
+
+  static const bool value =
+      sizeof(hasDereference<T, Result>(nullptr)) == sizeof(Yes);
+};
+template <class V, class M> struct IsValidPointer {
+  static const bool value = std::is_base_of<Constant, V>::value &&
+                            HasDereference<M, const Metadata &>::value;
+};
+template <class V, class M> struct IsValidReference {
+  static const bool value = std::is_base_of<Constant, V>::value &&
+                            std::is_convertible<M, const Metadata &>::value;
+};
+} // end namespace detail
+
+/// \brief Check whether Metadata has a Value.
+///
+/// As an analogue to \a isa(), check whether \c MD has an \a Value inside of
+/// type \c X.
+template <class X, class Y>
+inline typename std::enable_if<detail::IsValidPointer<X, Y>::value, bool>::type
+hasa(Y &&MD) {
+  assert(MD && "Null pointer sent into hasa");
+  if (auto *V = dyn_cast<ConstantAsMetadata>(MD))
+    return isa<X>(V->getValue());
+  return false;
+}
+template <class X, class Y>
+inline
+    typename std::enable_if<detail::IsValidReference<X, Y &>::value, bool>::type
+    hasa(Y &MD) {
+  return hasa(&MD);
+}
+
+/// \brief Extract a Value from Metadata.
+///
+/// As an analogue to \a cast(), extract the \a Value subclass \c X from \c MD.
+template <class X, class Y>
+inline typename std::enable_if<detail::IsValidPointer<X, Y>::value, X *>::type
+extract(Y &&MD) {
+  return cast<X>(cast<ConstantAsMetadata>(MD)->getValue());
+}
+template <class X, class Y>
+inline
+    typename std::enable_if<detail::IsValidReference<X, Y &>::value, X *>::type
+    extract(Y &MD) {
+  return extract(&MD);
+}
+
+/// \brief Extract a Value from Metadata, allowing null.
+///
+/// As an analogue to \a cast_or_null(), extract the \a Value subclass \c X
+/// from \c MD, allowing \c MD to be null.
+template <class X, class Y>
+inline typename std::enable_if<detail::IsValidPointer<X, Y>::value, X *>::type
+extract_or_null(Y &&MD) {
+  if (auto *V = cast_or_null<ConstantAsMetadata>(MD))
+    return cast<X>(V->getValue());
+  return nullptr;
+}
+
+/// \brief Extract a Value from Metadata, if any.
+///
+/// As an analogue to \a dyn_cast_or_null(), extract the \a Value subclass \c X
+/// from \c MD, return null if \c MD doesn't contain a \a Value or if the \a
+/// Value it does contain is of the wrong subclass.
+template <class X, class Y>
+inline typename std::enable_if<detail::IsValidPointer<X, Y>::value, X *>::type
+dyn_extract(Y &&MD) {
+  if (auto *V = dyn_cast<ConstantAsMetadata>(MD))
+    return dyn_cast<X>(V->getValue());
+  return nullptr;
+}
+
+/// \brief Extract a Value from Metadata, if any, allowing null.
+///
+/// As an analogue to \a dyn_cast_or_null(), extract the \a Value subclass \c X
+/// from \c MD, return null if \c MD doesn't contain a \a Value or if the \a
+/// Value it does contain is of the wrong subclass, allowing \c MD to be null.
+template <class X, class Y>
+inline typename std::enable_if<detail::IsValidPointer<X, Y>::value, X *>::type
+dyn_extract_or_null(Y &&MD) {
+  if (auto *V = dyn_cast_or_null<ConstantAsMetadata>(MD))
+    return dyn_cast<X>(V->getValue());
+  return nullptr;
+}
+
+} // end namespace mdconst
+
 //===----------------------------------------------------------------------===//
 /// \brief A single uniqued string.
 ///
@@ -60,15 +430,13 @@ public:
 class MDString : public Metadata {
   friend class StringMapEntry<MDString>;
 
-  virtual void anchor();
   MDString(const MDString &) LLVM_DELETED_FUNCTION;
+  MDString &operator=(MDString &&) LLVM_DELETED_FUNCTION;
+  MDString &operator=(const MDString &) LLVM_DELETED_FUNCTION;
 
   StringMapEntry<MDString> *Entry;
-  explicit MDString(LLVMContext &Context)
-      : Metadata(Context, Value::MDStringVal), Entry(nullptr) {}
-
-  /// \brief Shadow Value::getName() to prevent its use.
-  StringRef getName() const LLVM_DELETED_FUNCTION;
+  MDString() : Metadata(MDStringKind), Entry(nullptr) {}
+  MDString(MDString &&) : Metadata(MDStringKind) {}
 
 public:
   static MDString *get(LLVMContext &Context, StringRef Str);
@@ -89,8 +457,8 @@ public:
   iterator end() const { return getString().end(); }
 
   /// \brief Methods for support type inquiry through isa, cast, and dyn_cast.
-  static bool classof(const Value *V) {
-    return V->getValueID() == MDStringVal;
+  static bool classof(const Metadata *MD) {
+    return MD->getMetadataID() == MDStringKind;
   }
 };
 
@@ -138,18 +506,80 @@ struct DenseMapInfo<AAMDNodes> {
   }
 };
 
-class MDNodeOperand;
+/// \brief Tracking metadata reference owned by Metadata.
+///
+/// Similar to \a TrackingMDRef, but it's expected to be owned by an instance
+/// of \a Metadata, which has the option of registering itself for callbacks to
+/// re-unique itself.
+///
+/// In particular, this is used by \a MDNode.
+class MDOperand {
+  MDOperand(MDOperand &&) LLVM_DELETED_FUNCTION;
+  MDOperand(const MDOperand &) LLVM_DELETED_FUNCTION;
+  MDOperand &operator=(MDOperand &&) LLVM_DELETED_FUNCTION;
+  MDOperand &operator=(const MDOperand &) LLVM_DELETED_FUNCTION;
+
+  Metadata *MD;
+
+public:
+  MDOperand() : MD(nullptr) {}
+  ~MDOperand() { untrack(); }
+
+  LLVM_EXPLICIT operator bool() const { return get(); }
+  Metadata *get() const { return MD; }
+  operator Metadata *() const { return get(); }
+  Metadata *operator->() const { return get(); }
+  Metadata &operator*() const { return *get(); }
+
+  void reset() {
+    untrack();
+    MD = nullptr;
+  }
+  void reset(Metadata *MD, Metadata *Owner) {
+    untrack();
+    this->MD = MD;
+    track(Owner);
+  }
+
+private:
+  void track(Metadata *Owner) {
+    if (MD) {
+      if (Owner)
+        MetadataTracking::track(this, *MD, *Owner);
+      else
+        MetadataTracking::track(MD);
+    }
+  }
+  void untrack() {
+    assert(static_cast<void *>(this) == &MD && "Expected same address");
+    if (MD)
+      MetadataTracking::untrack(MD);
+  }
+};
+
+template <> struct simplify_type<MDOperand> {
+  typedef Metadata *SimpleType;
+  static SimpleType getSimplifiedValue(MDOperand &MD) { return MD.get(); }
+};
+
+template <> struct simplify_type<const MDOperand> {
+  typedef Metadata *SimpleType;
+  static SimpleType getSimplifiedValue(const MDOperand &MD) { return MD.get(); }
+};
 
 //===----------------------------------------------------------------------===//
 /// \brief Tuple of metadata.
 class MDNode : public Metadata {
   MDNode(const MDNode &) LLVM_DELETED_FUNCTION;
   void operator=(const MDNode &) LLVM_DELETED_FUNCTION;
-  friend class MDNodeOperand;
-  friend class LLVMContextImpl;
   void *operator new(size_t) LLVM_DELETED_FUNCTION;
 
+  LLVMContext &Context;
+  unsigned NumOperands;
+
 protected:
+  unsigned MDNodeSubclassData;
+
   void *operator new(size_t Size, unsigned NumOps);
 
   /// \brief Required by std, but never called.
@@ -165,83 +595,83 @@ protected:
     llvm_unreachable("Constructor throws?");
   }
 
-  /// \brief Subclass data enums.
-  enum {
-    /// FunctionLocalBit - This bit is set if this MDNode is function local.
-    /// This is true when it (potentially transitively) contains a reference to
-    /// something in a function, like an argument, basicblock, or instruction.
-    FunctionLocalBit = 1 << 0,
+  MDNode(LLVMContext &Context, unsigned ID, ArrayRef<Metadata *> MDs);
+  ~MDNode() { dropAllReferences(); }
 
-    /// NotUniquedBit - This is set on MDNodes that are not uniqued because they
-    /// have a null operand.
-    NotUniquedBit    = 1 << 1
-  };
+  void dropAllReferences();
+  void storeDistinctInContext();
 
-  /// \brief FunctionLocal enums.
-  enum FunctionLocalness {
-    FL_Unknown = -1,
-    FL_No = 0,
-    FL_Yes = 1
-  };
+  static MDNode *getMDNode(LLVMContext &C, ArrayRef<Metadata *> MDs,
+                           bool Insert = true);
 
-  /// \brief Replace each instance of the given operand with a new value.
-  void replaceOperand(MDNodeOperand *Op, Value *NewVal);
+  MDOperand *mutable_begin() { return mutable_end() - NumOperands; }
+  MDOperand *mutable_end() { return reinterpret_cast<MDOperand *>(this); }
 
-  MDNode(LLVMContext &C, unsigned ID, ArrayRef<Value *> Vals,
-         bool isFunctionLocal);
-  ~MDNode() {}
-
-  static MDNode *getMDNode(LLVMContext &C, ArrayRef<Value*> Vals,
-                           FunctionLocalness FL, bool Insert = true);
 public:
-  static MDNode *get(LLVMContext &Context, ArrayRef<Value*> Vals);
-  /// \brief Construct MDNode with an explicit function-localness.
-  ///
-  /// Don't analyze Vals; trust isFunctionLocal.
+  static MDNode *get(LLVMContext &Context, ArrayRef<Metadata *> MDs) {
+    return getMDNode(Context, MDs, true);
+  }
   static MDNode *getWhenValsUnresolved(LLVMContext &Context,
-                                       ArrayRef<Value*> Vals,
-                                       bool isFunctionLocal);
+                                       ArrayRef<Metadata *> MDs) {
+    // TODO: Remove this.
+    return get(Context, MDs);
+  }
 
-  static MDNode *getIfExists(LLVMContext &Context, ArrayRef<Value*> Vals);
+  static MDNode *getIfExists(LLVMContext &Context, ArrayRef<Metadata *> MDs) {
+    return getMDNode(Context, MDs, false);
+  }
 
   /// \brief Return a temporary MDNode
   ///
   /// For use in constructing cyclic MDNode structures. A temporary MDNode is
   /// not uniqued, may be RAUW'd, and must be manually deleted with
   /// deleteTemporary.
-  static MDNode *getTemporary(LLVMContext &Context, ArrayRef<Value*> Vals);
+  static MDNodeFwdDecl *getTemporary(LLVMContext &Context,
+                                     ArrayRef<Metadata *> MDs);
 
   /// \brief Deallocate a node created by getTemporary.
   ///
   /// The node must not have any users.
   static void deleteTemporary(MDNode *N);
 
-  /// \brief Replace a specific operand.
-  void replaceOperandWith(unsigned i, Value *NewVal);
+  LLVMContext &getContext() const { return Context; }
 
-  /// \brief Return specified operand.
-  Value *getOperand(unsigned i) const LLVM_READONLY;
+  /// \brief Replace a specific operand.
+  void replaceOperandWith(unsigned I, Metadata *New);
+
+  /// \brief Check if node is fully resolved.
+  bool isResolved() const;
+
+protected:
+  /// \brief Set an operand.
+  ///
+  /// Sets the operand directly, without worrying about uniquing.
+  void setOperand(unsigned I, Metadata *New);
+
+public:
+  typedef const MDOperand *op_iterator;
+  typedef iterator_range<op_iterator> op_range;
+
+  op_iterator op_begin() const {
+    return const_cast<MDNode *>(this)->mutable_begin();
+  }
+  op_iterator op_end() const {
+    return const_cast<MDNode *>(this)->mutable_end();
+  }
+  op_range operands() const { return op_range(op_begin(), op_end()); }
+
+  const MDOperand &getOperand(unsigned I) const {
+    assert(I < NumOperands && "Out of range");
+    return op_begin()[I];
+  }
 
   /// \brief Return number of MDNode operands.
   unsigned getNumOperands() const { return NumOperands; }
 
-  /// \brief Return whether MDNode is local to a function.
-  bool isFunctionLocal() const {
-    return (getSubclassDataFromValue() & FunctionLocalBit) != 0;
-  }
-
-  /// \brief Return the first function-local operand's function.
-  ///
-  /// If this metadata is function-local and recursively has a function-local
-  /// operand, return the first such operand's parent function.  Otherwise,
-  /// return null. getFunction() should not be used for performance- critical
-  /// code because it recursively visits all the MDNode's operands.
-  const Function *getFunction() const;
-
   /// \brief Methods for support type inquiry through isa, cast, and dyn_cast:
-  static bool classof(const Value *V) {
-    return V->getValueID() == GenericMDNodeVal ||
-           V->getValueID() == MDNodeFwdDeclVal;
+  static bool classof(const Metadata *MD) {
+    return MD->getMetadataID() == GenericMDNodeKind ||
+           MD->getMetadataID() == MDNodeFwdDeclKind;
   }
 
   /// \brief Check whether MDNode is a vtable access.
@@ -254,18 +684,6 @@ public:
   static AAMDNodes getMostGenericAA(const AAMDNodes &A, const AAMDNodes &B);
   static MDNode *getMostGenericFPMath(MDNode *A, MDNode *B);
   static MDNode *getMostGenericRange(MDNode *A, MDNode *B);
-
-protected:
-  bool isNotUniqued() const {
-    return (getSubclassDataFromValue() & NotUniquedBit) != 0;
-  }
-  void setIsNotUniqued();
-
-  // Shadow Value::setValueSubclassData with a private forwarding method so that
-  // any future subclasses cannot accidentally use it.
-  void setValueSubclassData(unsigned short D) {
-    Value::setValueSubclassData(D);
-  }
 };
 
 /// \brief Generic metadata node.
@@ -279,24 +697,59 @@ protected:
 /// TODO: Make uniquing opt-out (status: mandatory, sometimes dropped).
 /// TODO: Drop support for RAUW.
 class GenericMDNode : public MDNode {
+  friend class Metadata;
   friend class MDNode;
   friend class LLVMContextImpl;
+  friend class ReplaceableMetadataImpl;
 
-  unsigned Hash;
+  /// \brief Support RAUW as long as one of its arguments is replaceable.
+  ///
+  /// If an operand is an \a MDNodeFwdDecl (or a replaceable \a GenericMDNode),
+  /// support RAUW to support uniquing as forward declarations are resolved.
+  /// As soon as operands have been resolved, drop support.
+  ///
+  /// FIXME: Save memory by storing this in a pointer union with the
+  /// LLVMContext, and adding an LLVMContext reference to RMI.
+  std::unique_ptr<ReplaceableMetadataImpl> ReplaceableUses;
 
-  GenericMDNode(LLVMContext &C, ArrayRef<Value *> Vals, bool isFunctionLocal)
-      : MDNode(C, GenericMDNodeVal, Vals, isFunctionLocal), Hash(0) {}
+  GenericMDNode(LLVMContext &C, ArrayRef<Metadata *> Vals);
   ~GenericMDNode();
 
-  void dropAllReferences();
+  void setHash(unsigned Hash) { MDNodeSubclassData = Hash; }
 
 public:
   /// \brief Get the hash, if any.
-  unsigned getHash() const { return Hash; }
+  unsigned getHash() const { return MDNodeSubclassData; }
 
-  static bool classof(const Value *V) {
-    return V->getValueID() == GenericMDNodeVal;
+  static bool classof(const Metadata *MD) {
+    return MD->getMetadataID() == GenericMDNodeKind;
   }
+
+  /// \brief Check whether any operands are forward declarations.
+  ///
+  /// Returns \c true as long as any operands (or their operands, etc.) are \a
+  /// MDNodeFwdDecl.
+  ///
+  /// As forward declarations are resolved, their containers should get
+  /// resolved automatically.  However, if this (or one of its operands) is
+  /// involved in a cycle, \a resolveCycles() needs to be called explicitly.
+  bool isResolved() const { return !ReplaceableUses; }
+
+  /// \brief Resolve cycles.
+  ///
+  /// Once all forward declarations have been resolved, force cycles to be
+  /// resolved.
+  ///
+  /// \pre No operands (or operands' operands, etc.) are \a MDNodeFwdDecl.
+  void resolveCycles();
+
+private:
+  void handleChangedOperand(void *Ref, Metadata *New);
+
+  bool hasUnresolvedOperands() const { return SubclassData32; }
+  void incrementUnresolvedOperands() { ++SubclassData32; }
+  void decrementUnresolvedOperands() { --SubclassData32; }
+  void resolve();
 };
 
 /// \brief Forward declaration of metadata.
@@ -304,17 +757,21 @@ public:
 /// Forward declaration of metadata, in the form of a metadata node.  Unlike \a
 /// GenericMDNode, this class has support for RAUW and is suitable for forward
 /// references.
-class MDNodeFwdDecl : public MDNode {
+class MDNodeFwdDecl : public MDNode, ReplaceableMetadataImpl {
+  friend class Metadata;
   friend class MDNode;
+  friend class ReplaceableMetadataImpl;
 
-  MDNodeFwdDecl(LLVMContext &C, ArrayRef<Value *> Vals, bool isFunctionLocal)
-      : MDNode(C, MDNodeFwdDeclVal, Vals, isFunctionLocal) {}
+  MDNodeFwdDecl(LLVMContext &C, ArrayRef<Metadata *> Vals)
+      : MDNode(C, MDNodeFwdDeclKind, Vals) {}
   ~MDNodeFwdDecl() {}
 
 public:
-  static bool classof(const Value *V) {
-    return V->getValueID() == MDNodeFwdDeclVal;
+  static bool classof(const Metadata *MD) {
+    return MD->getMetadataID() == MDNodeFwdDeclKind;
   }
+
+  using ReplaceableMetadataImpl::replaceAllUsesWith;
 };
 
 //===----------------------------------------------------------------------===//
@@ -333,7 +790,7 @@ class NamedMDNode : public ilist_node<NamedMDNode> {
 
   std::string Name;
   Module *Parent;
-  void *Operands; // SmallVector<TrackingVH<MDNode>, 4>
+  void *Operands; // SmallVector<TrackingMDRef, 4>
 
   void setParent(Module *M) { Parent = M; }