From 1ab929cbd525118635b0944d257b22d4f236fd9e Mon Sep 17 00:00:00 2001 From: Reid Spencer Date: Mon, 5 Jul 2004 08:18:07 +0000 Subject: [PATCH] - Reorganized the document contents - Provided a "General Layout" section that currently covers just the block structure of the bytecode file. - Wrote the section on the Global Type Pool - Wrote the section on differences between LLVM file format versions. Only five sections left to write! git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@14628 91177308-0d34-0410-b5e6-96231b3b80d8 --- docs/BytecodeFormat.html | 691 +++++++++++++++++++++++++++------------ 1 file changed, 476 insertions(+), 215 deletions(-) diff --git a/docs/BytecodeFormat.html b/docs/BytecodeFormat.html index 0b0f15606d2..ab93c1a4386 100644 --- a/docs/BytecodeFormat.html +++ b/docs/BytecodeFormat.html @@ -4,24 +4,31 @@ LLVM Bytecode File Format -
LLVM Bytecode File Format
  1. Abstract
    2. -
  2. General Concepts +
  3. Concepts
    1. Blocks
    2. Lists
    3. Fields
      4. -
    4. Slots
      5. -
    5. Encoding Rules
    6. Alignment
      7. +
    7. Encoding Primitives
      8. +
    8. Slots
      9. +
    +
    4. +
  4. General Layout +
      +
    1. Structure
  5. Detailed Layout @@ -30,11 +37,13 @@
  6. Blocks Types
  7. Signature Block
  8. Module Block
    9. -
  9. Global Type Pool
    10. -
  10. Module Info Block
    11. -
  11. Global Constant Pool
    12. -
  12. Function Blocks
    13. -
  13. Module Symbol Table
    14. +
  14. Global Type Pool
    15. +
  15. Module Info Block
    16. +
  16. Global Constant Pool
    17. +
  17. Function Definition
    18. +
  18. Compaction Table
    19. +
  19. Instruction List
    20. +
  20. Symbol Table
  • Version Differences @@ -52,19 +61,21 @@

    Warning: This is a work in progress.

    +
    Abstract
    -

    This document describes the LLVM bytecode -file format. It specifies the binary encoding rules of the bytecode file format +

    This document describes the LLVM bytecode file format as of version 1.3. +It specifies the binary encoding rules of the bytecode file format so that equivalent systems can encode bytecode files correctly. The LLVM bytecode representation is used to store the intermediate representation on disk in compacted form.

    + -
    General Concepts
    +
    Concepts

    This section describes the general concepts of the bytecode file format @@ -73,48 +84,42 @@ format may change in the future, but will always be backwards compatible with older formats. This document only describes the most current version of the bytecode format.

    +
    Blocks

    LLVM bytecode files consist simply of a sequence of blocks of bytes. -Each block begins with an identification value that determines the type of -the next block. The possible types of blocks are described below in the section -Block Types. The block identifier is used because -it is possible for entire blocks to be omitted from the file if they are -empty. The block identifier helps the reader determine which kind of block is -next in the file.

    -

    The following block identifiers are currently in use -(from llvm/Bytecode/Format.h):

    -
      -
    1. Module (0x01).
      2. -
    2. Function (0x11).
      3. -
    3. ConstantPool (0x12).
      4. -
    4. SymbolTable (0x13).
      5. -
    5. ModuleGlobalInfo (0x14).
      6. -
    6. GlobalTypePlane (0x15).
      7. -
    7. BasicBlock (0x31).
      8. -
    8. InstructionList (0x32).
      9. -
    9. CompactionTable (0x33).
      10. -
    +Each block begins with an header of two unsigned integers. The first value +identifies the type of block and the second value provides the size of the +block in bytes. The block identifier is used because it is possible for entire +blocks to be omitted from the file if they are empty. The block identifier helps +the reader determine which kind of block is next in the file. Note that blocks +can be nested within other blocks.

    All blocks are variable length, and the block header specifies the size of -the block. All blocks are rounded aligned to even 32-bit boundaries, so they -always start and end of this boundary. Each block begins with an integer -identifier and the length of the block, which does not include the padding -bytes needed for alignment.

    +the block. All blocks begin on a byte index that is aligned to an even 32-bit +boundary. That is, the first block is 32-bit aligned because it starts at offset +0. Each block is padded with zero fill bytes to ensure that the next block also +starts on a 32-bit boundary.

    +
    Lists
    -

    Most blocks are constructed of lists of information. Lists can be constructed -of other lists, etc. This decomposition of information follows the containment -hierarchy of the LLVM Intermediate Representation. For example, a function -contains a list of instructions (the terminator instructions implicitly define -the end of the basic blocks).

    -

    A list is encoded into the file simply by encoding the number of entries as -an integer followed by each of the entries. The reader knows when the list is -done because it will have filled the list with the required numbe of entries. -

    +

    LLVM Bytecode blocks often contain lists of things of a similar type. For + example, a function contains a list of instructions and a function type + contains a list of argument types. There are two basic types of lists: + length lists, and null terminated lists, as described here:

    +
      +
    • Length Lists. Length lists are simply preceded by the number + of items in the list. The bytecode reader will read the count first and + then iterate that many times to read in the list contents.
      • +
    • Null Terminated Lists. For some lists, the number of elements + in the list is not readily available at the time of writing the bytecode. + In these cases, the list is terminated by some null value. What constitutes + a null value differs, but it almost always boils down to a zero value.
      • +
    +
    Fields
    @@ -129,46 +134,16 @@ they must contain. As few bits as possible are written for each field. The sections that follow will provide the details on how these fields are written and how the bits are to be interpreted.

    + -
    Slots
    +
    Alignment
    -

    The bytecode format uses the notion of a "slot" to reference Types and -Values. Since the bytecode file is a direct representation of LLVM's -intermediate representation, there is a need to represent pointers in the file. -Slots are used for this purpose. For example, if one has the following assembly: -

    - -
    - %MyType = type { int, sbyte }
    - %MyVar = external global %MyType +

    To support cross-platform differences, the bytecode file is aligned on + certain boundaries. This means that a small amount of padding (at most 3 + bytes) will be added to ensure that the next entry is aligned to a 32-bit + boundary.

    -

    there are two definitions. The definition of %MyVar uses -%MyType. In the C++ IR this linkage between %MyVar and -%MyType is -explicit through the use of C++ pointers. In bytecode, however, there's no -ability to store memory addresses. Instead, we compute and write out slot -numbers for every type and Value written to the file.

    -

    A slot number is simply an unsigned 32-bit integer encoded in the variable -bit rate scheme (see encoding below). This ensures that -low slot numbers are encoded in one byte. Through various bits of magic LLVM -attempts to always keep the slot numbers low. The first attempt is to associate -slot numbers with their "type plane". That is, Values of the same type are -written to the bytecode file in a list (sequentially). Their order in that list -determines their slot number. This means that slot #1 doesn't mean anything -unless you also specify for which type you want slot #1. Types are handled -specially and are always written to the file first (in the Global Type Pool) and -in such a way that both forward and backward references of the types can often be -resolved with a single pass through the type pool.

    -

    Slot numbers are also kept small by rearranging their order. Because of the -structure of LLVM, certain values are much more likely to be used frequently -in the body of a function. For this reason, a compaction table is provided in -the body of a function if its use would make the function body smaller. -Suppose you have a function body that uses just the types "int*" and "{double}" -but uses them thousands of time. Its worthwhile to ensure that the slot number -for these types are low so they can be encoded in a single byte (via vbr). -This is exactly what the compaction table does.

    -
    Encoding Primitives
    @@ -219,80 +194,196 @@ variable bit rate encoding as described above.

    - + - - + - - + - - + - - + - - + + - - + + - - + - - + + + +
    TypeRuleRule
    unsignedA 32-bit unsigned integer that always occupies four + unsignedA 32-bit unsigned integer that always occupies four consecutive bytes. The unsigned integer is encoded using LSB first ordering. That is bits 20 through 27 are in the byte with the lowest file offset (little endian).
    uint_vbrA 32-bit unsigned integer that occupies from one to five + uint_vbrA 32-bit unsigned integer that occupies from one to five bytes using variable bit rate encoding.
    uint64_vbrA 64-bit unsigned integer that occupies from one to ten + uint64_vbrA 64-bit unsigned integer that occupies from one to ten bytes using variable bit rate encoding.
    int64_vbrA 64-bit signed integer that occupies from one to ten + int64_vbrA 64-bit signed integer that occupies from one to ten bytes using the signed variable bit rate encoding.
    charA single unsigned character encoded into one bytecharA single unsigned character encoded into one byte
    bitA single bit within a byte.bitA single bit within some larger integer field.
    stringA uint_vbr indicating the length of the character string + stringA uint_vbr indicating the length of the character string immediately followed by the characters of the string. There is no terminating null byte in the string.
    dataAn arbitrarily long segment of data to which no + dataAn arbitrarily long segment of data to which no interpretation is implied. This is used for float, double, and constant initializers.
    blockA block of data that is logically related. A block + begins with an unsigned that provides the block + identifier (constant value) and an unsigned that + provides the length of the block. Blocks may compose other blocks. +
    + -
    Alignment
    +
    Slots
    -

    To support cross-platform differences, the bytecode file is aligned on -certain boundaries. This means that a small amount of padding (at most 3 bytes) -will be added to ensure that the next entry is aligned to a 32-bit boundary. +

    The bytecode format uses the notion of a "slot" to reference Types and +Values. Since the bytecode file is a direct representation of LLVM's +intermediate representation, there is a need to represent pointers in the file. +Slots are used for this purpose. For example, if one has the following assembly: +

    +
    + %MyType = type { int, sbyte }
    + %MyVar = external global %MyType +
    +

    there are two definitions. The definition of %MyVar uses +%MyType. In the C++ IR this linkage between %MyVar and +%MyType is +explicit through the use of C++ pointers. In bytecode, however, there's no +ability to store memory addresses. Instead, we compute and write out slot +numbers for every Type and Value written to the file.

    +

    A slot number is simply an unsigned 32-bit integer encoded in the variable +bit rate scheme (see encoding). This ensures that +low slot numbers are encoded in one byte. Through various bits of magic LLVM +attempts to always keep the slot numbers low. The first attempt is to associate +slot numbers with their "type plane". That is, Values of the same type are +written to the bytecode file in a list (sequentially). Their order in that list +determines their slot number. This means that slot #1 doesn't mean anything +unless you also specify for which type you want slot #1. Types are handled +specially and are always written to the file first (in the +Global Type Pool) and +in such a way that both forward and backward references of the types can often be +resolved with a single pass through the type pool.

    +

    Slot numbers are also kept small by rearranging their order. Because of the +structure of LLVM, certain values are much more likely to be used frequently +in the body of a function. For this reason, a compaction table is provided in +the body of a function if its use would make the function body smaller. +Suppose you have a function body that uses just the types "int*" and "{double}" +but uses them thousands of time. Its worthwhile to ensure that the slot number +for these types are low so they can be encoded in a single byte (via vbr). +This is exactly what the compaction table does.

    +
    + + +
    General Layout
    + +
    +

    This section provides the general layout of the LLVM bytecode file format. + The detailed layout can be found in the next section.

    + + +
    Structure
    +
    +

    The bytecode file format requires blocks to be in a certain order and +nested in a particular way so that an LLVM module can be constructed +efficiently from the contents of the file. This ordering defines a general +structure for bytecode files as shown below. The table below shows the order +in which all block types may appear. Please note that some of the blocks are +optional and some may be repeated. The structure is fairly loose because +optional blocks, if empty, are completely omitted from the file. +

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    IDParentOptional?Repeated?LevelBlock Type
    N/AFileNoNo0Signature
    0x01FileNoNo0Module
    0x15ModuleNoNo1    + Global Type Pool
    0x14ModuleNoNo1    + Module Globals Info
    0x12ModuleYesNo1    + Module Constant Pool
    0x11ModuleYesYes1    + Function Definitions
    0x12FunctionYesNo2       + Function Constant Pool
    0x33FunctionYesNo2       + Compaction Table
    0x32FunctionNoNo2       + Instruction List
    0x13FunctionYesNo2       + Function Symbol Table
    0x13ModuleYesNo1    + Module Symbol Table
    +

    Use the links in the table or see Block Types for +details about the contents of each of the block types.

    +
    +
    Detailed Layout

    This section provides the detailed layout of the LLVM bytecode file format. -bit and byte level specifics.

    +

    Notation
    -

    The descriptions of the bytecode format that follow describe the bit - fields in detail. These descriptions are provided in tabular form. Each table - has four columns that specify:

    -
      -
    1. Byte(s): The offset in bytes of the field from the start of - its container (block, list, other field).
      2. -
    2. Bit(s): The offset in bits of the field from the start of - the byte field. Bits are always little endian. That is, bit addresses with - smaller values have smaller address (i.e. 20 is at bit 0, - 21 at 1, etc.) -
      3. -
    3. Align?: Indicates if this field is aligned to 32 bits or not. - This indicates where the next field starts, always on a 32 bit - boundary.
      4. -
    4. Type: The basic type of information contained in the field.
      5. -
    5. Description: Describes the contents of the field.
      6. -
    +

    The descriptions of the bytecode format that follow describe the order, type +and bit fields in detail. These descriptions are provided in tabular form. +Each table has four columns that specify:

    +
      +
    1. Byte(s): The offset in bytes of the field from the start of + its container (block, list, other field).
      2. +
    2. Bit(s): The offset in bits of the field from the start of + the byte field. Bits are always little endian. That is, bit addresses with + smaller values have smaller address (i.e. 20 is at bit 0, + 21 at 1, etc.) +
      3. +
    3. Align?: Indicates if this field is aligned to 32 bits or not. + This indicates where the next field starts, always on a 32 bit + boundary.
      4. +
    4. Type: The basic type of information contained in the field.
      5. +
    5. Description: Describes the contents of the field.
      6. +
    Block Types
    @@ -330,90 +421,109 @@ other blocks because there is no identifier and no block length at the start of the block. Essentially, this block is just the "magic number" for the file. - - - - + - - + + - - + + - - + + - - + +
    Byte(s)Bit(s)Align? TypeField DescriptionField Description
    00-NocharConstant "l" (0x6C)charConstant "l" (0x6C)
    01-NocharConstant "l" (0x6C)charConstant "l" (0x6C)
    02-NocharConstant "v" (0x76)charConstant "v" (0x76)
    03-NocharConstant "m" (0x6D)charConstant "m" (0x6D)
    +
    Module Block

    The module block contains a small pre-amble and all the other blocks in -the file. Of particular note, the bytecode format number is simply a 28-bit -monotonically increase integer that identifiers the version of the bytecode +the file. The table below shows the structure of the module block. Note that it +only provides the module identifier, size of the module block, and the format +information. Everything else is contained in other blocks, described in other +sections.

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    TypeField Description
    unsignedModule Identifier (0x01)
    unsignedSize of the module block in bytes
    uint32_vbrFormat Information
    blockGlobal Type Pool
    blockModule Globals Info
    blockModule Constant Pool
    blockFunction Definitions
    blockModule Symbol Table
    +
    + + +
    Format Information
    +
    +

    The format information field is encoded into a 32-bit vbr-encoded unsigned +integer as shown in the following table.

    + + + + + + + + + + + + + + + + + + + + + +
    Bit(s)TypeDescription
    0bitBig Endian?
    1bitPointers Are 64-bit?
    2bitHas No Endianess?
    3bitHas No Pointer Size?
    4-31bitBytecode Format Version
    +

    +Of particular note, the bytecode format number is simply a 28-bit +monotonically increase integer that identifies the version of the bytecode format (which is not directly related to the LLVM release number). The bytecode versions defined so far are (note that this document only describes -the latest version):

    - +the latest version, 1.3):

    • #0: LLVM 1.0 & 1.1
    • #1: LLVM 1.2
    • #2: LLVM 1.3
    - -

    The table below shows the format of the module block header. It is defined -by blocks described in other sections.

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    Byte(s)Bit(s)Align?TypeField Description
    04-07-NounsignedModule Identifier (0x01)
    08-11-NounsignedSize of the module block in bytes
    12-1500Yesuint32_vbrFormat Information
    ''0-bitBig Endian?
    ''1-bitPointers Are 64-bit?
    ''2-bitHas No Endianess?
    ''3-bitHas No Pointer Size?
    ''4-31-bitBytecode Format Version
    16-end--blocksThe remaining bytes in the block consist - solely of other block types in sequence.
    -

    Note that we plan to eventually expand the target description capabilities -of bytecode files to target -triples.

    - +of bytecode files to target triples. +

    -
    Global Type Pool
    +
    Global Type Pool

    The global type pool consists of type definitions. Their order of appearance in the file determines their slot number (0 based). Slot numbers are used to @@ -423,48 +533,161 @@ Since all values have types and are associated with the order in which the type pool is written, the global type pool must be written as the first block of a module. If it is not, attempts to read the file will fail because both forward and backward type resolution will not be possible.

    -

    The type pool is simply a list of types definitions, as shown in the table +

    The type pool is simply a list of type definitions, as shown in the table below.

    - - - - + - - + + - - + + - - + + - - + + - - - -
    Byte(s)Bit(s)Align? TypeField DescriptionField Description
    00-03-NounsignedType Pool Identifier (0x13)unsignedType Pool Identifier (0x13)
    04-07-NounsignedSize in bytes of the symbol table block.unsignedSize in bytes of the symbol table block.
    08-111-Nouint32_vbrNumber of entries in type planeuint32_vbrNumber of entries in type plane
    12-151-Nouint32_vbrType plane index for following entriestypeEach of the type definitions (see below)1
    16-end1,2-NotypeEach of the type definitions.
    1Maximum length shown, - may be smaller
    2Repeated field. +     		+ 1Repeated field.
    +
    -
    Module Info
    +
    Type Definitions
    +
    +

    Types in the type pool are defined using a different format for each +basic type of type as given in the following sections.

    +

    Primitive Types

    +

    The primitive types encompass the basic integer and floating point types

    + + + + + + + + + + +
    TypeDescription
    uint32_vbrType ID For The Primitive (1-11)1
    + 1See the definition of Type::TypeID in Type.h for the numeric + equivalents of the primitive type ids.
    +
    +

    Function Types

    + + + + + + + + + + + + + + + + + + + + + + +
    TypeDescription
    uint32_vbrType ID for function types (13)
    uint32_vbrSlot number of function's return type.
    uint32_vbrThe number of arguments in the function.
    uint32_vbrSlot number of each argument's type.1
    uint32_vbrValue 0 if this is a varargs function.2
    + 1Repeated field.
    + 2Optional field. +
    +

    Structure Types

    + + + + + + + + + + + + + + + + +
    TypeDescription
    uint32_vbrType ID for structure types (14)
    uint32_vbrSlot number of each of the element's fields.1
    uint32_vbrNull Terminator (VoidTy type id)
    + 1Repeated field.
    +
    +

    Array Types

    + + + + + + + + + + + + + + +
    TypeDescription
    uint32_vbrType ID for Array Types (15)
    uint32_vbrSlot number of array's element type.
    uint32_vbrThe number of elements in the array.
    +

    Pointer Types

    + + + + + + + + + + + +
    TypeDescription
    uint32_vbrType ID For Pointer Types (16)
    uint32_vbrSlot number of pointer's element type.
    +

    Opaque Types

    + + + + + + + + +
    TypeDescription
    uint32_vbrType ID For Opaque Types (17)
    +
    + +
    Module Global Info

    To be determined.

    -
    Constants
    +
    Constant Pool

    To be determined.

    -
    Functions
    +
    Function Definition
    +
    +

    To be determined.

    +
    + +
    Compaction Table
    +
    +

    To be determined.

    +
    + +
    Instruction List

    To be determined.

    @@ -483,28 +706,28 @@ format is given in the table below.

    Bit(s) Align? Type - Field Description + Field Description 00-03-Nounsigned - Symbol Table Identifier (0x13) + Symbol Table Identifier (0x13) 04-07-Nounsigned - Size in bytes of the symbol table block. + Size in bytes of the symbol table block. 08-111-Nouint32_vbr - Number of entries in type plane + Number of entries in type plane 12-151-Nouint32_vbr - Type plane index for following entries + Type plane index for following entries 16-191,2-Nouint32_vbr - Slot number of a value. + Slot number of a value. variable1,2-Nostring - Name of the value in the symbol table. + Name of the value in the symbol table. - 1Maximum length shown, + 1Maximum length shown, may be smaller
    2Repeated field. @@ -522,22 +745,60 @@ describes the differences between that version and the one that follows
    Version 1.2 Differences From 1.3
    + +
    Type Derives From Value
    -

    TBD: How version 1.2 differs from version 1.3

    +

    In version 1.2, the Type class in the LLVM IR derives from the Value class. + This is not the case in version 1.3. Consequently, in version 1.2 the notion + of a "Type Type" was used to write out values that were Types. The types + always occuped plane 12 (corresponding to the TypeTyID) of any type planed + set of values. In 1.3 this representation is not convenient because the + TypeTyID (12) is not present and its value is now used for LabelTyID. + Consequently, the data structures written that involve types do so by writing + all the types first and then each of the value planes according to those + types. In version 1.2, the types would have been written intermingled with + the values.

    +
    + + +
    Restricted getelementptr Types
    +
    +

    In version 1.2, the getelementptr instruction required a ubyte type index + for accessing a structure field and a long type index for accessing an array + element. Consequently, it was only possible to access structures of 255 or + fewer elements. Starting in version 1.3, this restriction was lifted. + Structures must now be indexed with int or uint types. Arrays must now be + indexed with long or ulong types. This requirement was needed so that LLVM + could compile several test cases that used large numbers of fields in their + structures. The consequence of this was that the bytecode format had to + change in order to accommodate the larger range of structure indices.

    Version 1.1 Differences From 1.2
    + +
    Explicit Primitive Zeros
    -

    TBD: How version 1.1 differs from version 1.2

    +

    In version 1.1, the zero value for primitives was explicitly encoded into + the bytecode format. Since these zero values are constant values in the + LLVM IR and never change, there is no reason to explicitly encode them. This + explicit encoding was removed in version 1.2.

    +
    + + +
    Inconsistent Module Global Info
    +
    +

    In version 1.1, the Module Global Info block was not aligned causing the + next block to be read in on an unaligned boundary. This problem was corrected + in version 1.2.

    Version 1.0 Differences From 1.1
    -

    TBD: How version 1.0 differs from version 1.1

    +

    None. Version 1.0 and 1.1 bytecode formats are identical.