prog8/docs/source/structpointers.rst

.. _pointers:

********************
Structs and Pointers
********************

.. index::
      pair: Structs; Overview
      pair: Pointers; Overview

.. attention::
    The 6502 cpu lacks some features (addressing modes, registers) to make pointers work efficiently.
    Also it requires that pointer variables have to be in zero page, or copied to a temporary zero page variable,
    before they can even be used as a pointer. This means that pointer operations in prog8 compile
    to rather large and inefficient assembly code most of the time, when compared to direct array access or regular variables.
    At least try to place heavily used pointer variables in zero page using ``@requirezp`` on their declaration,
    if zero page space allows.
    Pointer variables that don't have a zeropage tag specified will be treated as having ``@zp`` so they get
    priority over other variables to be placed into zeropage.

.. note::
    Indexed field access through a pointer works for reading and writing,
    with either implicit or explicit pointer dereferencing::

        ^^Node  np
        np[2].field = 9999
        np[2]^^.field = 9999
        ubyte value = np[2].field

    This also works when the struct field itself is a pointer or array that you index::

        struct Node { ^^uword s }
        ^^Node np
        np[2]^^.s[0] = 9999
        uword value = np[2]^^.s[0]

    To save a repeated pointer calculation when assigning multiple fields,
    break up the expression in multiple steps::

        ^^Node thenode = &&np[2]
        thenode.field = 9999

    Writing through a parenthesized casting expression or a function call does work however;
    the compiler evaluates the expression once and stores through the computed address::

        (np[2] as ^^Node).field = 9999
        get_node(i).field = 9999


Untyped pointers and the ``pointer`` type
-----------------------------------------
.. index:: single: Pointers; untyped

Prior to version 12 of the language, the only pointer type available was a plain ``uword`` value (the memory address)
which could be used as a pointer to an ``ubyte`` (the byte value at that memory address).
Array indexing on an ``uword`` simply means to point to the ``ubyte`` at the location of the address + index value.

When the address of a value (explicitly) or a value of a reference type (string, array) was passed as an argument to a subroutine call,
it became one of these plain ``uword`` 'pointers'. The subroutine receiving it always had to interpret the 'pointer'
explicitly for what it actually pointed to, if that wasn't a simple byte.

Some implicit conversions were allowed too (such as putting ``str`` as the type of a subroutine parameter,
which would be changed to ``uword`` by the compiler).

Since version 12.6 there now is a dedicated ``pointer`` type keyword that serves as an untyped address
and automatically takes the :ref:`pointer size <pointer_size>` for the target.
This is useful when writing code that is compiled for different targets,
because you don't need to manually switch between ``uword`` and ``long`` via conditional compilation::

    pointer ptr = $a000
    ; same program works on both 6502 and m68k targets
    ; ptr is 2 bytes on C64, CX16, etc.
    ; ptr is 4 bytes on qemu68k

You can assign any other pointer type to an untyped pointer variable (``uword``, ``long``` or ``pointer``) without the need for an explicit cast.
You can assign an untyped pointer to a typed pointer variable without the need for an explicit cast.
Prog8 *typed pointers* better express the intent and better tell the compiler how to use the pointer. 
These are preferred to be used in new code, and are explained below.


Typed pointer to simple datatype
--------------------------------
.. index:: single: Pointers; Simple Typed

Prog8 syntax has the 'double hat' token ``^^`` that appears either in front of a type ("pointer to this type") or
after a pointer variable ("get the value it points to" - a pointer dereference).

So the syntax for declaring typed pointers looks like this:

``^^type``: pointer to a type
    You can declare a pointer to any numeric datatype (bytes, words, longs, floats), and booleans, and struct types.
    Structs are explained in the next section.
    So for example; ``^^float fptr`` declares fptr as a pointer to a float value.

``^^type[size]``: array with size size containing pointers to a type.
    So for example; ``^^word[100] values`` declares values to be an array of 100 pointers to words.
    Note that an array of pointers (regardless of the type they point to) is always a split word array.
    (this is the most efficient way to access the pointers, and they need to be copied to zeropage first to
    be able to use them anyway. It also allows for arrays of up to 256 pointers instead of 128.)

It is not possible to define pointers to *arrays*; ``^^(type[])`` is invalid syntax.

Pointers of different types cannot be assigned to one another, unless you use an explicit cast.
This rule is not enforced for untyped pointers/regular uword, as described earlier.

The ``str`` type in subroutine parameters and return values has always been a bit weird in the sense that in these cases,
the string is actually passed by reference (it's address pointer is passed) instead of a ``str`` variable that is accessed by value.
In previous Prog8 versions these were untyped uword pointers, but since version 12, these are now translated as ``^^ubyte``.
Resulting assembly code should be equivalent still.


Dereferencing a pointer, pointer arithmetic
-------------------------------------------
.. index::
    pair: Pointers; Dereferencing a pointer
    pair: Pointers; Pointer arithmetic

To get the value the pointer points at, you *dereference* the pointer. The syntax for that is: ``pointer^^``.
Say the pointer variable is of type ``^^float``, then ``pointer^^`` will return the float value it points at.

You can also use array indexing syntax to get the n-th value. For example: ``floatpointer[3]`` will return the
fourth floating point value in the sequence that the floatpointer points at. Because the pointer is a typed pointer,
the compiler knows what the size of the value is that it points at and correctly skips forward the required number of bytes in memory.
In this case, say a float takes 5 bytes, then ``floatpointer[3]`` will return the float value stored at memory address floatpointer+15.
Notice that ``floatpointer[0]`` is equivalent to ``floatpointer^^``.

You can add and subtract values from a pointer, this is called **pointer arithmetic**.
For example, to advance a pointer to the next value, you can use ``pointer++``.
To make it point to the preceding value, you can use ``pointer--``.
**Adding or subtracting X to a pointer will change the pointer by X times the size of the value it points at (the same as the C language does it),
instead of simply adding or subtracting the value from the pointer address value.**
(that is what Prog8 still does for untyped uword pointers, or pointers to a type that just takes up a single byte of memory).

That special pointer arithmetic is also performed for pointers to struct types:
the compiler knows the memory storage size of the whole struct type and advances or rewinds
the pointer value (memory address) by the appropriate number of bytes (X times the size of the struct). More info about structs can be found below.

You can subtract two pointers of the **same** type from each other. The result is the number of elements between them (C-style),
using the same scaling rules as adding or subtracting an integer. The result has the target's address type (``uword`` on 16-bit targets, ``long`` on 32-bit targets).
For example, if ``^^uword p1`` points at address ``$2000`` and ``^^uword p2`` points at address ``$2006``, then ``p2 - p1`` is ``3`` (because each ``uword`` is 2 bytes).
Adding two pointers together is not supported. Other arithmetic operators such as ``*``, ``/``, ``%`` and bitwise operators are also not supported on typed pointers; cast to an unsigned word or long first if you need those operations.

Untyped pointers (plain ``pointer`` or ``uword``/``long`` used as an address) do not use this scaling at all; arithmetic on them is plain numeric address arithmetic.


Structs
-------
.. index:: pair: Structs; Definition

A struct is a grouping of multiple variables. Say your game is going to track several enemy sprites on the screen,
in which case it may be useful to describe the various properties of an enemy together in a struct type, rather than
dealing with all of them separately.  You first define the struct type like so::

    struct Enemy {
        ubyte xpos, ypos
        uword health
        bool elite
    }

Allowed field datatypes: all numeric types (``byte``, ``ubyte``, ``word``, ``uword``, ``long``, ``float``),
``bool``, ``str``, typed pointers to those or to other structs, and inline 1D arrays of these element types.
Not allowed: struct instances themselves (structs cannot be nested, use a pointer to the struct instead),
arrays of struct instances, arrays of typed pointers such as ``^^Enemy[4]`` (use untyped pointers instead), and 2D arrays.
The typed pointer array and 2D array restrictions are limitations of the current compiler; they may or may not be lifted in a future version.

Fields in a struct are 'packed' (meaning the values are placed back-to-back in memory), and placed in memory in order of declaration. This guarantees exact size and place of the fields.
``sizeof()`` knows how to calculate the combined size of a struct, and ``offsetof()`` can be used to get the byte offset of a given field in the struct.
The size of a struct cannot exceed 256 bytes on 6502-based targets (C64, CX16, C128, PET32 - one memory page).
On 32-bit targets the limit is much larger: 32768 bytes on Amiga/Qemu68k and 65535 bytes on the virtual target.

You can copy the whole contents of a struct to another one by assigning the dereferenced pointers::

    ^^Enemy e1,e2
    e1^^ = e2^^     ; copies all fields of e2 into e1


The struct type creates a new name scope, so accessing the fields of a struct is done as usual with the dotted notation.
Because it implies pointer dereferencing you can usually omit the explicit `^^`, prog8 will know what it means::

    if e1.ypos > 300
        e1.health -= 10

    ; explicit dereferencing notation:

    if e1^^.ypos > 300
        e1^^.health -= 10

You can also assign to a struct field through an arbitrary pointer expression by wrapping it in parentheses.
This works with typecasts and with function calls that return a pointer::

    (enemies[i] as ^^Enemy).health = 42
    (get_enemy(index)).shield -= 10
    get_enemy(index).next.target = 0

The compiler evaluates the pointer expression exactly once and then stores the value through the computed address.


.. note::
    Structs are primarily supported as a *reference type* accessed through a pointer.
    However, you can also declare an *array of struct instances* (see below), and a plain struct instance
    can be used as an lvalue in a few specific assignment cases.
    It is not yet possible to use a struct as a standalone value type, or as a memory-mapped type.
    The compiler replaces whole-struct assignments with a memory copy. You are allowed to write::

        ptr2^^ = ptr1^^
        ptr2^^ = ptr1[2]
        ptr2[2] = ptr1^^
        arr[i] = arr[j]           ; array of struct instances

    In the future more cases may be supported.


.. _structarrays:

Arrays of struct instances
--------------------------
.. index:: pair: Structs; Arrays of instances

You can declare an array that contains struct instances directly, rather than pointers to structs.
The syntax places the array size between the struct type and the variable name::

    struct Enemy {
        ubyte xpos, ypos
        uword health
        bool elite
    }

    Enemy[10] enemies          ; 10 Enemy instances in a contiguous block

Indexing works as expected and the fields are accessed with the dotted notation::

    enemies[i].health = 100
    enemies[i].elite = true
    ubyte x = enemies[i].xpos

You can copy one whole element to another with a single assignment; the compiler generates a memory copy::

    enemies[i] = enemies[j]

Array-of-struct variables can also be statically initialized by nesting struct initializers inside an array literal.
The initializer elements are struct values, not pointer allocations, so you must not use ``^^``.
You can write them with an explicit type or let the compiler infer it::

    Enemy[3] squad = [
        Enemy : [10, 20, 100, false],
        Enemy : [30, 40, 200, true],
        Enemy : [50, 60, 150, false]
    ]

    ; inferred form (preferred)
    Enemy[3] squad2 = [
        [10, 20, 100, false],
        [30, 40, 200, true],
        [50, 60, 150, false]
    ]

The struct size counts toward the same target size limits as other arrays.
On 6502-family targets a struct cannot exceed 256 bytes, so the maximum number of elements in an
array of structs is limited by that size (for example, at most 128 elements for a 2-byte struct, and
fewer for larger structs). On 32-bit targets the per-array size limit is much larger.

.. note::
    Arrays of struct instances cannot be used as subroutine parameters (the same restriction applies
    to most array types other than ``ubyte`` arrays and strings). Pass a typed pointer to the first
    element instead, or use ``sys.memcopy`` if you need to pass a copy.

.. note::
    Arrays of struct instances are stored contiguously, so accessing fields with a variable index
    is less efficient than accessing fields through a typed pointer. For performance-critical code,
    prefer typed pointers to structs unless the contiguous array layout is required.

.. note::
    Using structs instead of plain arrays usually results in more and less efficient code being generated.
    This is because the 6502 CPU is not particularly well equipped to dealing with pointers and accessing struct fields via offsets,
    as compared to direct variable access or array indexing. The prog8 program code may be easier to work with though!

.. note::
    Accessing the first field in a struct is more efficient than subsequent fields, because it
    is at offset 0 so no additional addition has to be computed on a pointer to reach the first field.
    Try to put the most often accessed field as the first field to potentially gain a rather substantial boost in code efficiency.


.. _unions:

Unions
------
.. index:: pair: Unions; Definition

A union is like a struct, but every field starts at byte offset 0.
The fields overlap in memory, so writing to one field overwrites the others.
The storage size of a union is the size of its largest field, not the sum of all fields.

You define a union type like this::

    union Value {
        ubyte byte_val
        uword word_val
        float float_val
    }

Allowed field datatypes are the same as for structs: all numeric types, ``bool``, ``str``, typed pointers to those or to other structs, and inline 1D arrays of these element types.
Not allowed are struct instances themselves, unions themselves, arrays of struct instances, arrays of typed pointers such as ``^^Value[4]``, and 2D arrays.
So neither structs nor unions can be nested inside another struct or union; use a typed pointer to them instead.
``sizeof(Value)`` returns the size of the largest field, and ``offsetof(Value.field)`` is always ``0`` for every field.
A union type can be used wherever a struct type is used: typed pointers, pointer arrays, arrays of union instances, subroutine parameters, and return values.
Whole-union copy semantics are the same as for structs: assigning one dereferenced union pointer to another copies ``sizeof(Value)`` bytes.

When to use a union
^^^^^^^^^^^^^^^^^^^
Unions are for interpreting the same memory bytes in more than one way.
Typical uses:

- **Type punning:** read or write the same bytes through different views, e.g. set the low byte with the 1-byte view and then read the 2-byte view::

      v.byte_val = 42
      chk = v.word_val      ; sees the overlapping bytes

- **Hardware or memory-mapped registers:** one location that the hardware interprets as a byte or a word depending on how it is accessed.
- **Overlays:** reinterpret a buffer, such as the raw bytes of a ``float``, for debugging or serialization, or reuse the same scratch space between different phases of a program.

Note that unions do not (yet) help save space inside structs, because nesting a union (or struct) as a field is not allowed; see above.

Because all fields overlap, static initialization with values is not allowed.
The only valid initializer is the empty form ``[]``, which places a zero-initialized union instance in BSS::

    ^^Value v = []         ; zero-initialized union instance

Values must be written at runtime through the pointer::

    v.byte_val = 42
    ; v.word_val and v.float_val now see the overlapping bytes

.. note::
    Unions, like structs, cannot yet be declared as a local instance - you always access
    a union through a typed pointer, as shown above. See the note in the structs section
    for the current limitations around using structs (and now unions) as standalone values.


Static initialization of structs
================================
.. index:: pair: Structs; Static initialization

You can 'allocate' and statically initialize a struct. This behave much like initializing arrays does,
and it won't reset to the original value when the program is restarted, so beware.
*Remember that the struct is statically allocated, and appears just once in the memory:*
This means that, for instance, if you do this in a subroutine that gets
called multiple times, or inside a loop, the struct *will be the same instance every time*.
Read below if you need *dynamic* struct allocation!
You write a static struct initialization expression like this:

``^^Node : [1,"one", 1000, true, 1.111]``
    statically places an instance of struct 'Node' in memory, with its fields set to 1, "one", 1000 etcetera and returns the address of this struct.
    The values in the initialization array must correspond exactly with the first to last declared fields in the struct type.

Named field initializers
^^^^^^^^^^^^^^^^^^^^^^^^

For a more readable initializer, you can specify field names instead of relying
on declaration order::

    ^^Node : [name="one", flag=true, id=1, array=1000, perc=1.111]

Named fields may be written in any order. Fields that are not specified are
initialized to zero. This also applies to inline array fields, which are filled
with zeroes. Each field may be specified only once, and every name must refer
to a field in the struct. Named and positional values cannot be mixed in the
same initializer.

If the struct contains inline arrays, you can initialize them by nesting another list inside the initialization list:

``^^Node : [1, [10, 20, 30, 40], 1000]``
    Initializes a struct Node where the second field is an inline array (e.g., ``ubyte[4] data``). The nested array ``[10, 20, 30, 40]`` initializes that array field.
``^^Node : []``
    (without values) Places a 'Node' instance in BSS variable space instead, which gets zeroed out at program startup.
    Returns the address of this empty struct.

Initializing byte-array fields with a string
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. index:: pair: Structs; String initializer

For a sized ``ubyte`` (or ``byte``) array field, a string literal may be used in place of an explicit
byte array. The string is encoded using the current ``%encoding`` and then filled into the array using
C-style semantics: if the encoded string is shorter than the array, the remaining bytes are filled with
zero (the first zero acts as the implicit C string terminator); if it is exactly the array length, no
extra terminator is added; and if it is longer than the array, a compilation error is reported.

For example, given ``ubyte[4] name``, the initializers ``"abcd"``, ``"abc"`` and ``"a"`` are all
accepted and produce ``[a,b,c,d]``, ``[a,b,c,0]`` and ``[a,0,0,0]`` respectively, while ``"abcde"``
causes an error. The same applies to inferred-form initializers such as ``^^Node n = [1, "abc", true]``.


The field values in a struct initializer must be constants. You can use numeric literals, ``address-of`` expressions,
or ``memory()`` calls (either directly or via ``const`` variables).
It is also possible to put struct initializer inside arrays to make them all statically initialized and accessible via the array::

    ^^Node[] allnodes = [
        ^^Node: [1,"one", 1000, true, 1.111],
        ^^Node: [2,"two", 2000, false, 2.222],
        ^^Node: [],
        ^^Node: [],
    ]

Short form initializers
^^^^^^^^^^^^^^^^^^^^^^^
.. index:: pair: Structs; Initializers

If the required type can be inferred from the context you can also omit the struct pointer type prefix altogether.
The initializer value then is syntactically the same as an array, but Prog8 internally turns it back into a proper
struct initializer value based on the the type of the array element or pointer variable it is assigned to.
So you can write the above in short form as::

    ^^Node nodepointer = [1,2,3,4]

    ^^Node[] allnodes = [
        [1,"one", 1000, true, 1.111],
        [2,"two", 2000, false, 2.222],
        [],
        []
    ]



Dynamic allocation of structs
=============================
.. index:: pair: Structs; Dynamic allocation

There is no real 'dynamic' memory allocation in Prog8. Everything is statically allocated. This doesn't change with struct types.
However, it is possible to write a dynamic memory handling library yourself (it has to track memory blocks manually).
If you ask such a library to give you a pointer to a piece of memory with size ``sizeof(Enemy)`` you can use that as
a dynamic pointer to an Enemy struct.

For this pattern, use the standard ``arena`` library module. Create a static
memory slab, pass it directly to the initializer, and then allocate from it::

    %import arena

    arena.init(memory("arena", 2000, 0), 2000)

    ^^Node newnode = arena.alloc(sizeof(Node))
    ...
    arena.reset()

Each compilation target selects the appropriate ``arena`` implementation:
6502 targets use ``uword`` sizes without even-address rounding, while m68k
and virtual targets use ``long`` sizes and round odd allocations up to an
even address. ``arena.alloc()`` returns 0 when the slab is exhausted. The
module tracks only one slab at a time; calling ``arena.init()`` again
discards any previous allocations. Individual allocations cannot be freed;
``arena.reset()`` makes the whole slab available again.


Address-Of: untyped vs typed
----------------------------
.. index:: single: Structs; Address-Of

``&`` still returns an untyped (uword) pointer, as it did in older Prog8 versions. This is for backward compatibility reasons so existing programs don't break.
The new *double ampersand* operator ``&&`` returns a *typed* pointer to the value. The semantics are slightly different from the old untyped address-of operator, because adding or subtracting
a number from a typed pointer uses *pointer arithmetic* that takes the size of the value that it points to into account.


Accessing struct definitions in Assembly code
---------------------------------------------
.. index:: single: Structs; Access from assembly

Prog8 lets you query the size of a struct type, and the offsets of a field.
You can do the same in assembly code if needed: the struct definition gets written as `.struct` into the assembly file::

    ; prog8 struct declaration:
    struct Node {
        ubyte type
        uword value
        bool flag
    }

    ; generates this assembly code:
    ; (the symbol prefixes are explained in the 'Technical details' chapter)
    p8b_main.p8t_Node    .struct f0,f1,f2
    p8v_type  .byte  \f0
    p8v_value  .word  \f1
    p8v_flag  .byte  \f2
        .endstruct


64tass then lets you query that information::

    ; prog8 code:
    ubyte size = sizeof(Node)
    ubyte offset1 = offsetof(Node.type)
    ubyte offset2 = offsetof(Node.value)
    ubyte offset3 = offsetof(Node.flag)

    ; assembly equivalents (64tass syntax):
    lda  #size(p8t_Node)
    lda  #p8t_Node.p8v_type
    lda  #p8t_Node.p8v_value
    lda  #p8t_Node.p8v_flag