prog8/docs/source/structpointers.rst

.. _pointers:

********************
Structs and Pointers
********************

.. index::
      pair: Structs; Overview
      pair: Pointers; Overview

.. attention::
    The 6502 cpu lacks some features (addressing modes, registers) to make pointers work efficiently.
    Also it requires that pointer variables have to be in zero page, or copied to a temporary zero page variable,
    before they can even be used as a pointer. This means that pointer operations in prog8 compile
    to rather large and inefficient assembly code most of the time, when compared to direct array access or regular variables.
    At least try to place heavily used pointer variables in zero page using ``@requirezp`` on their declaration,
    if zero page space allows.
    Pointer variables that don't have a zeropage tag specified will be treated as having ``@zp`` so they get
    priority over other variables to be placed into zeropage.

.. note::
    Due to a few limitations in the language parser, some pointer related syntax is currently unsupported.
    The compiler tries its best to give a descriptive error message but sometimes there is still a
    parser limitation that has to be worked around at the moment. For example, this assignment syntax doesn't parse correctly::

        ^^Node  np
        np[2].field = 9999          ; cannot use this syntax as assignment target right now
        ubyte value = np[2].field   ; note that using it as expression value works fine

    To work around this you'll have to explicitly write the pointer dereferencing operator,
    or break up the expression in multiple steps (which can be beneficial too when you are assigning multiple fields
    because it will save a pointer calculation for every assignment)::

        ^^Node  np
        np[2]^^.field = 9999

        ; alternatively, split up:
        ^^Node thenode = &&np[2]
        thenode.field = 9999

    Writing through a parenthesized casting expression or a function call does work however;
    the compiler evaluates the expression once and stores through the computed address::

        (np[2] as ^^Node).field = 9999
        get_node(i).field = 9999


Untyped pointers and the ``pointer`` type
-----------------------------------------
.. index:: single: Pointers; untyped

Prior to version 12 of the language, the only pointer type available was a plain ``uword`` value (the memory address)
which could be used as a pointer to an ``ubyte`` (the byte value at that memory address).
Array indexing on an ``uword`` simply means to point to the ``ubyte`` at the location of the address + index value.

When the address of a value (explicitly) or a value of a reference type (string, array) was passed as an argument to a subroutine call,
it became one of these plain ``uword`` 'pointers'. The subroutine receiving it always had to interpret the 'pointer'
explicitly for what it actually pointed to, if that wasn't a simple byte.

Some implicit conversions were allowed too (such as putting ``str`` as the type of a subroutine parameter,
which would be changed to ``uword`` by the compiler).

Since version 12.6 there now is a dedicated ``pointer`` type keyword that serves as an untyped address
and automatically takes the :ref:`pointer size <pointer_size>` for the target.
This is useful when writing code that is compiled for different targets,
because you don't need to manually switch between ``uword`` and ``long`` via conditional compilation::

    pointer ptr = $a000
    ; same program works on both 6502 and m68k targets
    ; ptr is 2 bytes on C64, CX16, etc.
    ; ptr is 4 bytes on qemu68k

You can assign any other pointer type to an untyped pointer variable (``uword``, ``long``` or ``pointer``) without the need for an explicit cast.
You can assign an untyped pointer to a typed pointer variable without the need for an explicit cast.
Prog8 *typed pointers* better express the intent and better tell the compiler how to use the pointer. 
These are preferred to be used in new code, and are explained below.


Typed pointer to simple datatype
--------------------------------
.. index:: single: Pointers; Simple Typed

Prog8 syntax has the 'double hat' token ``^^`` that appears either in front of a type ("pointer to this type") or
after a pointer variable ("get the value it points to" - a pointer dereference).

So the syntax for declaring typed pointers looks like this:

``^^type``: pointer to a type
    You can declare a pointer to any numeric datatype (bytes, words, longs, floats), and booleans, and struct types.
    Structs are explained in the next section.
    So for example; ``^^float fptr`` declares fptr as a pointer to a float value.

``^^type[size]``: array with size size containing pointers to a type.
    So for example; ``^^word[100] values`` declares values to be an array of 100 pointers to words.
    Note that an array of pointers (regardless of the type they point to) is always a split word array.
    (this is the most efficient way to access the pointers, and they need to be copied to zeropage first to
    be able to use them anyway. It also allows for arrays of up to 256 pointers instead of 128.)

It is not possible to define pointers to *arrays*; ``^^(type[])`` is invalid syntax.

Pointers of different types cannot be assigned to one another, unless you use an explicit cast.
This rule is not enforced for untyped pointers/regular uword, as described earlier.

The ``str`` type in subroutine parameters and return values has always been a bit weird in the sense that in these cases,
the string is actually passed by reference (it's address pointer is passed) instead of a ``str`` variable that is accessed by value.
In previous Prog8 versions these were untyped uword pointers, but since version 12, these are now translated as ``^^ubyte``.
Resulting assembly code should be equivalent still.


Dereferencing a pointer, pointer arithmetic
-------------------------------------------
.. index::
    pair: Pointers; Dereferencing a pointer
    pair: Pointers; Pointer arithmetic

To get the value the pointer points at, you *dereference* the pointer. The syntax for that is: ``pointer^^``.
Say the pointer variable is of type ``^^float``, then ``pointer^^`` will return the float value it points at.

You can also use array indexing syntax to get the n-th value. For example: ``floatpointer[3]`` will return the
fourth floating point value in the sequence that the floatpointer points at. Because the pointer is a typed pointer,
the compiler knows what the size of the value is that it points at and correctly skips forward the required number of bytes in memory.
In this case, say a float takes 5 bytes, then ``floatpointer[3]`` will return the float value stored at memory address floatpointer+15.
Notice that ``floatpointer[0]`` is equivalent to ``floatpointer^^``.

You can add and subtract values from a pointer, this is called **pointer arithmetic**.
For example, to advance a pointer to the next value, you can use ``pointer++``.
To make it point to the preceding value, you can use ``pointer--``.
**Adding or subtracting X to a pointer will change the pointer by X times the size of the value it points at (the same as the C language does it),
instead of simply adding or subtracting the value from the pointer address value.**
(that is what Prog8 still does for untyped uword pointers, or pointers to a type that just takes up a single byte of memory).

That special pointer arithmetic is also performed for pointers to struct types:
the compiler knows the memory storage size of the whole struct type and advances or rewinds
the pointer value (memory address) by the appropriate number of bytes (X times the size of the struct). More info about structs can be found below.


Structs
-------
.. index:: pair: Structs; Definition

A struct is a grouping of multiple variables. Say your game is going to track several enemy sprites on the screen,
in which case it may be useful to describe the various properties of an enemy together in a struct type, rather than
dealing with all of them separately.  You first define the struct type like so::

    struct Enemy {
        ubyte xpos, ypos
        uword health
        bool elite
    }

Allowed field datatypes: all numeric types (``byte``, ``ubyte``, ``word``, ``uword``, ``long``, ``float``),
``bool``, ``str``, typed pointers to those or to other structs, and inline 1D arrays of these element types.
Not allowed: struct instances themselves (structs cannot be nested, use a pointer to the struct instead),
arrays of struct instances, arrays of typed pointers such as ``^^Enemy[4]`` (use untyped pointers instead), and 2D arrays.
The typed pointer array and 2D array restrictions are limitations of the current compiler; they may or may not be lifted in a future version.

Fields in a struct are 'packed' (meaning the values are placed back-to-back in memory), and placed in memory in order of declaration. This guarantees exact size and place of the fields.
``sizeof()`` knows how to calculate the combined size of a struct, and ``offsetof()`` can be used to get the byte offset of a given field in the struct.
The size of a struct cannot exceed 256 bytes on 6502-based targets (C64, CX16, C128, PET32 - one memory page).
On 32-bit targets the limit is much larger: 32768 bytes on Amiga/Qemu68k and 65535 bytes on the virtual target.

You can copy the whole contents of a struct to another one by assigning the dereferenced pointers::

    ^^Enemy e1,e2
    e1^^ = e2^^     ; copies all fields of e2 into e1


The struct type creates a new name scope, so accessing the fields of a struct is done as usual with the dotted notation.
Because it implies pointer dereferencing you can usually omit the explicit `^^`, prog8 will know what it means::

    if e1.ypos > 300
        e1.health -= 10

    ; explicit dereferencing notation:

    if e1^^.ypos > 300
        e1^^.health -= 10

You can also assign to a struct field through an arbitrary pointer expression by wrapping it in parentheses.
This works with typecasts and with function calls that return a pointer::

    (enemies[i] as ^^Enemy).health = 42
    (get_enemy(index)).shield -= 10
    get_enemy(index).next.target = 0

The compiler evaluates the pointer expression exactly once and then stores the value through the computed address.


.. note::
    Structs are primarily supported as a *reference type* accessed through a pointer.
    However, you can also declare an *array of struct instances* (see below), and a plain struct instance
    can be used as an lvalue in a few specific assignment cases.
    It is not yet possible to use a struct as a standalone value type, or as a memory-mapped type.
    The compiler replaces whole-struct assignments with a memory copy. You are allowed to write::

        ptr2^^ = ptr1^^
        ptr2^^ = ptr1[2]
        ptr2[2] = ptr1^^
        arr[i] = arr[j]           ; array of struct instances

    In the future more cases may be supported.


.. _structarrays:

Arrays of struct instances
--------------------------
.. index:: pair: Structs; Arrays of instances

You can declare an array that contains struct instances directly, rather than pointers to structs.
The syntax places the array size between the struct type and the variable name::

    struct Enemy {
        ubyte xpos, ypos
        uword health
        bool elite
    }

    Enemy[10] enemies          ; 10 Enemy instances in a contiguous block

Indexing works as expected and the fields are accessed with the dotted notation::

    enemies[i].health = 100
    enemies[i].elite = true
    ubyte x = enemies[i].xpos

You can copy one whole element to another with a single assignment; the compiler generates a memory copy::

    enemies[i] = enemies[j]

Array-of-struct variables can also be statically initialized by nesting struct initializers inside an array literal.
The initializer elements are struct values, not pointer allocations, so you must not use ``^^``.
You can write them with an explicit type or let the compiler infer it::

    Enemy[3] squad = [
        Enemy : [10, 20, 100, false],
        Enemy : [30, 40, 200, true],
        Enemy : [50, 60, 150, false]
    ]

    ; inferred form (preferred)
    Enemy[3] squad2 = [
        [10, 20, 100, false],
        [30, 40, 200, true],
        [50, 60, 150, false]
    ]

The struct size counts toward the same target size limits as other arrays.
On 6502-family targets a struct cannot exceed 256 bytes, so the maximum number of elements in an
array of structs is limited by that size (for example, at most 128 elements for a 2-byte struct, and
fewer for larger structs). On 32-bit targets the per-array size limit is much larger.

.. note::
    Arrays of struct instances cannot be used as subroutine parameters (the same restriction applies
    to most array types other than ``ubyte`` arrays and strings). Pass a typed pointer to the first
    element instead, or use ``sys.memcopy`` if you need to pass a copy.

.. note::
    Arrays of struct instances are stored contiguously, so accessing fields with a variable index
    is less efficient than accessing fields through a typed pointer. For performance-critical code,
    prefer typed pointers to structs unless the contiguous array layout is required.

.. note::
    Using structs instead of plain arrays usually results in more and less efficient code being generated.
    This is because the 6502 CPU is not particularly well equipped to dealing with pointers and accessing struct fields via offsets,
    as compared to direct variable access or array indexing. The prog8 program code may be easier to work with though!

.. note::
    Accessing the first field in a struct is more efficient than subsequent fields, because it
    is at offset 0 so no additional addition has to be computed on a pointer to reach the first field.
    Try to put the most often accessed field as the first field to potentially gain a rather substantial boost in code efficiency.


Static initialization of structs
================================
.. index:: pair: Structs; Static initialization

You can 'allocate' and statically initialize a struct. This behave much like initializing arrays does,
and it won't reset to the original value when the program is restarted, so beware.
*Remember that the struct is statically allocated, and appears just once in the memory:*
This means that, for instance, if you do this in a subroutine that gets
called multiple times, or inside a loop, the struct *will be the same instance every time*.
Read below if you need *dynamic* struct allocation!
You write a static struct initialization expression like this:

``^^Node : [1,"one", 1000, true, 1.111]``
    statically places an instance of struct 'Node' in memory, with its fields set to 1, "one", 1000 etcetera and returns the address of this struct.
    The values in the initialization array must correspond exactly with the first to last declared fields in the struct type.

If the struct contains inline arrays, you can initialize them by nesting another list inside the initialization list:

``^^Node : [1, [10, 20, 30, 40], 1000]``
    Initializes a struct Node where the second field is an inline array (e.g., ``ubyte[4] data``). The nested array ``[10, 20, 30, 40]`` initializes that array field.
``^^Node : []``
    (without values) Places a 'Node' instance in BSS variable space instead, which gets zeroed out at program startup.
    Returns the address of this empty struct.

Initializing byte-array fields with a string
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. index:: pair: Structs; String initializer

For a sized ``ubyte`` (or ``byte``) array field, a string literal may be used in place of an explicit
byte array. The string is encoded using the current ``%encoding`` and then filled into the array using
C-style semantics: if the encoded string is shorter than the array, the remaining bytes are filled with
zero (the first zero acts as the implicit C string terminator); if it is exactly the array length, no
extra terminator is added; and if it is longer than the array, a compilation error is reported.

For example, given ``ubyte[4] name``, the initializers ``"abcd"``, ``"abc"`` and ``"a"`` are all
accepted and produce ``[a,b,c,d]``, ``[a,b,c,0]`` and ``[a,0,0,0]`` respectively, while ``"abcde"``
causes an error. The same applies to inferred-form initializers such as ``^^Node n = [1, "abc", true]``.


The field values in a struct initializer must be constants. You can use numeric literals, ``address-of`` expressions,
or ``memory()`` calls (either directly or via ``const`` variables).
It is also possible to put struct initializer inside arrays to make them all statically initialized and accessible via the array::

    ^^Node[] allnodes = [
        ^^Node: [1,"one", 1000, true, 1.111],
        ^^Node: [2,"two", 2000, false, 2.222],
        ^^Node: [],
        ^^Node: [],
    ]

Short form initializers
^^^^^^^^^^^^^^^^^^^^^^^
.. index:: pair: Structs; Initializers

If the required type can be inferred from the context you can also omit the struct pointer type prefix altogether.
The initializer value then is syntactically the same as an array, but Prog8 internally turns it back into a proper
struct initializer value based on the the type of the array element or pointer variable it is assigned to.
So you can write the above in short form as::

    ^^Node nodepointer = [1,2,3,4]

    ^^Node[] allnodes = [
        [1,"one", 1000, true, 1.111],
        [2,"two", 2000, false, 2.222],
        [],
        []
    ]



Dynamic allocation of structs
=============================
.. index:: pair: Structs; Dynamic allocation

There is no real 'dynamic' memory allocation in Prog8. Everything is statically allocated. This doesn't change with struct types.
However, it is possible to write a dynamic memory handling library yourself (it has to track memory blocks manually).
If you ask such a library to give you a pointer to a piece of memory with size ``sizeof(Enemy)`` you can use that as
a dynamic pointer to an Enemy struct.

An example of how a super simple dynamic allocator could look like follows. 
Noe that the ``pointer`` type is used instead of ``uword`` so that the code also works correctly on targets 
where memory addresses are wider than 2 bytes (such as the m68k target with 4-byte pointers)::

    ^^Node newnode = allocator.alloc(sizeof(Node))
    ...

    allocator {
        ; extremely trivial arena allocator
        pointer buffer = memory("arena", 2000, 0)
        pointer next = buffer

        sub alloc(ubyte size) -> pointer {
            defer next += size
            return next
        }

        sub freeall() {
            ; cannot free individual allocations only the whole arena at once
            next = buffer
        }
    }


Address-Of: untyped vs typed
----------------------------
.. index:: single: Structs; Address-Of

``&`` still returns an untyped (uword) pointer, as it did in older Prog8 versions. This is for backward compatibility reasons so existing programs don't break.
The new *double ampersand* operator ``&&`` returns a *typed* pointer to the value. The semantics are slightly different from the old untyped address-of operator, because adding or subtracting
a number from a typed pointer uses *pointer arithmetic* that takes the size of the value that it points to into account.


Accessing struct definitions in Assembly code
---------------------------------------------
.. index:: single: Structs; Access from assembly

Prog8 lets you query the size of a struct type, and the offsets of a field.
You can do the same in assembly code if needed: the struct definition gets written as `.struct` into the assembly file::

    ; prog8 struct declaration:
    struct Node {
        ubyte type
        uword value
        bool flag
    }

    ; generates this assembly code:
    ; (the symbol prefixes are explained in the 'Technical details' chapter)
    p8b_main.p8t_Node    .struct f0,f1,f2
    p8v_type  .byte  \f0
    p8v_value  .word  \f1
    p8v_flag  .byte  \f2
        .endstruct


64tass then lets you query that information::

    ; prog8 code:
    ubyte size = sizeof(Node)
    ubyte offset1 = offsetof(Node.type)
    ubyte offset2 = offsetof(Node.value)
    ubyte offset3 = offsetof(Node.flag)

    ; assembly equivalents (64tass syntax):
    lda  #size(p8t_Node)
    lda  #p8t_Node.p8v_type
    lda  #p8t_Node.p8v_value
    lda  #p8t_Node.p8v_flag