prog8/CompilerDevelopment.md

#### Just a few remarks upfront:
* There is the (gradle/IDEA) module `parser`: that's the parser generated by ANTLR4, in Java. The only file to be edited here is the grammar, `prog8.g4`.
* Then we have the module `compilerAst` - in Kotlin - which uses `parser` and adds AST nodes. Here we put our additions to the generated thing, *including any tests of the parsing stage*.
  - the name is a bit misleading, as this module isn't (or, resp. shouldn't be; see below) about *compiling*, only the parsing stage
  - also, the tree that comes out isn't much of an *abstraction*, but rather still more or less a parse tree (this might very well change).
  - **However, let's not *yet* rename the module.** We'll find a good name during refactoring.

#### Problems with `compilerAst`:
* `ModuleImporter.kt`, doing (Prog8-) module resolution. That's not the parser's job.
* During parsing, character literals are turned into UBYTEs (since there is no basic type e.g. CHAR). That's bad because it depends on a specific character encoding (`IStringEncoding` in `compilerAst/src/prog8/ast/AstToplevel.kt`) of/for some target platform. Note that *strings* are indeed encoded later, in the `compiler` module.
* The same argument applies to `IMemSizer`, and - not entirely sure about that - `IBuiltinFunctions`.

#### Steps to take, in order:
1. introduce an abstraction `SourceCode` that encapsulates the origin and actual loading of Prog8 source code
   - from the local file system (use case: user programs)
   - from resources (prog8lib)
   - from plain strings (for testing)
2. introduce a minimal interface to the outside, input: `SourceCode`, output: a tree with a `Module` node as the root
   - this will be the Kotlin singleton `Prog8Parser` with the main method `parseModule`
   - plus, optionally, method's for registering/unregistering a listener with the parser
   - anything related to the lexer, error strategies, character/token streams is hidden from the outside
   - to make a clear distinction between the *generated* parser (and lexer) vs. `Prog8Parser`, and to discourage directly using the generated stuff, we'll rename the existing `prog8Parser`/`prog8Lexer` to `Prog8ANTLRParser` and `Prog8ANTLRLexer` and move them to package `prog8.parser.generated`
3. introduce AST node `CharLiteral` and keep them until after identifier resolution and type checking; insert there an AST transformation step that turns them in UBYTE constants (literals)
4. remove uses of `IStringEncoding` from module `compilerAst` - none should be necessary anymore
5. move `IStringEncoding` to module `compiler`
6. same with `ModuleImporter`, then rewrite that (addressing #46)
7. refactor AST nodes and grammar: less generated parse tree nodes (`XyzContext`), less intermediary stuff (private classes in `Antr2Kotlin.kt` [sic]), more compact code. Also: nicer names such as simply `StringLiteral` instead of `StringLiteralValue`
8. re-think `IStringEncoding` to address #38