Module templates

docs/README.md

@@ -16,6 +16,8 @@
 
 * [Preprocessor](lang/preprocessor.md)
 
+* [Modules](lang/modules.md)
+
 * [Syntax](lang/syntax.md)
 
 * [Types](lang/types.md)

docs/api/custom-platform.md

@@ -60,7 +60,8 @@ See [the list of available encodings](../lang/text.md).
 * `screen_encoding` – default encoding for screencodes (literals with encoding specified as `scr`). 
 Default: the same as `encoding`.
 
-* `modules` – comma-separated list of modules that will be automatically imported
+* `modules` – comma-separated list of modules that will be automatically imported.
+This list cannot contain module template instantiations.
 
 * other compilation options (they can be overridden using commandline options):
 

docs/doc_index.md

@@ -16,6 +16,8 @@
 
 * [Preprocessor](lang/preprocessor.md)
 
+* [Modules](lang/modules.md)
+
 * [Syntax](lang/syntax.md)
 
 * [Types](lang/types.md)

docs/lang/modules.md

@@ -0,0 +1,97 @@
+[< back to index](../doc_index.md)
+
+# Program structure
+
+A Millfork program is build from one or more modules.
+
+Each module is stored in a single file.
+All source filenames passed to the compiler are considered to be modules of that program, called _root modules_.
+
+Each module has a name, which is its unique identifier. 
+A module name is a sequence of slash-separated valid Millfork identifiers.
+The name also defines where the module is located:
+a module named `a/b` is presumed to exist in `a/b.mfk`
+and it's looked up first in the current working directory,
+and then in the include directories.
+
+A module can import other modules, using the `import` statement. 
+Importing the same module multiple times merely marks it as imported by multiple modules,
+but the program will still contain only one copy of it. 
+Examples:
+
+    import string
+    import cbm_file
+    
+Usually, the imported module will undergo the first phase of compilation first.
+This means that the constants in the imported module will be resolved first, allowing you to use them in the importing module.
+
+
+The only exception to this rule is when the importing graph has a cycle, in which case the order of modules within the cycle is unspecified.
+
+A platform may define starting modules using the `modules=` directive of the `[compilation]` section.
+All starting modules are considered to be imported by all source files explicitly mentioned on the command line.  
+
+### Module templates
+
+If the first line of a source file starts with the `#template` directive,
+then the source is considered to be a _module template_.
+Module templates are a tool for generating repetitive code, similar to COBOL copybooks or Go Generate.
+
+The template directive contains a comma-separated list of parameters.
+It's recommended that the names of parameters begin and end with non-alphanumeric characters:
+
+    #template $P1$, $P2$
+
+A module template cannot be imported as-is.
+When importing a module template, you import a concrete instantiation of it.
+
+For example, if the file `temp.mfk` contains the `#template` from above,
+you can import it by providing a list of numeric literals or identifiers:
+
+    import temp<1, 2>
+
+This instantiates a new module named `temp<1,2>` (if it hasn't been instantiated anywhere else).
+The code in that module is generated by replacing every instance of the parameter names with the actual argument content.
+Parameters that are numeric literals are normalized to their decimal representations.
+
+Your program may contain multiple modules created from the same template with different parameters. For example,
+
+    import temp<3, 4>
+    import temp<5, 6>
+    import temp<$5, $6>
+    
+instantiates and imports two similar, yet different modules: `temp<3,4>` and `temp<5,6>`.
+The third import imports a module that has already been instantiated and imported, so it's redundant.
+
+The instantiation works through simple text replacement. For example, if `temp.mfk` contains:
+
+    #template $P1$, $P2$
+    const byte a$P1$ = $P2$
+
+then the `temp<1,2>` module will contain
+
+    const byte a1 = 2
+
+This substitution is performed before preprocessing, so those substitutions are available for the preprocessor directives.
+It applies to identifiers, string literals, keywords, preprocesor directives etc.
+
+**Warning:** This mechanism provides no direct way for preventing duplicates of code
+that does not depend on the template parameters, or depends on only some template parameters.
+In such situations, it might be advisable to put the non-dependent definitions in another module that is not a template,
+or in a module template with fewer parameters. For example, instead of writing:
+
+    #template $N$
+    const byte X = 50
+    array a$N$ [X]
+    
+(which would define duplicate `X`s if imported multiple times), consider writing two files:
+
+    #template $N$
+    import define_X
+    array a$N$ [X]
+>
+
+    //define_X.mfk:
+    const byte X = 50
+
+

docs/lang/preprocessor.md

@@ -129,6 +129,16 @@ These features are used to identify the target machine in multiplatform programs
 
 ### Built-in preprocessor functions and operators
 
+The `same` function returns 1 if given identical identifiers and 0 otherwise.
+It is the only function that does not support any other kind of parameters, and it's only useful in module templates.
+
+    // prints 1:
+    #infoeval same(a,a)
+    // prints 0:
+    #infoeval same(a,b)
+    // fails to compile
+    #infoeval same(a,1)
+
 The `defined` function returns 1 if the feature is defined, 0 otherwise.  
 All the other functions and operators treat undefined features as if they were defined as 0. 
 
@@ -145,6 +155,11 @@ TODO
 The following Millfork operators and functions are not available in the preprocessor:  
 `+'`, `-'`, `*'`, `<<'`, `>>'`, `:`, `>>>>`, `nonet`, all the assignment operators
 
+### `#template`
+
+Defines the source to be a module template. See [Modules](./modules.md) for more information.
+
+
 ### `#if/#elseif/#else/#endif`
 
     #if <expr>

examples/tests/main.mfk

@@ -2,6 +2,11 @@ import test_fibonacci
 import test_pstring
 import test_string
 import test_encconv
+#if MILLFORK_VERSION >= 000317
+import test_template<ignored1, 1>
+import test_template<ignored1, 1>
+import test_template<ignored2, 2>
+#endif
 
 void main() {
     ensure_mixedcase()

examples/tests/test_template.mfk

@@ -0,0 +1,3 @@
+#template $NAME, $VALUE
+
+const byte $NAME = $VALUE

mkdocs.yml

@@ -20,6 +20,7 @@ nav:
         - NES/Famicom: api/famicom-programming-guide.md
   - Language reference:
     - Preprocessor: lang/preprocessor.md
+    - Modules: lang/modules.md
     - Syntax: lang/syntax.md
     - Types: lang/types.md
     - Literals: lang/literals.md

src/main/scala/millfork/node/Node.scala

@@ -590,7 +590,7 @@ case class ArrayDeclarationStatement(name: String,
 case class ParameterDeclaration(typ: String,
                                 assemblyParamPassingConvention: ParamPassingConvention) extends Node
 
-case class ImportStatement(filename: String) extends DeclarationStatement {
+case class ImportStatement(filename: String, templateParams: List[String]) extends DeclarationStatement {
   override def getAllExpressions: List[Expression] = Nil
 
   override def name: String = ""

src/main/scala/millfork/parser/AbstractSourceLoadingQueue.scala

@@ -57,10 +57,10 @@ abstract class AbstractSourceLoadingQueue[T](val initialFilenames: List[String],
       moduleDependecies += standardModules(later) -> standardModules(earlier)
     }
     initialFilenames.foreach { i =>
-      parseModule(extractName(i), includePath, Right(i))
+      parseModule(extractName(i), includePath, Right(i), Nil)
     }
     options.platform.startingModules.foreach {m =>
-      moduleQueue.enqueue(() => parseModule(m, includePath, Left(None)))
+      moduleQueue.enqueue(() => parseModule(m, includePath, Left(None), Nil))
     }
     enqueueStandardModules()
     while (moduleQueue.nonEmpty) {
@@ -91,13 +91,17 @@ abstract class AbstractSourceLoadingQueue[T](val initialFilenames: List[String],
 
   def createParser(filename: String, src: String, parentDir: String, featureConstants: Map[String, Long], pragmas: Set[String]) : MfParser[T]
 
-  def parseModule(moduleName: String, includePath: List[String], why: Either[Option[Position], String]): Unit = {
+  def fullModuleName(moduleNameBase: String, templateParams: List[String]): String = {
+    if (templateParams.isEmpty) moduleNameBase else moduleNameBase + templateParams.mkString("<", ",", ">")
+  }
+
+  def parseModule(moduleName: String, includePath: List[String], why: Either[Option[Position], String], templateParams: List[String]): Unit = {
     val filename: String = why.fold(p => lookupModuleFile(includePath, moduleName, p), s => s)
     options.log.debug(s"Parsing $filename")
     val path = Paths.get(filename)
     val parentDir = path.toFile.getAbsoluteFile.getParent
     val shortFileName = path.getFileName.toString
-    val PreprocessingResult(src, featureConstants, pragmas) = Preprocessor(options, shortFileName, Files.readAllLines(path, StandardCharsets.UTF_8).toIndexedSeq)
+    val PreprocessingResult(src, featureConstants, pragmas) = Preprocessor(options, shortFileName, Files.readAllLines(path, StandardCharsets.UTF_8).toIndexedSeq, templateParams)
     for (pragma <- pragmas) {
       if (!supportedPragmas(pragma._1) && options.flag(CompilationFlag.BuggyCodeWarning)) {
         options.log.warn(s"Unsupported pragma: #pragma ${pragma._1}", Some(Position(moduleName, pragma._2, 1, 0)))
@@ -109,12 +113,12 @@ abstract class AbstractSourceLoadingQueue[T](val initialFilenames: List[String],
     parser.toAst match {
       case Success(prog, _) =>
         parsedModules.synchronized {
-          parsedModules.put(moduleName, prog)
+          parsedModules.put(fullModuleName(moduleName, templateParams), prog)
           prog.declarations.foreach {
-            case s@ImportStatement(m) =>
+            case s@ImportStatement(m, ps) =>
               moduleDependecies += moduleName -> m
-              if (!parsedModules.contains(m)) {
-                moduleQueue.enqueue(() => parseModule(m, parentDir :: includePath, Left(s.position)))
+              if (!parsedModules.contains(fullModuleName(m, ps))) {
+                moduleQueue.enqueue(() => parseModule(m, parentDir :: includePath, Left(s.position), ps))
               }
             case _ => ()
           }

src/main/scala/millfork/parser/MfParser.scala

@@ -83,8 +83,6 @@ abstract class MfParser[T](fileId: String, input: String, currentDirectory: Stri
 
   val continueStatement: P[Seq[ExecutableStatement]] = ("continue" ~ !letterOrDigit ~/ HWS ~ identifier.?).map(l => Seq(ContinueStatement(l.getOrElse(""))))
 
-  val importStatement: P[Seq[ImportStatement]] = ("import" ~ !letterOrDigit ~/ SWS ~/ identifier.rep(min = 1, sep = "/")).map(x => Seq(ImportStatement(x.mkString("/"))))
-
   val forDirection: P[ForDirection.Value] =
     ("parallel" ~ HWS ~ "to").!.map(_ => ForDirection.ParallelTo) |
       ("parallel" ~ HWS ~ "until").!.map(_ => ForDirection.ParallelUntil) |
@@ -169,6 +167,15 @@ abstract class MfParser[T](fileId: String, input: String, currentDirectory: Stri
 
   val atomWithIntel: P[Expression] = P(position() ~ (variableAtom | literalAtomWithIntel | textLiteralAtom)).map{case (p,a) => a.pos(p)}
 
+  val quotedAtom: P[String] = variableAtom.! | literalAtomWithIntel.map{
+    case LiteralExpression(value, _) => value.toString
+    case x => x.toString
+  } | textLiteralAtom.!
+
+  val importStatement: P[Seq[ImportStatement]] = ("import" ~ !letterOrDigit ~/ SWS ~/
+    identifier.rep(min = 1, sep = "/") ~ HWS ~ ("<" ~/ HWS ~/ quotedAtom.rep(min = 1, sep = HWS ~ "," ~/ HWS) ~/ HWS ~/ ">" ~/ Pass).?).
+    map{case (name, params) => Seq(ImportStatement(name.mkString("/"), params.getOrElse(Nil).toList))}
+
   val globalVariableDefinition: P[Seq[BankedDeclarationStatement]] = variableDefinition(true)
   val localVariableDefinition: P[Seq[DeclarationStatement]] = variableDefinition(false)
 

src/main/scala/millfork/parser/MosSourceLoadingQueue.scala

@@ -17,10 +17,10 @@ class MosSourceLoadingQueue(initialFilenames: List[String],
 
   def enqueueStandardModules(): Unit = {
     if (options.zpRegisterSize > 0) {
-      moduleQueue.enqueue(() => parseModule("m6502/zp_reg", includePath, Left(None)))
+      moduleQueue.enqueue(() => parseModule("m6502/zp_reg", includePath, Left(None), Nil))
     }
     if (options.zpRegisterSize >= 4 && !options.flag(CompilationFlag.DecimalMode)) {
-      moduleQueue.enqueue(() => parseModule("m6502/bcd_6502", includePath, Left(None)))
+      moduleQueue.enqueue(() => parseModule("m6502/bcd_6502", includePath, Left(None), Nil))
     }
   }
 

src/main/scala/millfork/parser/Preprocessor.scala

@@ -18,7 +18,7 @@ object Preprocessor {
   private val Regex = """\A\s*(?:#|\$\$)\s*([a-z]+)\s*(.*?)\s*\z""".r
 
   def preprocessForTest(options: CompilationOptions, code: String): PreprocessingResult = {
-    apply(options, "", code.linesIterator.toSeq)
+    apply(options, "", code.linesIterator.toSeq, Nil)
   }
 
   case class IfContext(hadEnabled: Boolean, hadElse: Boolean, enabledBefore: Boolean)
@@ -28,7 +28,7 @@ object Preprocessor {
     ! isEmpty
   }
 
-  def apply(options: CompilationOptions, shortFileName: String, lines: Seq[String]): PreprocessingResult = {
+  def apply(options: CompilationOptions, shortFileName: String, lines: Seq[String], templateParams: List[String]): PreprocessingResult = {
     val platform = options.platform
     val log = options.log
 //    if (log.traceEnabled) {
@@ -38,6 +38,7 @@ object Preprocessor {
 //    }
     val result = mutable.ListBuffer[String]()
     val featureConstants = mutable.Map[String, Long]()
+    val actualTemplateParams = mutable.Map[String, String]()
     val pragmas = mutable.Map[String, Int]()
     var enabled = true
     val ifStack = mutable.Stack[IfContext]()
@@ -69,11 +70,42 @@ object Preprocessor {
 
     for (line <- lines) {
       lineNo += 1
+      val pos = Some(Position(shortFileName, lineNo, 0, 0))
+      val lineWithParamsSubstituted: String = actualTemplateParams.foldLeft[String](line)((l, kv) => l.replace(kv._1, kv._2))
       var resulting = ""
-      line match {
+      lineWithParamsSubstituted match {
         case Regex(keyword, param) =>
-          val pos = Some(Position(shortFileName, lineNo, 0, 0))
           keyword match {
+            case "template" =>
+              if (lineNo != 1) {
+                log.error("#template should be the first line in the file", pos)
+              }
+              val paramNames = param.split(",").map(_.trim)
+              if (paramNames.length == 1 && paramNames(0).isEmpty) {
+                log.error("#template should be followed by a parameter list", pos)
+              } else if (paramNames.exists(_.isEmpty)) {
+                log.error("#template is followed by an invalid parameter list", pos)
+              } else if (paramNames.length != templateParams.length) {
+                log.error(s"#template has ${paramNames.length} parameters, but the module was instantiated with ${templateParams.length} parameters", pos)
+              }
+              if (paramNames.toSet.size != paramNames.length) {
+                log.error(s"#template has duplicate parameter names", pos)
+              }
+              for {
+                p <- paramNames
+                q <- paramNames
+                if p != q
+              } {
+                if (p.contains(q)) {
+                  log.error(s"#template has duplicate parameters whose names are contained in names of other parameters", pos)
+                }
+              }
+              for((paramName, actualParam) <- paramNames.zip(templateParams)) {
+                if (paramName.nonEmpty) {
+                  actualTemplateParams(paramName) = actualParam
+                }
+              }
+
             case "use" => if (enabled) {
               if (param == "") log.error("#use should have a parameter", pos)
               param.split("=", 2) match {
@@ -167,7 +199,10 @@ object Preprocessor {
               log.error("Invalid preprocessor directive: #" + keyword, pos)
 
           }
-        case _ => if (enabled) resulting = line.replace("\t", "    ")
+        case _ => if (enabled) resulting = lineWithParamsSubstituted.replace("\t", "    ")
+      }
+      if (lineNo == 1 && templateParams.nonEmpty && actualTemplateParams.isEmpty) {
+        log.error("A template module imported without actual parameters", pos)
       }
       result += resulting
     }
@@ -203,6 +238,16 @@ class PreprocessorParser(options: CompilationOptions) {
 
   def mfParenExpr: P[Q] = P("(" ~/ HWS ~/ mfExpression(nonStatementLevel) ~ HWS ~/ ")")
 
+  def quotedFunctionCall: P[Q] = for {
+    name <- identifier
+    _ <- HWS ~ "("
+    if name == "same"
+    params <- "" ~/ HWS ~/ identifier.rep(min = 0, sep = HWS ~ "," ~/ HWS) ~ HWS ~/ ")" ~/ ""
+  } yield (name, params.toList) match {
+    case ("same", identifiers) => _ => Some(if (identifiers.toSet.size <= 1) 1L else 0L)
+    case _ => alwaysNone
+  }
+
   def functionCall: P[Q] = for {
     name <- identifier
     params <- HWS ~ "(" ~/ HWS ~/ mfExpression(nonStatementLevel).rep(min = 0, sep = HWS ~ "," ~/ HWS) ~ HWS ~/ ")" ~/ ""
@@ -223,7 +268,7 @@ class PreprocessorParser(options: CompilationOptions) {
   }
 
 
-  def tightMfExpression: P[Q] = P(mfParenExpr | functionCall | atom) // TODO
+  def tightMfExpression: P[Q] = P(mfParenExpr | quotedFunctionCall | functionCall | atom) // TODO
 
   def tightMfExpressionButNotCall: P[Q] = P(mfParenExpr | atom) // TODO
 

src/test/resources/include/silly_template.mfk

@@ -0,0 +1,3 @@
+#template $TYPE, $NAME, $VALUE
+
+const $TYPE $NAME = $VALUE

src/test/scala/millfork/test/TemplateSuite.scala

@@ -0,0 +1,45 @@
+package millfork.test
+
+import millfork.test.emu.EmuUnoptimizedCmosRun
+import org.scalatest.{FunSuite, Matchers}
+/**
+  * @author Karol Stasiak
+  */
+class TemplateSuite extends FunSuite with Matchers {
+
+  test("Template test") {
+    val src =
+      """
+        | import silly_template<byte, a, 1>
+        | import silly_template<byte, b, 2>
+        | import silly_template<byte, b, $2>
+        | array output [30] @$c000
+        | void main() {
+        |   output[0] = a
+        |   output[1] = b
+        | }
+      """.stripMargin
+    val m = EmuUnoptimizedCmosRun(src)
+    m.readByte(0xc000) should equal(1)
+    m.readByte(0xc001) should equal(2)
+  }
+
+  test("same() test") {
+    val src =
+      """
+        |
+        | byte output @$c000
+        | void main() {
+        | #if same(a,a)
+        |   output = 1
+        | #endif
+        | #if same(a,b)
+        |   output = 2
+        | #endif
+        | }
+      """.stripMargin
+    val m = EmuUnoptimizedCmosRun(src)
+    m.readByte(0xc000) should equal(1)
+  }
+}
+

src/test/scala/millfork/test/emu/EmuRun.scala

@@ -14,7 +14,7 @@ import millfork.compiler.{CompilationContext, LabelGenerator}
 import millfork.compiler.mos.MosCompiler
 import millfork.env.{Environment, InitializedArray, InitializedMemoryVariable, NormalFunction}
 import millfork.error.Logger
-import millfork.node.{Program, StandardCallGraph}
+import millfork.node.{ImportStatement, Program, StandardCallGraph}
 import millfork.node.opt.NodeOptimization
 import millfork.output.{MemoryBank, MosAssembler}
 import millfork.parser.{MosParser, PreprocessingResult, Preprocessor}
@@ -22,6 +22,7 @@ import millfork.{CompilationFlag, CompilationOptions, CpuFamily, JobContext}
 import org.scalatest.Matchers
 
 import scala.collection.JavaConverters._
+import scala.collection.mutable
 
 /**
   * @author Karol Stasiak
@@ -183,14 +184,36 @@ class EmuRun(cpu: millfork.Cpu.Value, nodeOptimizations: List[NodeOptimization],
     parserF.toAst match {
       case Success(unoptimized, _) =>
         log.assertNoErrors("Parse failed")
-
         // prepare
+        val alreadyImported = mutable.Set[String]()
         val withLibraries = {
           var tmp = unoptimized
-          if(source.contains("import zp_reg") || source.contains("import m6502/zp_reg"))
-            tmp += EmuRun.cachedZpreg
-          if(source.contains("import stdio"))
-            tmp += EmuRun.cachedStdio
+          unoptimized.declarations.foreach {
+            case ImportStatement("zp_reg", Nil) =>
+              if (alreadyImported.add("zp_reg")) {
+                tmp += EmuRun.cachedZpreg
+              }
+            case ImportStatement("m6502/zp_reg", Nil) =>
+              if (alreadyImported.add("zp_reg")) {
+                tmp += EmuRun.cachedZpreg
+              }
+            case ImportStatement("stdio", Nil) =>
+              if (alreadyImported.add("stdio")) {
+                tmp += EmuRun.cachedStdio
+              }
+            case ImportStatement(name, params) =>
+              val fullName = if(params.isEmpty) name else name + params.mkString("<", ",", ">")
+              if (alreadyImported.add(fullName)) {
+                val source2 = Files.readAllLines(Paths.get(s"src/test/resources/include/$name.mfk"))
+                val PreprocessingResult(preprocessedSource2, _, _) = Preprocessor(options, name, source2.asScala.toList, params)
+                MosParser("", preprocessedSource2, "", options, features).toAst match {
+                  case Success(unoptimized2, _) =>
+                    tmp += unoptimized2
+                  case _ => ???
+                }
+              }
+            case _ =>
+          }
           if(!options.flag(CompilationFlag.DecimalMode) && (source.contains("+'") || source.contains("-'") || source.contains("<<'") || source.contains("*'")))
             tmp += EmuRun.cachedBcd
           tmp