From 718245c56a6bc94a4cd8b2b97cd94c0718ea3c23 Mon Sep 17 00:00:00 2001 From: Karol Stasiak Date: Wed, 3 Jun 2020 23:13:17 +0200 Subject: [PATCH] Module templates --- docs/README.md | 2 + docs/api/custom-platform.md | 3 +- docs/doc_index.md | 2 + docs/lang/modules.md | 97 +++++++++++++++++++ docs/lang/preprocessor.md | 15 +++ examples/tests/main.mfk | 5 + examples/tests/test_template.mfk | 3 + mkdocs.yml | 1 + src/main/scala/millfork/node/Node.scala | 2 +- .../parser/AbstractSourceLoadingQueue.scala | 20 ++-- src/main/scala/millfork/parser/MfParser.scala | 11 ++- .../parser/MosSourceLoadingQueue.scala | 4 +- .../scala/millfork/parser/Preprocessor.scala | 57 +++++++++-- src/test/resources/include/silly_template.mfk | 3 + .../scala/millfork/test/TemplateSuite.scala | 45 +++++++++ src/test/scala/millfork/test/emu/EmuRun.scala | 35 +++++-- 16 files changed, 279 insertions(+), 26 deletions(-) create mode 100644 docs/lang/modules.md create mode 100644 examples/tests/test_template.mfk create mode 100644 src/test/resources/include/silly_template.mfk create mode 100644 src/test/scala/millfork/test/TemplateSuite.scala diff --git a/docs/README.md b/docs/README.md index 18331ce5..0ce8210f 100644 --- a/docs/README.md +++ b/docs/README.md @@ -16,6 +16,8 @@ * [Preprocessor](lang/preprocessor.md) +* [Modules](lang/modules.md) + * [Syntax](lang/syntax.md) * [Types](lang/types.md) diff --git a/docs/api/custom-platform.md b/docs/api/custom-platform.md index 4a0ce838..de0d0ca6 100644 --- a/docs/api/custom-platform.md +++ b/docs/api/custom-platform.md @@ -60,7 +60,8 @@ See [the list of available encodings](../lang/text.md). * `screen_encoding` – default encoding for screencodes (literals with encoding specified as `scr`). Default: the same as `encoding`. -* `modules` – comma-separated list of modules that will be automatically imported +* `modules` – comma-separated list of modules that will be automatically imported. +This list cannot contain module template instantiations. * other compilation options (they can be overridden using commandline options): diff --git a/docs/doc_index.md b/docs/doc_index.md index 7eef2cfb..7450d0cd 100644 --- a/docs/doc_index.md +++ b/docs/doc_index.md @@ -16,6 +16,8 @@ * [Preprocessor](lang/preprocessor.md) +* [Modules](lang/modules.md) + * [Syntax](lang/syntax.md) * [Types](lang/types.md) diff --git a/docs/lang/modules.md b/docs/lang/modules.md new file mode 100644 index 00000000..f05a3c50 --- /dev/null +++ b/docs/lang/modules.md @@ -0,0 +1,97 @@ +[< back to index](../doc_index.md) + +# Program structure + +A Millfork program is build from one or more modules. + +Each module is stored in a single file. +All source filenames passed to the compiler are considered to be modules of that program, called _root modules_. + +Each module has a name, which is its unique identifier. +A module name is a sequence of slash-separated valid Millfork identifiers. +The name also defines where the module is located: +a module named `a/b` is presumed to exist in `a/b.mfk` +and it's looked up first in the current working directory, +and then in the include directories. + +A module can import other modules, using the `import` statement. +Importing the same module multiple times merely marks it as imported by multiple modules, +but the program will still contain only one copy of it. +Examples: + + import string + import cbm_file + +Usually, the imported module will undergo the first phase of compilation first. +This means that the constants in the imported module will be resolved first, allowing you to use them in the importing module. + + +The only exception to this rule is when the importing graph has a cycle, in which case the order of modules within the cycle is unspecified. + +A platform may define starting modules using the `modules=` directive of the `[compilation]` section. +All starting modules are considered to be imported by all source files explicitly mentioned on the command line. + +### Module templates + +If the first line of a source file starts with the `#template` directive, +then the source is considered to be a _module template_. +Module templates are a tool for generating repetitive code, similar to COBOL copybooks or Go Generate. + +The template directive contains a comma-separated list of parameters. +It's recommended that the names of parameters begin and end with non-alphanumeric characters: + + #template $P1$, $P2$ + +A module template cannot be imported as-is. +When importing a module template, you import a concrete instantiation of it. + +For example, if the file `temp.mfk` contains the `#template` from above, +you can import it by providing a list of numeric literals or identifiers: + + import temp<1, 2> + +This instantiates a new module named `temp<1,2>` (if it hasn't been instantiated anywhere else). +The code in that module is generated by replacing every instance of the parameter names with the actual argument content. +Parameters that are numeric literals are normalized to their decimal representations. + +Your program may contain multiple modules created from the same template with different parameters. For example, + + import temp<3, 4> + import temp<5, 6> + import temp<$5, $6> + +instantiates and imports two similar, yet different modules: `temp<3,4>` and `temp<5,6>`. +The third import imports a module that has already been instantiated and imported, so it's redundant. + +The instantiation works through simple text replacement. For example, if `temp.mfk` contains: + + #template $P1$, $P2$ + const byte a$P1$ = $P2$ + +then the `temp<1,2>` module will contain + + const byte a1 = 2 + +This substitution is performed before preprocessing, so those substitutions are available for the preprocessor directives. +It applies to identifiers, string literals, keywords, preprocesor directives etc. + +**Warning:** This mechanism provides no direct way for preventing duplicates of code +that does not depend on the template parameters, or depends on only some template parameters. +In such situations, it might be advisable to put the non-dependent definitions in another module that is not a template, +or in a module template with fewer parameters. For example, instead of writing: + + #template $N$ + const byte X = 50 + array a$N$ [X] + +(which would define duplicate `X`s if imported multiple times), consider writing two files: + + #template $N$ + import define_X + array a$N$ [X] +> + + //define_X.mfk: + const byte X = 50 + + diff --git a/docs/lang/preprocessor.md b/docs/lang/preprocessor.md index ba2f0cc8..66154206 100644 --- a/docs/lang/preprocessor.md +++ b/docs/lang/preprocessor.md @@ -129,6 +129,16 @@ These features are used to identify the target machine in multiplatform programs ### Built-in preprocessor functions and operators +The `same` function returns 1 if given identical identifiers and 0 otherwise. +It is the only function that does not support any other kind of parameters, and it's only useful in module templates. + + // prints 1: + #infoeval same(a,a) + // prints 0: + #infoeval same(a,b) + // fails to compile + #infoeval same(a,1) + The `defined` function returns 1 if the feature is defined, 0 otherwise. All the other functions and operators treat undefined features as if they were defined as 0. @@ -145,6 +155,11 @@ TODO The following Millfork operators and functions are not available in the preprocessor: `+'`, `-'`, `*'`, `<<'`, `>>'`, `:`, `>>>>`, `nonet`, all the assignment operators +### `#template` + +Defines the source to be a module template. See [Modules](./modules.md) for more information. + + ### `#if/#elseif/#else/#endif` #if diff --git a/examples/tests/main.mfk b/examples/tests/main.mfk index 7ff7e3af..acd33b06 100644 --- a/examples/tests/main.mfk +++ b/examples/tests/main.mfk @@ -2,6 +2,11 @@ import test_fibonacci import test_pstring import test_string import test_encconv +#if MILLFORK_VERSION >= 000317 +import test_template +import test_template +import test_template +#endif void main() { ensure_mixedcase() diff --git a/examples/tests/test_template.mfk b/examples/tests/test_template.mfk new file mode 100644 index 00000000..4122d008 --- /dev/null +++ b/examples/tests/test_template.mfk @@ -0,0 +1,3 @@ +#template $NAME, $VALUE + +const byte $NAME = $VALUE diff --git a/mkdocs.yml b/mkdocs.yml index 1d163961..764a9902 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -20,6 +20,7 @@ nav: - NES/Famicom: api/famicom-programming-guide.md - Language reference: - Preprocessor: lang/preprocessor.md + - Modules: lang/modules.md - Syntax: lang/syntax.md - Types: lang/types.md - Literals: lang/literals.md diff --git a/src/main/scala/millfork/node/Node.scala b/src/main/scala/millfork/node/Node.scala index b8e37b1c..66dca333 100644 --- a/src/main/scala/millfork/node/Node.scala +++ b/src/main/scala/millfork/node/Node.scala @@ -590,7 +590,7 @@ case class ArrayDeclarationStatement(name: String, case class ParameterDeclaration(typ: String, assemblyParamPassingConvention: ParamPassingConvention) extends Node -case class ImportStatement(filename: String) extends DeclarationStatement { +case class ImportStatement(filename: String, templateParams: List[String]) extends DeclarationStatement { override def getAllExpressions: List[Expression] = Nil override def name: String = "" diff --git a/src/main/scala/millfork/parser/AbstractSourceLoadingQueue.scala b/src/main/scala/millfork/parser/AbstractSourceLoadingQueue.scala index fb255394..35b54eb4 100644 --- a/src/main/scala/millfork/parser/AbstractSourceLoadingQueue.scala +++ b/src/main/scala/millfork/parser/AbstractSourceLoadingQueue.scala @@ -57,10 +57,10 @@ abstract class AbstractSourceLoadingQueue[T](val initialFilenames: List[String], moduleDependecies += standardModules(later) -> standardModules(earlier) } initialFilenames.foreach { i => - parseModule(extractName(i), includePath, Right(i)) + parseModule(extractName(i), includePath, Right(i), Nil) } options.platform.startingModules.foreach {m => - moduleQueue.enqueue(() => parseModule(m, includePath, Left(None))) + moduleQueue.enqueue(() => parseModule(m, includePath, Left(None), Nil)) } enqueueStandardModules() while (moduleQueue.nonEmpty) { @@ -91,13 +91,17 @@ abstract class AbstractSourceLoadingQueue[T](val initialFilenames: List[String], def createParser(filename: String, src: String, parentDir: String, featureConstants: Map[String, Long], pragmas: Set[String]) : MfParser[T] - def parseModule(moduleName: String, includePath: List[String], why: Either[Option[Position], String]): Unit = { + def fullModuleName(moduleNameBase: String, templateParams: List[String]): String = { + if (templateParams.isEmpty) moduleNameBase else moduleNameBase + templateParams.mkString("<", ",", ">") + } + + def parseModule(moduleName: String, includePath: List[String], why: Either[Option[Position], String], templateParams: List[String]): Unit = { val filename: String = why.fold(p => lookupModuleFile(includePath, moduleName, p), s => s) options.log.debug(s"Parsing $filename") val path = Paths.get(filename) val parentDir = path.toFile.getAbsoluteFile.getParent val shortFileName = path.getFileName.toString - val PreprocessingResult(src, featureConstants, pragmas) = Preprocessor(options, shortFileName, Files.readAllLines(path, StandardCharsets.UTF_8).toIndexedSeq) + val PreprocessingResult(src, featureConstants, pragmas) = Preprocessor(options, shortFileName, Files.readAllLines(path, StandardCharsets.UTF_8).toIndexedSeq, templateParams) for (pragma <- pragmas) { if (!supportedPragmas(pragma._1) && options.flag(CompilationFlag.BuggyCodeWarning)) { options.log.warn(s"Unsupported pragma: #pragma ${pragma._1}", Some(Position(moduleName, pragma._2, 1, 0))) @@ -109,12 +113,12 @@ abstract class AbstractSourceLoadingQueue[T](val initialFilenames: List[String], parser.toAst match { case Success(prog, _) => parsedModules.synchronized { - parsedModules.put(moduleName, prog) + parsedModules.put(fullModuleName(moduleName, templateParams), prog) prog.declarations.foreach { - case s@ImportStatement(m) => + case s@ImportStatement(m, ps) => moduleDependecies += moduleName -> m - if (!parsedModules.contains(m)) { - moduleQueue.enqueue(() => parseModule(m, parentDir :: includePath, Left(s.position))) + if (!parsedModules.contains(fullModuleName(m, ps))) { + moduleQueue.enqueue(() => parseModule(m, parentDir :: includePath, Left(s.position), ps)) } case _ => () } diff --git a/src/main/scala/millfork/parser/MfParser.scala b/src/main/scala/millfork/parser/MfParser.scala index 73a0f0fb..fcbe3987 100644 --- a/src/main/scala/millfork/parser/MfParser.scala +++ b/src/main/scala/millfork/parser/MfParser.scala @@ -83,8 +83,6 @@ abstract class MfParser[T](fileId: String, input: String, currentDirectory: Stri val continueStatement: P[Seq[ExecutableStatement]] = ("continue" ~ !letterOrDigit ~/ HWS ~ identifier.?).map(l => Seq(ContinueStatement(l.getOrElse("")))) - val importStatement: P[Seq[ImportStatement]] = ("import" ~ !letterOrDigit ~/ SWS ~/ identifier.rep(min = 1, sep = "/")).map(x => Seq(ImportStatement(x.mkString("/")))) - val forDirection: P[ForDirection.Value] = ("parallel" ~ HWS ~ "to").!.map(_ => ForDirection.ParallelTo) | ("parallel" ~ HWS ~ "until").!.map(_ => ForDirection.ParallelUntil) | @@ -169,6 +167,15 @@ abstract class MfParser[T](fileId: String, input: String, currentDirectory: Stri val atomWithIntel: P[Expression] = P(position() ~ (variableAtom | literalAtomWithIntel | textLiteralAtom)).map{case (p,a) => a.pos(p)} + val quotedAtom: P[String] = variableAtom.! | literalAtomWithIntel.map{ + case LiteralExpression(value, _) => value.toString + case x => x.toString + } | textLiteralAtom.! + + val importStatement: P[Seq[ImportStatement]] = ("import" ~ !letterOrDigit ~/ SWS ~/ + identifier.rep(min = 1, sep = "/") ~ HWS ~ ("<" ~/ HWS ~/ quotedAtom.rep(min = 1, sep = HWS ~ "," ~/ HWS) ~/ HWS ~/ ">" ~/ Pass).?). + map{case (name, params) => Seq(ImportStatement(name.mkString("/"), params.getOrElse(Nil).toList))} + val globalVariableDefinition: P[Seq[BankedDeclarationStatement]] = variableDefinition(true) val localVariableDefinition: P[Seq[DeclarationStatement]] = variableDefinition(false) diff --git a/src/main/scala/millfork/parser/MosSourceLoadingQueue.scala b/src/main/scala/millfork/parser/MosSourceLoadingQueue.scala index e946fcc3..fec13023 100644 --- a/src/main/scala/millfork/parser/MosSourceLoadingQueue.scala +++ b/src/main/scala/millfork/parser/MosSourceLoadingQueue.scala @@ -17,10 +17,10 @@ class MosSourceLoadingQueue(initialFilenames: List[String], def enqueueStandardModules(): Unit = { if (options.zpRegisterSize > 0) { - moduleQueue.enqueue(() => parseModule("m6502/zp_reg", includePath, Left(None))) + moduleQueue.enqueue(() => parseModule("m6502/zp_reg", includePath, Left(None), Nil)) } if (options.zpRegisterSize >= 4 && !options.flag(CompilationFlag.DecimalMode)) { - moduleQueue.enqueue(() => parseModule("m6502/bcd_6502", includePath, Left(None))) + moduleQueue.enqueue(() => parseModule("m6502/bcd_6502", includePath, Left(None), Nil)) } } diff --git a/src/main/scala/millfork/parser/Preprocessor.scala b/src/main/scala/millfork/parser/Preprocessor.scala index 2472a7d0..30da6021 100644 --- a/src/main/scala/millfork/parser/Preprocessor.scala +++ b/src/main/scala/millfork/parser/Preprocessor.scala @@ -18,7 +18,7 @@ object Preprocessor { private val Regex = """\A\s*(?:#|\$\$)\s*([a-z]+)\s*(.*?)\s*\z""".r def preprocessForTest(options: CompilationOptions, code: String): PreprocessingResult = { - apply(options, "", code.linesIterator.toSeq) + apply(options, "", code.linesIterator.toSeq, Nil) } case class IfContext(hadEnabled: Boolean, hadElse: Boolean, enabledBefore: Boolean) @@ -28,7 +28,7 @@ object Preprocessor { ! isEmpty } - def apply(options: CompilationOptions, shortFileName: String, lines: Seq[String]): PreprocessingResult = { + def apply(options: CompilationOptions, shortFileName: String, lines: Seq[String], templateParams: List[String]): PreprocessingResult = { val platform = options.platform val log = options.log // if (log.traceEnabled) { @@ -38,6 +38,7 @@ object Preprocessor { // } val result = mutable.ListBuffer[String]() val featureConstants = mutable.Map[String, Long]() + val actualTemplateParams = mutable.Map[String, String]() val pragmas = mutable.Map[String, Int]() var enabled = true val ifStack = mutable.Stack[IfContext]() @@ -69,11 +70,42 @@ object Preprocessor { for (line <- lines) { lineNo += 1 + val pos = Some(Position(shortFileName, lineNo, 0, 0)) + val lineWithParamsSubstituted: String = actualTemplateParams.foldLeft[String](line)((l, kv) => l.replace(kv._1, kv._2)) var resulting = "" - line match { + lineWithParamsSubstituted match { case Regex(keyword, param) => - val pos = Some(Position(shortFileName, lineNo, 0, 0)) keyword match { + case "template" => + if (lineNo != 1) { + log.error("#template should be the first line in the file", pos) + } + val paramNames = param.split(",").map(_.trim) + if (paramNames.length == 1 && paramNames(0).isEmpty) { + log.error("#template should be followed by a parameter list", pos) + } else if (paramNames.exists(_.isEmpty)) { + log.error("#template is followed by an invalid parameter list", pos) + } else if (paramNames.length != templateParams.length) { + log.error(s"#template has ${paramNames.length} parameters, but the module was instantiated with ${templateParams.length} parameters", pos) + } + if (paramNames.toSet.size != paramNames.length) { + log.error(s"#template has duplicate parameter names", pos) + } + for { + p <- paramNames + q <- paramNames + if p != q + } { + if (p.contains(q)) { + log.error(s"#template has duplicate parameters whose names are contained in names of other parameters", pos) + } + } + for((paramName, actualParam) <- paramNames.zip(templateParams)) { + if (paramName.nonEmpty) { + actualTemplateParams(paramName) = actualParam + } + } + case "use" => if (enabled) { if (param == "") log.error("#use should have a parameter", pos) param.split("=", 2) match { @@ -167,7 +199,10 @@ object Preprocessor { log.error("Invalid preprocessor directive: #" + keyword, pos) } - case _ => if (enabled) resulting = line.replace("\t", " ") + case _ => if (enabled) resulting = lineWithParamsSubstituted.replace("\t", " ") + } + if (lineNo == 1 && templateParams.nonEmpty && actualTemplateParams.isEmpty) { + log.error("A template module imported without actual parameters", pos) } result += resulting } @@ -203,6 +238,16 @@ class PreprocessorParser(options: CompilationOptions) { def mfParenExpr: P[Q] = P("(" ~/ HWS ~/ mfExpression(nonStatementLevel) ~ HWS ~/ ")") + def quotedFunctionCall: P[Q] = for { + name <- identifier + _ <- HWS ~ "(" + if name == "same" + params <- "" ~/ HWS ~/ identifier.rep(min = 0, sep = HWS ~ "," ~/ HWS) ~ HWS ~/ ")" ~/ "" + } yield (name, params.toList) match { + case ("same", identifiers) => _ => Some(if (identifiers.toSet.size <= 1) 1L else 0L) + case _ => alwaysNone + } + def functionCall: P[Q] = for { name <- identifier params <- HWS ~ "(" ~/ HWS ~/ mfExpression(nonStatementLevel).rep(min = 0, sep = HWS ~ "," ~/ HWS) ~ HWS ~/ ")" ~/ "" @@ -223,7 +268,7 @@ class PreprocessorParser(options: CompilationOptions) { } - def tightMfExpression: P[Q] = P(mfParenExpr | functionCall | atom) // TODO + def tightMfExpression: P[Q] = P(mfParenExpr | quotedFunctionCall | functionCall | atom) // TODO def tightMfExpressionButNotCall: P[Q] = P(mfParenExpr | atom) // TODO diff --git a/src/test/resources/include/silly_template.mfk b/src/test/resources/include/silly_template.mfk new file mode 100644 index 00000000..76c49981 --- /dev/null +++ b/src/test/resources/include/silly_template.mfk @@ -0,0 +1,3 @@ +#template $TYPE, $NAME, $VALUE + +const $TYPE $NAME = $VALUE diff --git a/src/test/scala/millfork/test/TemplateSuite.scala b/src/test/scala/millfork/test/TemplateSuite.scala new file mode 100644 index 00000000..c704e13e --- /dev/null +++ b/src/test/scala/millfork/test/TemplateSuite.scala @@ -0,0 +1,45 @@ +package millfork.test + +import millfork.test.emu.EmuUnoptimizedCmosRun +import org.scalatest.{FunSuite, Matchers} +/** + * @author Karol Stasiak + */ +class TemplateSuite extends FunSuite with Matchers { + + test("Template test") { + val src = + """ + | import silly_template + | import silly_template + | import silly_template + | array output [30] @$c000 + | void main() { + | output[0] = a + | output[1] = b + | } + """.stripMargin + val m = EmuUnoptimizedCmosRun(src) + m.readByte(0xc000) should equal(1) + m.readByte(0xc001) should equal(2) + } + + test("same() test") { + val src = + """ + | + | byte output @$c000 + | void main() { + | #if same(a,a) + | output = 1 + | #endif + | #if same(a,b) + | output = 2 + | #endif + | } + """.stripMargin + val m = EmuUnoptimizedCmosRun(src) + m.readByte(0xc000) should equal(1) + } +} + diff --git a/src/test/scala/millfork/test/emu/EmuRun.scala b/src/test/scala/millfork/test/emu/EmuRun.scala index ae331fa9..40ad1e4d 100644 --- a/src/test/scala/millfork/test/emu/EmuRun.scala +++ b/src/test/scala/millfork/test/emu/EmuRun.scala @@ -14,7 +14,7 @@ import millfork.compiler.{CompilationContext, LabelGenerator} import millfork.compiler.mos.MosCompiler import millfork.env.{Environment, InitializedArray, InitializedMemoryVariable, NormalFunction} import millfork.error.Logger -import millfork.node.{Program, StandardCallGraph} +import millfork.node.{ImportStatement, Program, StandardCallGraph} import millfork.node.opt.NodeOptimization import millfork.output.{MemoryBank, MosAssembler} import millfork.parser.{MosParser, PreprocessingResult, Preprocessor} @@ -22,6 +22,7 @@ import millfork.{CompilationFlag, CompilationOptions, CpuFamily, JobContext} import org.scalatest.Matchers import scala.collection.JavaConverters._ +import scala.collection.mutable /** * @author Karol Stasiak @@ -183,14 +184,36 @@ class EmuRun(cpu: millfork.Cpu.Value, nodeOptimizations: List[NodeOptimization], parserF.toAst match { case Success(unoptimized, _) => log.assertNoErrors("Parse failed") - // prepare + val alreadyImported = mutable.Set[String]() val withLibraries = { var tmp = unoptimized - if(source.contains("import zp_reg") || source.contains("import m6502/zp_reg")) - tmp += EmuRun.cachedZpreg - if(source.contains("import stdio")) - tmp += EmuRun.cachedStdio + unoptimized.declarations.foreach { + case ImportStatement("zp_reg", Nil) => + if (alreadyImported.add("zp_reg")) { + tmp += EmuRun.cachedZpreg + } + case ImportStatement("m6502/zp_reg", Nil) => + if (alreadyImported.add("zp_reg")) { + tmp += EmuRun.cachedZpreg + } + case ImportStatement("stdio", Nil) => + if (alreadyImported.add("stdio")) { + tmp += EmuRun.cachedStdio + } + case ImportStatement(name, params) => + val fullName = if(params.isEmpty) name else name + params.mkString("<", ",", ">") + if (alreadyImported.add(fullName)) { + val source2 = Files.readAllLines(Paths.get(s"src/test/resources/include/$name.mfk")) + val PreprocessingResult(preprocessedSource2, _, _) = Preprocessor(options, name, source2.asScala.toList, params) + MosParser("", preprocessedSource2, "", options, features).toAst match { + case Success(unoptimized2, _) => + tmp += unoptimized2 + case _ => ??? + } + } + case _ => + } if(!options.flag(CompilationFlag.DecimalMode) && (source.contains("+'") || source.contains("-'") || source.contains("<<'") || source.contains("*'"))) tmp += EmuRun.cachedBcd tmp