1
0
mirror of https://github.com/cc65/cc65.git synced 2024-12-23 04:30:10 +00:00
cc65/test
Spiro Trikaliotis d7a804d120 Split stdin and stdout in test/asm/
Some tests pipe the stdout and stderr of the compiler or assembler into
a file and compare the result with a reference. This has the subtle
problem that both stdout and stderr and buffered i/o streams that may or
may not work the same on different OSs or even shells. That means the
order of the lines in the resulting file is not guaranteed.

In practise it is "mostly" not an issue, but "we" still stumbled about
one test where different behaviour can be triggered depending on running
it in cmd.exe or bash (msys): test/asm/listing output of
010-paramcount.bin differs from the reference output when running the
test from cmd.exe.

The solution is most likely to have two reference files and not redirect
into one file.

This patch fixes the issue (cf. #1912) for the directory test/asm/.
2022-11-12 23:12:25 +01:00
..
asm Split stdin and stdout in test/asm/ 2022-11-12 23:12:25 +01:00
dasm Streamlined some makefiles. 2020-11-17 13:34:22 -05:00
err add check for line endings containing CR. 2022-11-03 18:27:10 +01:00
misc move test to test/ref 2022-11-11 21:22:20 +01:00
ref move test to test/ref 2022-11-11 21:22:20 +01:00
standard Add a directory for tests that exercise the supported C standard differences. 2022-05-08 19:29:57 +02:00
todo Clarify sprintf-test's output so it doesn't look like the entire test bench summary 2022-10-12 18:38:33 +03:00
val Merge pull request #1905 from acqn/DeclFix 2022-11-05 16:03:09 +01:00
CPYRIGHT.LCC Add test license description. 2020-06-27 02:23:40 +02:00
isequal.c isequal with --wildcard: Allow DOS/Win paths 2022-06-22 21:23:44 +02:00
Makefile Adjusted to 5df2de0629 (and slightly simplified). 2020-07-27 17:47:14 +02:00
readme.txt added examples for tests that compare the compiler output with a reference 2022-11-03 17:40:13 +01:00

This directory contains test code for automatic regression testing of the CC65
compiler and tools.

/asm  - contains the assembler regression tests

/dasm - contains the disassembler regression tests


/val, /ref and /err generally contain the tests that are used to verify that the
compiler is working as expected (when the tests behave as described):

/val  - The bulk of tests are contained here, individual tests should exit with
        an exit code of EXIT_SUCCESS when they pass, or EXIT_FAILURE on error.

/standard - like the tests in /val, the tests must exit with EXIT_SUCCESS on
        success. Unlike the tests in /val these are not compiled for every
        combination of optimizer options, but instead always with -Osir and then
        for each supported C-standard (C89, C99, CC65). The goal is to use these
        to check for regressions in standard conformance of the compiler and the
        library.

/ref  - These tests produce output that must be compared with reference output.
        Normally the reference output is produced by compiling the program on the
        host (using gcc mostly) and then running them on the host. Tests should
        be tweaked to produce the same output as on the host in the cases where
        it would be different.

        The Makefile also handles some special cases (add the tests to the
        respective list in the makefile):

        - Sometimes we want to check the warnings produced by the compiler. In
        that case use the CUSTOMSOURCES list. Whatever output the compiler writes
        to stderr will be compared against the matching .cref file. There is an
        example in custom-reference.c/.cref

        - Sometimes we want to check what kind of output the compiler produces
        for a file that does not compile. In that case use the ERRORSOURCES list.
        There is an example in custom-reference-error.c/.cref

        Warning: please understand that comparing the compiler output against
        a reference produces a moving target, ie the tests may break randomly
        at any time when the compiler output changes for whatever reason. So
        only ever use this as a last resort when something can not be tested by
        other means.

/err  - contains tests that MUST NOT compile


/todo and /misc generally contain the tests that fail because of known bugs:

/todo - These tests fail due to open compiler issues.

        The makefile in this directory _expects_ the tests to fail, because of
        that when an issue was fixed it will break the CI. The test should get 
        moved to /val in the PR fixing the issue, which will make CI pass again.
        No changes to makefiles are required!


/misc - a few tests that need special care of some sort

        Tests that (incorrectly) fail to compile and other tests that fail and
        do NOT return an exit code are collected here. The makefile _expects_
        those tests to fail, so when an issue is fixed it will break the CI.
        When this happens, the PR fixing the issue should also "invert" the
        failing condition in the makefile by adding a $(NOT) before the
        offending line (or removing it when it is already there), which will
        make the CI pass again. The test should then be moved elsewhere later,
        which will require additional changes to the makefile(s).


To run the tests use "make" in this (top) directory, the makefile should exit
with no error.

When a test failed you can use "make continue" to run further tests.

--------------------------------------------------------------------------------

TODO:

- reduce usage of "common.h" to a minimum
- convert more tests from using reference output to returning an exit code