mirror of
https://github.com/sheumann/hush.git
synced 2024-08-14 11:28:58 +00:00
272 lines
20 KiB
Plaintext
272 lines
20 KiB
Plaintext
|
UTF-8 decoder capability and stress test
|
|||
|
----------------------------------------
|
|||
|
|
|||
|
Markus Kuhn <http://www.cl.cam.ac.uk/~mgk25/> - 2003-02-19
|
|||
|
|
|||
|
This test file can help you examine, how your UTF-8 decoder handles
|
|||
|
various types of correct, malformed, or otherwise interesting UTF-8
|
|||
|
sequences. This file is not meant to be a conformance test. It does
|
|||
|
not prescribes any particular outcome and therefore there is no way to
|
|||
|
"pass" or "fail" this test file, even though the texts suggests a
|
|||
|
preferable decoder behaviour at some places. The aim is instead to
|
|||
|
help you think about and test the behaviour of your UTF-8 on a
|
|||
|
systematic collection of unusual inputs. Experience so far suggests
|
|||
|
that most first-time authors of UTF-8 decoders find at least one
|
|||
|
serious problem in their decoder by using this file.
|
|||
|
|
|||
|
The test lines below cover boundary conditions, malformed UTF-8
|
|||
|
sequences as well as correctly encoded UTF-8 sequences of Unicode code
|
|||
|
points that should never occur in a correct UTF-8 file.
|
|||
|
|
|||
|
According to ISO 10646-1:2000, sections D.7 and 2.3c, a device
|
|||
|
receiving UTF-8 shall interpret a "malformed sequence in the same way
|
|||
|
that it interprets a character that is outside the adopted subset" and
|
|||
|
"characters that are not within the adopted subset shall be indicated
|
|||
|
to the user" by a receiving device. A quite commonly used approach in
|
|||
|
UTF-8 decoders is to replace any malformed UTF-8 sequence by a
|
|||
|
replacement character (U+FFFD), which looks a bit like an inverted
|
|||
|
question mark, or a similar symbol. It might be a good idea to
|
|||
|
visually distinguish a malformed UTF-8 sequence from a correctly
|
|||
|
encoded Unicode character that is just not available in the current
|
|||
|
font but otherwise fully legal, even though ISO 10646-1 doesn't
|
|||
|
mandate this. In any case, just ignoring malformed sequences or
|
|||
|
unavailable characters does not conform to ISO 10646, will make
|
|||
|
debugging more difficult, and can lead to user confusion.
|
|||
|
|
|||
|
Please check, whether a malformed UTF-8 sequence is (1) represented at
|
|||
|
all, (2) represented by exactly one single replacement character (or
|
|||
|
equivalent signal), and (3) the following quotation mark after an
|
|||
|
illegal UTF-8 sequence is correctly displayed, i.e. proper
|
|||
|
resynchronization takes place immageately after any malformed
|
|||
|
sequence. This file says "THE END" in the last line, so if you don't
|
|||
|
see that, your decoder crashed somehow before, which should always be
|
|||
|
cause for concern.
|
|||
|
|
|||
|
All lines in this file are exactly 79 characters long (plus the line
|
|||
|
feed). In addition, all lines end with "|", except for the two test
|
|||
|
lines 2.1.1 and 2.2.1, which contain non-printable ASCII controls
|
|||
|
U+0000 and U+007F. If you display this file with a fixed-width font,
|
|||
|
these "|" characters should all line up in column 79 (right margin).
|
|||
|
This allows you to test quickly, whether your UTF-8 decoder finds the
|
|||
|
correct number of characters in every line, that is whether each
|
|||
|
malformed sequences is replaced by a single replacement character.
|
|||
|
|
|||
|
Note that as an alternative to the notion of malformed sequence used
|
|||
|
here, it is also a perfectly acceptable (and in some situations even
|
|||
|
preferable) solution to represent each individual byte of a malformed
|
|||
|
sequence by a replacement character. If you follow this strategy in
|
|||
|
your decoder, then please ignore the "|" column.
|
|||
|
|
|||
|
|
|||
|
Here come the tests: |
|
|||
|
|
|
|||
|
1 Some correct UTF-8 text |
|
|||
|
|
|
|||
|
You should see the Greek word 'kosme': "κόσμε" |
|
|||
|
|
|
|||
|
2 Boundary condition test cases |
|
|||
|
|
|
|||
|
2.1 First possible sequence of a certain length |
|
|||
|
|
|
|||
|
2.1.1 1 byte (U-00000000): " |