Update comments and fix some bugs

make_edit_distance
- use MASKED_DOTS since it does not have a simple relationship to the
  HEADER_BITS/BODY_BITS for HGR
- try disabling transposition distances for Damerau-Levenshtein, this
  may give better quality

screen
- introduce separate notion of MASKED_DOTS which is the number of
  (coloured) pixels we can extract from MASKED_BITS.  For HGR this is
  not the same.
- fix bug in _fix_array_neighbours that was not fixing headers for HGR
- don't cache everything in byte_pair_differences, it's effectively
  unbounded.  Using 1M for LRU size seems to work just as well in
  practise, without leaking memory.
- fix bug in _diff_weights when comparing content, we want to evaluate
  the effect of storing content byte in each offset separately, not
  cumulatively.
- add a consistency check function (not currently wired up) to assert
  that headers/footers are in sync across columns
- HGR should have 16 body bits, this was causing headers not to
  propagate correctly to/from neighbouring column
- add test case for this bug

video
- Use 8 random bits consistently, using 16 in some places may have
  introduced bias
- ignore palette bit when comparing 0x00 and 0x7f in sanity check

transcoder/video_test.py

@@ -67,11 +67,11 @@ class TestVideo(unittest.TestCase):
 
         diff = target_pixelmap.diff_weights(v.pixelmap, is_aux=True)
 
-        # Expect byte 0 to map to 0b01111111 01101101 XXX
+        # Masked offset 0 changes from 0001111111000 to 0001101101000
         expect0 = target_pixelmap.edit_distances(pal.ID)[0][
             0b00011111110000001101101000]
 
-        # Expect byte 2 to map to 0b000101010100 000011011000
+        # Masked offset 2 changes from 0001010101000 to 0000110110000
         expect2 = target_pixelmap.edit_distances(pal.ID)[2][
             0b00010101010000000110110000]