Apple-2-SW/ii-vision
1
0
Fork 0
mirror of https://github.com/KrisKennaway/ii-vision.git synced 2024-09-27 04:54:59 +00:00
Code Issues Projects Releases Wiki Activity
d5f2482a0a
ii-vision/transcoder/video.py

504 lines
17 KiB
Python
Raw Normal View History

Add module docstrings
2019-03-21 16:42:47 +00:00
"""Encode a sequence of images as an optimized stream of screen changes."""
Add a video Mode enum.
2019-04-25 15:28:25 +00:00
import enum
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
import functools
Integrated audio + video player! - Introduce a new Movie() class that multiplexes audio and video. - Every N audio frames we grab a new video frame and begin pulling opcodes from the audio and video streams - Grab frames from the input video using bmp2dhr if the .BIN file does not already exist. Run bmp2dhr in a background thread to not block encoding - move the output byte streaming from Video to Movie - For now, manually clip updates to pages > 56 since the client doesn't support them yet The way we encode video is now: - iterate in descending order over update_priority - begin a new (page, content) opcode - for all of the other offset bytes in that page, compute the error between the candidate content byte and the target content byte - iterate over offsets in order of increasing error and decreasing update_priority to fill out the remaining opcode
2019-03-07 23:07:24 +00:00
import heapq
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
import os
import queue
Fix imports and point to player debug symbols
2019-03-21 16:24:40 +00:00
import random
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
import subprocess
Fix imports and point to player debug symbols
2019-03-21 16:24:40 +00:00
import threading
Vectorize the computation of diff weights, by precomputing a map of all possible weights. We encode the two 8-bit inputs into a single 16 bit value instead of dealing with an array of tuples. Fix an important bug in _compute_delta: old and is_odd were transposed so we weren't actually subtracting the old deltas! Surprisingly, when I accidentally fixed this bug in the vectorized version, the video encoding was much worse! This turned out to be because the edit distance metric allowed reducing diffs by turning on pixels, which meant it would tend to do this when "minimizing error" in a way that was visually unappealing. To remedy this, introduce a separate notion of substitution cost for errors, and weight pixel colour changes more highly to discourage them unless absolutely necessary. This gives very good quality results! Also vectorize the selection of page offsets and priorities having a negative error delta, instead of heapifying the entire page. Also it turns out to be a bit faster to compute (and memoize) the delta between a proposed content byte and the entire target screen at once, since we'll end up recomputing the same content diffs multiple times. (Committing messy version in case I want to revisit some of those interim versions)
2019-03-14 22:08:50 +00:00
from typing import List, Iterator, Tuple
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
Introduction more general notion of update priority used to increase weight of diffs that persist across multiple frames. For each frame, zero out update priority of bytes that no longer have a pending diff, and add the edit distance of the remaining diffs. Zero these out as opcodes are retired. Replace hamming distance with Damerau-Levenshtein distance of the encoded pixel colours in the byte, e.g. 0x2A --> GGG0 (taking into account the half-pixel) This has a couple of benefits over hamming distance of the bit patterns: - transposed pixels are weighted less (edit distance 1, not 2+ for Hamming) - coloured pixels are weighted equally as white pixels (not half as much) - weighting changes in palette bit that flip multiple pixel colours While I'm here, the RLE opcode should emit run_length - 1 so that we can encode runs of 256 bytes.
2019-03-04 23:09:00 +00:00
import numpy as np
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
import skvideo.io
Fix imports and point to player debug symbols
2019-03-21 16:24:40 +00:00
from PIL import Image
Introduction more general notion of update priority used to increase weight of diffs that persist across multiple frames. For each frame, zero out update priority of bytes that no longer have a pending diff, and add the edit distance of the remaining diffs. Zero these out as opcodes are retired. Replace hamming distance with Damerau-Levenshtein distance of the encoded pixel colours in the byte, e.g. 0x2A --> GGG0 (taking into account the half-pixel) This has a couple of benefits over hamming distance of the bit patterns: - transposed pixels are weighted less (edit distance 1, not 2+ for Hamming) - coloured pixels are weighted equally as white pixels (not half as much) - weighting changes in palette bit that flip multiple pixel colours While I'm here, the RLE opcode should emit run_length - 1 so that we can encode runs of 256 bytes.
2019-03-04 23:09:00 +00:00
Fix imports and point to player debug symbols
2019-03-21 16:24:40 +00:00
import opcodes
import screen
Refactor the world
2019-01-05 23:31:56 +00:00
Add a video Mode enum.
2019-04-25 15:28:25 +00:00
class Mode(enum.Enum):
HGR = 0
DHGR = 1
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
class FrameSequencer:
def __init__(self, mode: Mode):
self.video_mode = mode
self.input_frame_rate = 30
Refactor the world
2019-01-05 23:31:56 +00:00
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
def frames(self) -> Iterator[screen.MemoryMap]:
raise NotImplementedError
Video() is now aware of target frame rate, and will continue to emit opcodes until the cycle budget for the frame is exhausted. Output stream is also now aware of TCP framing, and schedules an ACK opcode every 2048 output bytes to instruct the client to perform TCP ACK and buffer management. Fixes several serious bugs in RLE encoding, including: - we were emitting the RLE opcode with the next content byte after the run completed! - we were looking at the wrong field for the start offset! - handle the case where the entire page is a single run - stop trying to allow accumulating error when RLE -- this does not respect the Apple II colour encoding, i.e. may introduce colour fringing. - also because of this we're unlikely to actually be able to find many runs because odd and even columns are encoded differently. In a followup we should start encoding odd and even columns separately Optimize after profiling -- encoder is now about 2x faster Add tests.
2019-02-23 23:52:25 +00:00
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
class FileFrameSequencer(FrameSequencer):
def __init__(self, filename, mode: Mode):
super(FileFrameSequencer, self).__init__(mode)
self.filename = filename # type: str
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
self._reader = skvideo.io.FFmpegReader(filename)
# Compute frame rate from input video
Style cleanups
2019-03-14 23:05:15 +00:00
# TODO: possible to compute time offset for each frame instead?
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
data = skvideo.io.ffprobe(self.filename)['video']
rate_data = data['@r_frame_rate'].split("/") # e.g. 12000/1001
Print input frame rate, and default to every_n_video_frames=2 which gives better quality without compromising frame rate too much.
2019-03-21 23:25:51 +00:00
self.input_frame_rate = float(
Style cleanups
2019-03-14 23:05:15 +00:00
rate_data[0]) / float(rate_data[1]) # type: float
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
- Don't hardcode clock speed - Commented-out experimentation with using hitherdither library to use yliluoma dithering (see https://bisqwit.iki.fi/story/howto/dither/jy/) instead of the (error-diffusion) based BMP2DHR which introduces a lot of noise between frames since it is easily perturbed. Unfortunately apart from being extremely slow, it also doesn't give good results, even for (simulated) DHGR palette. There's a lot of banding and for HGR the available colours are just too far apart in colour space. This is even without (somehow) applying the HGR colour constraints. - Also return the priority from _compute_error as preparation for reinserting the offset back into the priority heap, in case we can do a better job later. In order to do this properly we need to compute both the error edit distance and the "true" edit distance and only insert the priority of the latter.
2019-03-21 15:57:09 +00:00
def _frame_grabber(self) -> Iterator[Image.Image]:
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
for frame_array in self._reader.nextFrame():
yield Image.fromarray(frame_array)
Refactor the world
2019-01-05 23:31:56 +00:00
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
def frames(self) -> Iterator[screen.MemoryMap]:
"""Encode frame to HGR using bmp2dhr.
Refactor the world
2019-01-05 23:31:56 +00:00
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
We do the encoding in a background thread to parallelize.
Refactor the world
2019-01-05 23:31:56 +00:00
"""
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
frame_dir = ".".join(self.filename.split(".")[:-1])
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
try:
os.mkdir(frame_dir)
except FileExistsError:
pass
q = queue.Queue(maxsize=10)
Add --video_mode HGR|DHGR option to support both video modes uniformly.
2019-04-25 16:38:04 +00:00
def _hgr_decode(_idx, _frame):
outfile = "%s/%08dC.BIN" % (frame_dir, _idx)
bmpfile = "%s/%08d.bmp" % (frame_dir, _idx)
try:
os.stat(outfile)
except FileNotFoundError:
_frame = _frame.resize((280, 192), resample=Image.LANCZOS)
_frame.save(bmpfile)
# TODO: parametrize palette
subprocess.call([
"/usr/local/bin/bmp2dhr", bmpfile, "hgr",
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
"P5",
# "P0", # Kegs32 RGB Color palette(for //gs playback)
Add --video_mode HGR|DHGR option to support both video modes uniformly.
2019-04-25 16:38:04 +00:00
"D9" # Buckels dither
])
os.remove(bmpfile)
_main = np.fromfile(outfile, dtype=np.uint8)
Proof of concept DHGR encoding/playback - Every time we process an ACK opcode, toggle page 1/page 2 soft switches to steer subsequent writes between MAIN and AUX memory - while I'm here, squeeze out some unnecessary operations from the buffer management On the player side, this is implemented by maintaining two screen memory maps, and alternating between opcode streams for each of them. This is using entirely the wrong colour model for errors, but surprisingly it already works pretty well in practise (and the frame rate is acceptable on test videos) DHGR/HGR could be made runtime selectable by adding a header byte that determines whether to set the DHGR soft switches before initiating the decode loop. While I'm in here, fix op_terminate to clear keyboard strobe before waiting.
2019-03-27 21:37:06 +00:00
Add --video_mode HGR|DHGR option to support both video modes uniformly.
2019-04-25 16:38:04 +00:00
return _main, None
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
Add --video_mode HGR|DHGR option to support both video modes uniformly.
2019-04-25 16:38:04 +00:00
def _dhgr_decode(_idx, _frame):
mainfile = "%s/%08d.BIN" % (frame_dir, _idx)
auxfile = "%s/%08d.AUX" % (frame_dir, _idx)
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
Add --video_mode HGR|DHGR option to support both video modes uniformly.
2019-04-25 16:38:04 +00:00
bmpfile = "%s/%08d.bmp" % (frame_dir, _idx)
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
Add --video_mode HGR|DHGR option to support both video modes uniformly.
2019-04-25 16:38:04 +00:00
try:
os.stat(mainfile)
os.stat(auxfile)
except FileNotFoundError:
_frame = _frame.resize((280, 192), resample=Image.LANCZOS)
_frame.save(bmpfile)
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
Add --video_mode HGR|DHGR option to support both video modes uniformly.
2019-04-25 16:38:04 +00:00
# TODO: parametrize palette
subprocess.call([
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
"/usr/local/bin/bmp2dhr", bmpfile, "dhgr", # "v",
"P5", # "P0", # Kegs32 RGB Color palette (for //gs
# playback)
Add --video_mode HGR|DHGR option to support both video modes uniformly.
2019-04-25 16:38:04 +00:00
"A", # Output separate .BIN and .AUX files
"D9" # Buckels dither
])
os.remove(bmpfile)
_main = np.fromfile(mainfile, dtype=np.uint8)
_aux = np.fromfile(auxfile, dtype=np.uint8)
return _main, _aux
def worker():
"""Invoke bmp2dhr to encode input image frames and push to queue."""
for _idx, _frame in enumerate(self._frame_grabber()):
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
if self.video_mode == Mode.DHGR:
Add --video_mode HGR|DHGR option to support both video modes uniformly.
2019-04-25 16:38:04 +00:00
res = _dhgr_decode(_idx, _frame)
else:
res = _hgr_decode(_idx, _frame)
q.put(res)
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
Proof of concept DHGR encoding/playback - Every time we process an ACK opcode, toggle page 1/page 2 soft switches to steer subsequent writes between MAIN and AUX memory - while I'm here, squeeze out some unnecessary operations from the buffer management On the player side, this is implemented by maintaining two screen memory maps, and alternating between opcode streams for each of them. This is using entirely the wrong colour model for errors, but surprisingly it already works pretty well in practise (and the frame rate is acceptable on test videos) DHGR/HGR could be made runtime selectable by adding a header byte that determines whether to set the DHGR soft switches before initiating the decode loop. While I'm in here, fix op_terminate to clear keyboard strobe before waiting.
2019-03-27 21:37:06 +00:00
q.put((None, None))
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
Style cleanups
2019-03-14 23:05:15 +00:00
t = threading.Thread(target=worker, daemon=True)
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
t.start()
while True:
Proof of concept DHGR encoding/playback - Every time we process an ACK opcode, toggle page 1/page 2 soft switches to steer subsequent writes between MAIN and AUX memory - while I'm here, squeeze out some unnecessary operations from the buffer management On the player side, this is implemented by maintaining two screen memory maps, and alternating between opcode streams for each of them. This is using entirely the wrong colour model for errors, but surprisingly it already works pretty well in practise (and the frame rate is acceptable on test videos) DHGR/HGR could be made runtime selectable by adding a header byte that determines whether to set the DHGR soft switches before initiating the decode loop. While I'm in here, fix op_terminate to clear keyboard strobe before waiting.
2019-03-27 21:37:06 +00:00
main, aux = q.get()
if main is None:
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
break
Add --video_mode HGR|DHGR option to support both video modes uniformly.
2019-04-25 16:38:04 +00:00
main_map = screen.FlatMemoryMap(
screen_page=1, data=main).to_memory_map()
if aux is None:
aux_map = None
else:
aux_map = screen.FlatMemoryMap(
screen_page=1, data=aux).to_memory_map()
yield (main_map, aux_map)
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
q.task_done()
t.join()
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
class Video:
"""Apple II screen memory map encoding a bitmapped frame."""
CLOCK_SPEED = 1024 * 1024 # type: int
def __init__(
self,
frame_sequencer: FrameSequencer,
mode: Mode = Mode.HGR
):
self.mode = mode # type: Mode
self.frame_sequencer = frame_sequencer # type: FrameSequencer
self.cycles_per_frame = (
self.CLOCK_SPEED / frame_sequencer.input_frame_rate
) # type: float
self.frame_number = 0 # type: int
# Initialize empty screen
self.memory_map = screen.MemoryMap(
screen_page=1) # type: screen.MemoryMap
if self.mode == mode.DHGR:
self.aux_memory_map = screen.MemoryMap(
screen_page=1) # type: screen.MemoryMap
self.pixelmap = screen.DHGRBitmap(
main_memory=self.memory_map,
aux_memory=self.aux_memory_map
)
# Accumulates pending edit weights across frames
self.update_priority = np.zeros((32, 256), dtype=np.int)
if self.mode == mode.DHGR:
self.aux_update_priority = np.zeros((32, 256), dtype=np.int)
def tick(self, cycles: int) -> bool:
if cycles > (self.cycles_per_frame * self.frame_number):
self.frame_number += 1
return True
return False
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
def encode_frame(
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
self,
target: screen.MemoryMap,
is_aux: bool,
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
) -> Iterator[opcodes.Opcode]:
"""Update to match content of frame within provided budget."""
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
if is_aux:
memory_map = self.aux_memory_map
update_priority = self.aux_update_priority
else:
memory_map = self.memory_map
update_priority = self.update_priority
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
Proof of concept DHGR encoding/playback - Every time we process an ACK opcode, toggle page 1/page 2 soft switches to steer subsequent writes between MAIN and AUX memory - while I'm here, squeeze out some unnecessary operations from the buffer management On the player side, this is implemented by maintaining two screen memory maps, and alternating between opcode streams for each of them. This is using entirely the wrong colour model for errors, but surprisingly it already works pretty well in practise (and the frame rate is acceptable on test videos) DHGR/HGR could be made runtime selectable by adding a header byte that determines whether to set the DHGR soft switches before initiating the decode loop. While I'm in here, fix op_terminate to clear keyboard strobe before waiting.
2019-03-27 21:37:06 +00:00
print("Similarity %f" % (update_priority.mean()))
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
yield from self._index_changes(
memory_map, target, update_priority, is_aux)
Video() is now aware of target frame rate, and will continue to emit opcodes until the cycle budget for the frame is exhausted. Output stream is also now aware of TCP framing, and schedules an ACK opcode every 2048 output bytes to instruct the client to perform TCP ACK and buffer management. Fixes several serious bugs in RLE encoding, including: - we were emitting the RLE opcode with the next content byte after the run completed! - we were looking at the wrong field for the start offset! - handle the case where the entire page is a single run - stop trying to allow accumulating error when RLE -- this does not respect the Apple II colour encoding, i.e. may introduce colour fringing. - also because of this we're unlikely to actually be able to find many runs because odd and even columns are encoded differently. In a followup we should start encoding odd and even columns separately Optimize after profiling -- encoder is now about 2x faster Add tests.
2019-02-23 23:52:25 +00:00
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
def _index_changes(
self,
source: screen.MemoryMap,
Proof of concept DHGR encoding/playback - Every time we process an ACK opcode, toggle page 1/page 2 soft switches to steer subsequent writes between MAIN and AUX memory - while I'm here, squeeze out some unnecessary operations from the buffer management On the player side, this is implemented by maintaining two screen memory maps, and alternating between opcode streams for each of them. This is using entirely the wrong colour model for errors, but surprisingly it already works pretty well in practise (and the frame rate is acceptable on test videos) DHGR/HGR could be made runtime selectable by adding a header byte that determines whether to set the DHGR soft switches before initiating the decode loop. While I'm in here, fix op_terminate to clear keyboard strobe before waiting.
2019-03-27 21:37:06 +00:00
target: screen.MemoryMap,
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
update_priority: np.array,
is_aux: True
Move edit distance functions into separate module and clean up partially. Slight optimization to not heapppush() many times, instead build a regular list and then heapify.
2019-03-14 22:32:52 +00:00
) -> Iterator[Tuple[int, int, List[int]]]:
"""Transform encoded screen to sequence of change tuples."""
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
if is_aux:
target_pixelmap = screen.DHGRBitmap(
main_memory=self.memory_map,
aux_memory=target
)
else:
target_pixelmap = screen.DHGRBitmap(
main_memory=target,
aux_memory=self.aux_memory_map
)
diff_weights = self._diff_weights(
self.pixelmap, target_pixelmap, is_aux
)
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
Move edit distance functions into separate module and clean up partially. Slight optimization to not heapppush() many times, instead build a regular list and then heapify.
2019-03-14 22:32:52 +00:00
# Clear any update priority entries that have resolved themselves
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
# with new frame
Proof of concept DHGR encoding/playback - Every time we process an ACK opcode, toggle page 1/page 2 soft switches to steer subsequent writes between MAIN and AUX memory - while I'm here, squeeze out some unnecessary operations from the buffer management On the player side, this is implemented by maintaining two screen memory maps, and alternating between opcode streams for each of them. This is using entirely the wrong colour model for errors, but surprisingly it already works pretty well in practise (and the frame rate is acceptable on test videos) DHGR/HGR could be made runtime selectable by adding a header byte that determines whether to set the DHGR soft switches before initiating the decode loop. While I'm in here, fix op_terminate to clear keyboard strobe before waiting.
2019-03-27 21:37:06 +00:00
update_priority[diff_weights == 0] = 0
update_priority += diff_weights
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
Proof of concept DHGR encoding/playback - Every time we process an ACK opcode, toggle page 1/page 2 soft switches to steer subsequent writes between MAIN and AUX memory - while I'm here, squeeze out some unnecessary operations from the buffer management On the player side, this is implemented by maintaining two screen memory maps, and alternating between opcode streams for each of them. This is using entirely the wrong colour model for errors, but surprisingly it already works pretty well in practise (and the frame rate is acceptable on test videos) DHGR/HGR could be made runtime selectable by adding a header byte that determines whether to set the DHGR soft switches before initiating the decode loop. While I'm in here, fix op_terminate to clear keyboard strobe before waiting.
2019-03-27 21:37:06 +00:00
priorities = self._heapify_priorities(update_priority)
Vectorize the computation of diff weights, by precomputing a map of all possible weights. We encode the two 8-bit inputs into a single 16 bit value instead of dealing with an array of tuples. Fix an important bug in _compute_delta: old and is_odd were transposed so we weren't actually subtracting the old deltas! Surprisingly, when I accidentally fixed this bug in the vectorized version, the video encoding was much worse! This turned out to be because the edit distance metric allowed reducing diffs by turning on pixels, which meant it would tend to do this when "minimizing error" in a way that was visually unappealing. To remedy this, introduce a separate notion of substitution cost for errors, and weight pixel colour changes more highly to discourage them unless absolutely necessary. This gives very good quality results! Also vectorize the selection of page offsets and priorities having a negative error delta, instead of heapifying the entire page. Also it turns out to be a bit faster to compute (and memoize) the delta between a proposed content byte and the entire target screen at once, since we'll end up recomputing the same content diffs multiple times. (Committing messy version in case I want to revisit some of those interim versions)
2019-03-14 22:08:50 +00:00
content_deltas = {}
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
while priorities:
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
pri, _, page, offset = heapq.heappop(priorities)
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
# Check whether we've already cleared this diff while processing
# an earlier opcode
Proof of concept DHGR encoding/playback - Every time we process an ACK opcode, toggle page 1/page 2 soft switches to steer subsequent writes between MAIN and AUX memory - while I'm here, squeeze out some unnecessary operations from the buffer management On the player side, this is implemented by maintaining two screen memory maps, and alternating between opcode streams for each of them. This is using entirely the wrong colour model for errors, but surprisingly it already works pretty well in practise (and the frame rate is acceptable on test videos) DHGR/HGR could be made runtime selectable by adding a header byte that determines whether to set the DHGR soft switches before initiating the decode loop. While I'm in here, fix op_terminate to clear keyboard strobe before waiting.
2019-03-27 21:37:06 +00:00
if update_priority[page, offset] == 0:
Integrated audio + video player! - Introduce a new Movie() class that multiplexes audio and video. - Every N audio frames we grab a new video frame and begin pulling opcodes from the audio and video streams - Grab frames from the input video using bmp2dhr if the .BIN file does not already exist. Run bmp2dhr in a background thread to not block encoding - move the output byte streaming from Video to Movie - For now, manually clip updates to pages > 56 since the client doesn't support them yet The way we encode video is now: - iterate in descending order over update_priority - begin a new (page, content) opcode - for all of the other offset bytes in that page, compute the error between the candidate content byte and the target content byte - iterate over offsets in order of increasing error and decreasing update_priority to fill out the remaining opcode
2019-03-07 23:07:24 +00:00
continue
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
Integrated audio + video player! - Introduce a new Movie() class that multiplexes audio and video. - Every N audio frames we grab a new video frame and begin pulling opcodes from the audio and video streams - Grab frames from the input video using bmp2dhr if the .BIN file does not already exist. Run bmp2dhr in a background thread to not block encoding - move the output byte streaming from Video to Movie - For now, manually clip updates to pages > 56 since the client doesn't support them yet The way we encode video is now: - iterate in descending order over update_priority - begin a new (page, content) opcode - for all of the other offset bytes in that page, compute the error between the candidate content byte and the target content byte - iterate over offsets in order of increasing error and decreasing update_priority to fill out the remaining opcode
2019-03-07 23:07:24 +00:00
offsets = [offset]
content = target.page_offset[page, offset]
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
assert content < 0x80 # DHGR palette bit not expected to be set
Integrated audio + video player! - Introduce a new Movie() class that multiplexes audio and video. - Every N audio frames we grab a new video frame and begin pulling opcodes from the audio and video streams - Grab frames from the input video using bmp2dhr if the .BIN file does not already exist. Run bmp2dhr in a background thread to not block encoding - move the output byte streaming from Video to Movie - For now, manually clip updates to pages > 56 since the client doesn't support them yet The way we encode video is now: - iterate in descending order over update_priority - begin a new (page, content) opcode - for all of the other offset bytes in that page, compute the error between the candidate content byte and the target content byte - iterate over offsets in order of increasing error and decreasing update_priority to fill out the remaining opcode
2019-03-07 23:07:24 +00:00
# Clear priority for the offset we're emitting
Proof of concept DHGR encoding/playback - Every time we process an ACK opcode, toggle page 1/page 2 soft switches to steer subsequent writes between MAIN and AUX memory - while I'm here, squeeze out some unnecessary operations from the buffer management On the player side, this is implemented by maintaining two screen memory maps, and alternating between opcode streams for each of them. This is using entirely the wrong colour model for errors, but surprisingly it already works pretty well in practise (and the frame rate is acceptable on test videos) DHGR/HGR could be made runtime selectable by adding a header byte that determines whether to set the DHGR soft switches before initiating the decode loop. While I'm in here, fix op_terminate to clear keyboard strobe before waiting.
2019-03-27 21:37:06 +00:00
update_priority[page, offset] = 0
Add some docstrings Clean up naming in edit_distance In video encoder, when we emit additional offsets as part of an opcode, reinsert back into the priority heapq if the new edit distance is nonzero, in case we get the chance to fix it up later in the frame. Also make sure to zero out the diff_weights and content_deltas so we don't consider the offset again as a side-effect of some other opcode. Instead of prioritizing side-effect offsets by their previous update priority, prioritize by those with the lowest (error - edit) delta i.e. not introducing too much error relative to their edit distance.
2019-03-21 22:56:45 +00:00
diff_weights[page, offset] = 0
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
# Update memory maps
source.page_offset[page, offset] = content
self.pixelmap.apply(page, offset, is_aux, content)
Add some docstrings Clean up naming in edit_distance In video encoder, when we emit additional offsets as part of an opcode, reinsert back into the priority heapq if the new edit distance is nonzero, in case we get the chance to fix it up later in the frame. Also make sure to zero out the diff_weights and content_deltas so we don't consider the offset again as a side-effect of some other opcode. Instead of prioritizing side-effect offsets by their previous update priority, prioritize by those with the lowest (error - edit) delta i.e. not introducing too much error relative to their edit distance.
2019-03-21 22:56:45 +00:00
# Make sure we don't emit this offset as a side-effect of some
# other offset later.
for cd in content_deltas.values():
cd[page, offset] = 0
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
# TODO: what if we add another content_deltas entry later?
# We might clobber it again
Integrated audio + video player! - Introduce a new Movie() class that multiplexes audio and video. - Every N audio frames we grab a new video frame and begin pulling opcodes from the audio and video streams - Grab frames from the input video using bmp2dhr if the .BIN file does not already exist. Run bmp2dhr in a background thread to not block encoding - move the output byte streaming from Video to Movie - For now, manually clip updates to pages > 56 since the client doesn't support them yet The way we encode video is now: - iterate in descending order over update_priority - begin a new (page, content) opcode - for all of the other offset bytes in that page, compute the error between the candidate content byte and the target content byte - iterate over offsets in order of increasing error and decreasing update_priority to fill out the remaining opcode
2019-03-07 23:07:24 +00:00
# Need to find 3 more offsets to fill this opcode
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
for err, o in self._compute_error(
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
page,
Integrated audio + video player! - Introduce a new Movie() class that multiplexes audio and video. - Every N audio frames we grab a new video frame and begin pulling opcodes from the audio and video streams - Grab frames from the input video using bmp2dhr if the .BIN file does not already exist. Run bmp2dhr in a background thread to not block encoding - move the output byte streaming from Video to Movie - For now, manually clip updates to pages > 56 since the client doesn't support them yet The way we encode video is now: - iterate in descending order over update_priority - begin a new (page, content) opcode - for all of the other offset bytes in that page, compute the error between the candidate content byte and the target content byte - iterate over offsets in order of increasing error and decreasing update_priority to fill out the remaining opcode
2019-03-07 23:07:24 +00:00
content,
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
target_pixelmap,
Vectorize the computation of diff weights, by precomputing a map of all possible weights. We encode the two 8-bit inputs into a single 16 bit value instead of dealing with an array of tuples. Fix an important bug in _compute_delta: old and is_odd were transposed so we weren't actually subtracting the old deltas! Surprisingly, when I accidentally fixed this bug in the vectorized version, the video encoding was much worse! This turned out to be because the edit distance metric allowed reducing diffs by turning on pixels, which meant it would tend to do this when "minimizing error" in a way that was visually unappealing. To remedy this, introduce a separate notion of substitution cost for errors, and weight pixel colour changes more highly to discourage them unless absolutely necessary. This gives very good quality results! Also vectorize the selection of page offsets and priorities having a negative error delta, instead of heapifying the entire page. Also it turns out to be a bit faster to compute (and memoize) the delta between a proposed content byte and the entire target screen at once, since we'll end up recomputing the same content diffs multiple times. (Committing messy version in case I want to revisit some of those interim versions)
2019-03-14 22:08:50 +00:00
diff_weights,
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
content_deltas,
is_aux
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
):
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
assert o != offset
if update_priority[page, o] == 0:
# print("Skipping page=%d, offset=%d" % (page, o))
continue
Add some docstrings Clean up naming in edit_distance In video encoder, when we emit additional offsets as part of an opcode, reinsert back into the priority heapq if the new edit distance is nonzero, in case we get the chance to fix it up later in the frame. Also make sure to zero out the diff_weights and content_deltas so we don't consider the offset again as a side-effect of some other opcode. Instead of prioritizing side-effect offsets by their previous update priority, prioritize by those with the lowest (error - edit) delta i.e. not introducing too much error relative to their edit distance.
2019-03-21 22:56:45 +00:00
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
# Make sure we don't end up considering this (page, offset)
# again until the next image frame. Even if a better match
# comes along, it's probably better to fix up some other byte.
# TODO: or should we recompute it with new error?
for cd in content_deltas.values():
cd[page, o] = 0
byte_offset = target_pixelmap.interleaved_byte_offset(o, is_aux)
old_packed = target_pixelmap.packed[page, o // 2]
p = self._byte_pair_difference(
target_pixelmap, byte_offset, old_packed, content)
Add some docstrings Clean up naming in edit_distance In video encoder, when we emit additional offsets as part of an opcode, reinsert back into the priority heapq if the new edit distance is nonzero, in case we get the chance to fix it up later in the frame. Also make sure to zero out the diff_weights and content_deltas so we don't consider the offset again as a side-effect of some other opcode. Instead of prioritizing side-effect offsets by their previous update priority, prioritize by those with the lowest (error - edit) delta i.e. not introducing too much error relative to their edit distance.
2019-03-21 22:56:45 +00:00
- Don't hardcode clock speed - Commented-out experimentation with using hitherdither library to use yliluoma dithering (see https://bisqwit.iki.fi/story/howto/dither/jy/) instead of the (error-diffusion) based BMP2DHR which introduces a lot of noise between frames since it is easily perturbed. Unfortunately apart from being extremely slow, it also doesn't give good results, even for (simulated) DHGR palette. There's a lot of banding and for HGR the available colours are just too far apart in colour space. This is even without (somehow) applying the HGR colour constraints. - Also return the priority from _compute_error as preparation for reinserting the offset back into the priority heap, in case we can do a better job later. In order to do this properly we need to compute both the error edit distance and the "true" edit distance and only insert the priority of the latter.
2019-03-21 15:57:09 +00:00
# Update priority for the offset we're emitting
Proof of concept DHGR encoding/playback - Every time we process an ACK opcode, toggle page 1/page 2 soft switches to steer subsequent writes between MAIN and AUX memory - while I'm here, squeeze out some unnecessary operations from the buffer management On the player side, this is implemented by maintaining two screen memory maps, and alternating between opcode streams for each of them. This is using entirely the wrong colour model for errors, but surprisingly it already works pretty well in practise (and the frame rate is acceptable on test videos) DHGR/HGR could be made runtime selectable by adding a header byte that determines whether to set the DHGR soft switches before initiating the decode loop. While I'm in here, fix op_terminate to clear keyboard strobe before waiting.
2019-03-27 21:37:06 +00:00
update_priority[page, o] = p # 0
Add some docstrings Clean up naming in edit_distance In video encoder, when we emit additional offsets as part of an opcode, reinsert back into the priority heapq if the new edit distance is nonzero, in case we get the chance to fix it up later in the frame. Also make sure to zero out the diff_weights and content_deltas so we don't consider the offset again as a side-effect of some other opcode. Instead of prioritizing side-effect offsets by their previous update priority, prioritize by those with the lowest (error - edit) delta i.e. not introducing too much error relative to their edit distance.
2019-03-21 22:56:45 +00:00
Proof of concept DHGR encoding/playback - Every time we process an ACK opcode, toggle page 1/page 2 soft switches to steer subsequent writes between MAIN and AUX memory - while I'm here, squeeze out some unnecessary operations from the buffer management On the player side, this is implemented by maintaining two screen memory maps, and alternating between opcode streams for each of them. This is using entirely the wrong colour model for errors, but surprisingly it already works pretty well in practise (and the frame rate is acceptable on test videos) DHGR/HGR could be made runtime selectable by adding a header byte that determines whether to set the DHGR soft switches before initiating the decode loop. While I'm in here, fix op_terminate to clear keyboard strobe before waiting.
2019-03-27 21:37:06 +00:00
source.page_offset[page, o] = content
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
self.pixelmap.apply(page, o, is_aux, content)
Add some docstrings Clean up naming in edit_distance In video encoder, when we emit additional offsets as part of an opcode, reinsert back into the priority heapq if the new edit distance is nonzero, in case we get the chance to fix it up later in the frame. Also make sure to zero out the diff_weights and content_deltas so we don't consider the offset again as a side-effect of some other opcode. Instead of prioritizing side-effect offsets by their previous update priority, prioritize by those with the lowest (error - edit) delta i.e. not introducing too much error relative to their edit distance.
2019-03-21 22:56:45 +00:00
if p:
# This content byte introduced an error, so put back on the
# heap in case we can get back to fixing it exactly
# during this frame. Otherwise we'll get to it later.
heapq.heappush(
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
priorities, (-p, random.randint(0, 10000), page, o))
offsets.append(o)
if len(offsets) == 3:
break
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
# Pad to 4 if we didn't find enough
Integrated audio + video player! - Introduce a new Movie() class that multiplexes audio and video. - Every N audio frames we grab a new video frame and begin pulling opcodes from the audio and video streams - Grab frames from the input video using bmp2dhr if the .BIN file does not already exist. Run bmp2dhr in a background thread to not block encoding - move the output byte streaming from Video to Movie - For now, manually clip updates to pages > 56 since the client doesn't support them yet The way we encode video is now: - iterate in descending order over update_priority - begin a new (page, content) opcode - for all of the other offset bytes in that page, compute the error between the candidate content byte and the target content byte - iterate over offsets in order of increasing error and decreasing update_priority to fill out the remaining opcode
2019-03-07 23:07:24 +00:00
for _ in range(len(offsets), 4):
offsets.append(offsets[0])
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
yield (page + 32, content, offsets)
Video() is now aware of target frame rate, and will continue to emit opcodes until the cycle budget for the frame is exhausted. Output stream is also now aware of TCP framing, and schedules an ACK opcode every 2048 output bytes to instruct the client to perform TCP ACK and buffer management. Fixes several serious bugs in RLE encoding, including: - we were emitting the RLE opcode with the next content byte after the run completed! - we were looking at the wrong field for the start offset! - handle the case where the entire page is a single run - stop trying to allow accumulating error when RLE -- this does not respect the Apple II colour encoding, i.e. may introduce colour fringing. - also because of this we're unlikely to actually be able to find many runs because odd and even columns are encoded differently. In a followup we should start encoding odd and even columns separately Optimize after profiling -- encoder is now about 2x faster Add tests.
2019-02-23 23:52:25 +00:00
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
# TODO: there is still a bug causing residual diffs when we have
# apparently run out of work to do
if not np.array_equal(source.page_offset, target.page_offset):
diffs = np.nonzero(source.page_offset != target.page_offset)
for i in range(len(diffs[0])):
diff_p = diffs[0][i]
diff_o = diffs[1][i]
print("Diff at (%d, %d): %d != %d" % (
diff_p, diff_o, source.page_offset[diff_p, diff_o],
target.page_offset[diff_p, diff_o]
))
# assert False
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
# If we run out of things to do, pad forever
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
content = target.page_offset[0, 0]
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
while True:
yield (32, content, [0, 0, 0, 0])
Move edit distance functions into separate module and clean up partially. Slight optimization to not heapppush() many times, instead build a regular list and then heapify.
2019-03-14 22:32:52 +00:00
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
@staticmethod
def _heapify_priorities(update_priority: np.array) -> List:
pages, offsets = update_priority.nonzero()
priorities = [tuple(data) for data in np.stack((
-update_priority[pages, offsets],
# Don't use deterministic order for page, offset
np.random.randint(0, 10000, size=pages.shape[0]),
pages,
offsets)
).T.tolist()]
heapq.heapify(priorities)
return priorities
Move edit distance functions into separate module and clean up partially. Slight optimization to not heapppush() many times, instead build a regular list and then heapify.
2019-03-14 22:32:52 +00:00
@staticmethod
def _diff_weights(
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
source: screen.DHGRBitmap,
target: screen.DHGRBitmap,
is_aux: bool
):
diff = np.ndarray((32, 256), dtype=np.int)
if is_aux:
# Pixels influenced by byte offset 0
source_pixels0 = source.mask_and_shift_data(source.packed, 0)
target_pixels0 = target.mask_and_shift_data(target.packed, 0)
# Concatenate 8-bit source and target into 16-bit values
pair0 = (source_pixels0 << 8) + target_pixels0
dist0 = source.edit_distances[0][pair0].reshape(pair0.shape)
# Pixels influenced by byte offset 2
source_pixels2 = source.mask_and_shift_data(source.packed, 2)
target_pixels2 = target.mask_and_shift_data(target.packed, 2)
# Concatenate 12-bit source and target into 24-bit values
pair2 = (source_pixels2 << 12) + target_pixels2
dist2 = source.edit_distances[2][pair2].reshape(pair2.shape)
diff[:, 0::2] = dist0
diff[:, 1::2] = dist2
else:
# Pixels influenced by byte offset 1
source_pixels1 = source.mask_and_shift_data(source.packed, 1)
target_pixels1 = target.mask_and_shift_data(target.packed, 1)
pair1 = (source_pixels1 << 12) + target_pixels1
dist1 = source.edit_distances[1][pair1].reshape(pair1.shape)
# Pixels influenced by byte offset 3
source_pixels3 = source.mask_and_shift_data(source.packed, 3)
target_pixels3 = target.mask_and_shift_data(target.packed, 3)
pair3 = (source_pixels3 << 8) + target_pixels3
dist3 = source.edit_distances[3][pair3].reshape(pair3.shape)
diff[:, 0::2] = dist1
diff[:, 1::2] = dist3
return diff
@functools.lru_cache(None)
def _byte_pair_difference(
self,
target_pixelmap,
byte_offset,
old_packed,
content
Move edit distance functions into separate module and clean up partially. Slight optimization to not heapppush() many times, instead build a regular list and then heapify.
2019-03-14 22:32:52 +00:00
):
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
old_pixels = target_pixelmap.mask_and_shift_data(
old_packed, byte_offset)
new_pixels = target_pixelmap.mask_and_shift_data(
target_pixelmap.masked_update(
byte_offset, old_packed, content), byte_offset)
Move edit distance functions into separate module and clean up partially. Slight optimization to not heapppush() many times, instead build a regular list and then heapify.
2019-03-14 22:32:52 +00:00
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
if byte_offset == 0 or byte_offset == 3:
pair = (old_pixels << 8) + new_pixels
else:
pair = (old_pixels << 12) + new_pixels
Move edit distance functions into separate module and clean up partially. Slight optimization to not heapppush() many times, instead build a regular list and then heapify.
2019-03-14 22:32:52 +00:00
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
p = target_pixelmap.edit_distances[byte_offset][pair]
return p
Move edit distance functions into separate module and clean up partially. Slight optimization to not heapppush() many times, instead build a regular list and then heapify.
2019-03-14 22:32:52 +00:00
@staticmethod
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
def _compute_delta(
content: int,
target: screen.DHGRBitmap,
old,
is_aux: bool
):
diff = np.ndarray((32, 256), dtype=np.int)
# TODO: use error edit distance
if is_aux:
# Pixels influenced by byte offset 0
source_pixels0 = target.mask_and_shift_data(
target.masked_update(0, target.packed, content), 0)
target_pixels0 = target.mask_and_shift_data(target.packed, 0)
# Concatenate 8-bit source and target into 16-bit values
pair0 = (source_pixels0 << 8) + target_pixels0
dist0 = target.edit_distances[0][pair0].reshape(pair0.shape)
# Pixels influenced by byte offset 2
source_pixels2 = target.mask_and_shift_data(
target.masked_update(2, target.packed, content), 2)
target_pixels2 = target.mask_and_shift_data(target.packed, 2)
# Concatenate 12-bit source and target into 24-bit values
pair2 = (source_pixels2 << 12) + target_pixels2
dist2 = target.edit_distances[2][pair2].reshape(pair2.shape)
diff[:, 0::2] = dist0
diff[:, 1::2] = dist2
else:
# Pixels influenced by byte offset 1
source_pixels1 = target.mask_and_shift_data(
target.masked_update(1, target.packed, content), 1)
target_pixels1 = target.mask_and_shift_data(target.packed, 1)
pair1 = (source_pixels1 << 12) + target_pixels1
dist1 = target.edit_distances[1][pair1].reshape(pair1.shape)
# Pixels influenced by byte offset 3
source_pixels3 = target.mask_and_shift_data(
target.masked_update(3, target.packed, content), 3)
target_pixels3 = target.mask_and_shift_data(target.packed, 3)
pair3 = (source_pixels3 << 8) + target_pixels3
dist3 = target.edit_distances[3][pair3].reshape(pair3.shape)
diff[:, 0::2] = dist1
diff[:, 1::2] = dist3
# TODO: try different weightings
return (diff * 5) - old
Move edit distance functions into separate module and clean up partially. Slight optimization to not heapppush() many times, instead build a regular list and then heapify.
2019-03-14 22:32:52 +00:00
_OFFSETS = np.arange(256)
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
def _compute_error(self, page, content, target_pixelmap, old_error,
content_deltas, is_aux):
Add some docstrings Clean up naming in edit_distance In video encoder, when we emit additional offsets as part of an opcode, reinsert back into the priority heapq if the new edit distance is nonzero, in case we get the chance to fix it up later in the frame. Also make sure to zero out the diff_weights and content_deltas so we don't consider the offset again as a side-effect of some other opcode. Instead of prioritizing side-effect offsets by their previous update priority, prioritize by those with the lowest (error - edit) delta i.e. not introducing too much error relative to their edit distance.
2019-03-21 22:56:45 +00:00
# TODO: move this up into parent
Move edit distance functions into separate module and clean up partially. Slight optimization to not heapppush() many times, instead build a regular list and then heapify.
2019-03-14 22:32:52 +00:00
delta_screen = content_deltas.get(content)
if delta_screen is None:
delta_screen = self._compute_delta(
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
content, target_pixelmap, old_error, is_aux)
Move edit distance functions into separate module and clean up partially. Slight optimization to not heapppush() many times, instead build a regular list and then heapify.
2019-03-14 22:32:52 +00:00
content_deltas[content] = delta_screen
delta_page = delta_screen[page]
cond = delta_page < 0
candidate_offsets = self._OFFSETS[cond]
Add some docstrings Clean up naming in edit_distance In video encoder, when we emit additional offsets as part of an opcode, reinsert back into the priority heapq if the new edit distance is nonzero, in case we get the chance to fix it up later in the frame. Also make sure to zero out the diff_weights and content_deltas so we don't consider the offset again as a side-effect of some other opcode. Instead of prioritizing side-effect offsets by their previous update priority, prioritize by those with the lowest (error - edit) delta i.e. not introducing too much error relative to their edit distance.
2019-03-21 22:56:45 +00:00
priorities = delta_page[cond]
Move edit distance functions into separate module and clean up partially. Slight optimization to not heapppush() many times, instead build a regular list and then heapify.
2019-03-14 22:32:52 +00:00
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
# TODO: vectorize this with numpy
deltas = [
(priorities[i], random.randint(0, 10000), candidate_offsets[i])
Move edit distance functions into separate module and clean up partially. Slight optimization to not heapppush() many times, instead build a regular list and then heapify.
2019-03-14 22:32:52 +00:00
for i in range(len(candidate_offsets))
]
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
heapq.heapify(deltas)
Add some docstrings Clean up naming in edit_distance In video encoder, when we emit additional offsets as part of an opcode, reinsert back into the priority heapq if the new edit distance is nonzero, in case we get the chance to fix it up later in the frame. Also make sure to zero out the diff_weights and content_deltas so we don't consider the offset again as a side-effect of some other opcode. Instead of prioritizing side-effect offsets by their previous update priority, prioritize by those with the lowest (error - edit) delta i.e. not introducing too much error relative to their edit distance.
2019-03-21 22:56:45 +00:00
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
while deltas:
pri, _, o = heapq.heappop(deltas)
assert pri < 0
assert o < 255
Move edit distance functions into separate module and clean up partially. Slight optimization to not heapppush() many times, instead build a regular list and then heapify.
2019-03-14 22:32:52 +00:00
FrameSequencer: - Extract out a (File)FrameSequencer class from Video to encapsulate the generation of still frames. This also makes Video easier to test. - Fix FileFrameSequencer.frames() to correctly handle filenames containing '.' - Temporarily switch to the BMP2DHR NTSC palette (#5) for evaluation. Video: - Temporarily hardcode DHGR decoding - Optimize _heapify_priorities() by using numpy to vectorize the construction of the list of tuples. This requires changing the random nonce to an int so the intermediate array has a uniform type. - Use the efficient 28-bit representation of DHGR (aux, main, aux, main) tuples introduced in DHGRBitmap to evaluate diffs - Switch to np.int type for accumulating diffs, and random.randint(0, 10000) instead of float for nonce values. - Fix/improve some of the error evaluation in _index_changes: - skip offsets whose diffs have already been cleared - hoist some stuff out of _compute_error into the parent - Add some validation that when we run out of work to do with a frame, the source and target memory maps should be equal. This isn't happening sometimes, i.e. there is a bug.
2019-06-13 23:12:26 +00:00
yield -pri, o
Reference in New Issue Copy Permalink