Apple-2-SW/ii-vision
1
0
Fork 0
mirror of https://github.com/KrisKennaway/ii-vision.git synced 2024-09-27 04:54:59 +00:00
Code Issues Projects Releases Wiki Activity
ede453d292
ii-vision/video.py

335 lines
9.8 KiB
Python
Raw Normal View History

Refactor the world
2019-01-05 23:31:56 +00:00
import functools
Integrated audio + video player! - Introduce a new Movie() class that multiplexes audio and video. - Every N audio frames we grab a new video frame and begin pulling opcodes from the audio and video streams - Grab frames from the input video using bmp2dhr if the .BIN file does not already exist. Run bmp2dhr in a background thread to not block encoding - move the output byte streaming from Video to Movie - For now, manually clip updates to pages > 56 since the client doesn't support them yet The way we encode video is now: - iterate in descending order over update_priority - begin a new (page, content) opcode - for all of the other offset bytes in that page, compute the error between the candidate content byte and the target content byte - iterate over offsets in order of increasing error and decreasing update_priority to fill out the remaining opcode
2019-03-07 23:07:24 +00:00
import heapq
import random
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
import os
import threading
import queue
import subprocess
Refactor the world
2019-01-05 23:31:56 +00:00
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
from typing import List, Iterator, Tuple, Iterable
from PIL import Image
Introduction more general notion of update priority used to increase weight of diffs that persist across multiple frames. For each frame, zero out update priority of bytes that no longer have a pending diff, and add the edit distance of the remaining diffs. Zero these out as opcodes are retired. Replace hamming distance with Damerau-Levenshtein distance of the encoded pixel colours in the byte, e.g. 0x2A --> GGG0 (taking into account the half-pixel) This has a couple of benefits over hamming distance of the bit patterns: - transposed pixels are weighted less (edit distance 1, not 2+ for Hamming) - coloured pixels are weighted equally as white pixels (not half as much) - weighting changes in palette bit that flip multiple pixel colours While I'm here, the RLE opcode should emit run_length - 1 so that we can encode runs of 256 bytes.
2019-03-04 23:09:00 +00:00
import numpy as np
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
import skvideo.io
import weighted_levenshtein
Introduction more general notion of update priority used to increase weight of diffs that persist across multiple frames. For each frame, zero out update priority of bytes that no longer have a pending diff, and add the edit distance of the remaining diffs. Zero these out as opcodes are retired. Replace hamming distance with Damerau-Levenshtein distance of the encoded pixel colours in the byte, e.g. 0x2A --> GGG0 (taking into account the half-pixel) This has a couple of benefits over hamming distance of the bit patterns: - transposed pixels are weighted less (edit distance 1, not 2+ for Hamming) - coloured pixels are weighted equally as white pixels (not half as much) - weighting changes in palette bit that flip multiple pixel colours While I'm here, the RLE opcode should emit run_length - 1 so that we can encode runs of 256 bytes.
2019-03-04 23:09:00 +00:00
Refactor the world
2019-01-05 23:31:56 +00:00
import opcodes
import screen
Video() is now aware of target frame rate, and will continue to emit opcodes until the cycle budget for the frame is exhausted. Output stream is also now aware of TCP framing, and schedules an ACK opcode every 2048 output bytes to instruct the client to perform TCP ACK and buffer management. Fixes several serious bugs in RLE encoding, including: - we were emitting the RLE opcode with the next content byte after the run completed! - we were looking at the wrong field for the start offset! - handle the case where the entire page is a single run - stop trying to allow accumulating error when RLE -- this does not respect the Apple II colour encoding, i.e. may introduce colour fringing. - also because of this we're unlikely to actually be able to find many runs because odd and even columns are encoded differently. In a followup we should start encoding odd and even columns separately Optimize after profiling -- encoder is now about 2x faster Add tests.
2019-02-23 23:52:25 +00:00
def hamming_weight(n):
Refactor the world
2019-01-05 23:31:56 +00:00
"""Compute hamming weight of 8-bit int"""
n = (n & 0x55) + ((n & 0xAA) >> 1)
n = (n & 0x33) + ((n & 0xCC) >> 2)
n = (n & 0x0F) + ((n & 0xF0) >> 4)
return n
Add some tests for edit_weight and byte_to_colour_string
2019-03-10 23:07:44 +00:00
# TODO: what about increasing transposition cost? Might be better to have
# any pixel at the right place even if the wrong colour?
Refactor the world
2019-01-05 23:31:56 +00:00
Add some tests for edit_weight and byte_to_colour_string
2019-03-10 23:07:44 +00:00
substitute_costs = np.ones((128, 128), dtype=np.float64)
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
# Penalty for turning on/off a black bit
for c in "01GVWOB":
Add some tests for edit_weight and byte_to_colour_string
2019-03-10 23:07:44 +00:00
substitute_costs[(ord('K'), ord(c))] = 5
substitute_costs[(ord(c), ord('K'))] = 5
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
# Penalty for changing colour
for c in "01GVWOB":
for d in "01GVWOB":
Add some tests for edit_weight and byte_to_colour_string
2019-03-10 23:07:44 +00:00
substitute_costs[(ord(c), ord(d))] = 1
substitute_costs[(ord(d), ord(c))] = 1
Introduction more general notion of update priority used to increase weight of diffs that persist across multiple frames. For each frame, zero out update priority of bytes that no longer have a pending diff, and add the edit distance of the remaining diffs. Zero these out as opcodes are retired. Replace hamming distance with Damerau-Levenshtein distance of the encoded pixel colours in the byte, e.g. 0x2A --> GGG0 (taking into account the half-pixel) This has a couple of benefits over hamming distance of the bit patterns: - transposed pixels are weighted less (edit distance 1, not 2+ for Hamming) - coloured pixels are weighted equally as white pixels (not half as much) - weighting changes in palette bit that flip multiple pixel colours While I'm here, the RLE opcode should emit run_length - 1 so that we can encode runs of 256 bytes.
2019-03-04 23:09:00 +00:00
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
insert_costs = np.ones(128, dtype=np.float64) * 1000
delete_costs = np.ones(128, dtype=np.float64) * 1000
@functools.lru_cache(None)
Add some tests for edit_weight and byte_to_colour_string
2019-03-10 23:07:44 +00:00
def edit_weight(a: int, b: int, is_odd_offset: bool):
Introduction more general notion of update priority used to increase weight of diffs that persist across multiple frames. For each frame, zero out update priority of bytes that no longer have a pending diff, and add the edit distance of the remaining diffs. Zero these out as opcodes are retired. Replace hamming distance with Damerau-Levenshtein distance of the encoded pixel colours in the byte, e.g. 0x2A --> GGG0 (taking into account the half-pixel) This has a couple of benefits over hamming distance of the bit patterns: - transposed pixels are weighted less (edit distance 1, not 2+ for Hamming) - coloured pixels are weighted equally as white pixels (not half as much) - weighting changes in palette bit that flip multiple pixel colours While I'm here, the RLE opcode should emit run_length - 1 so that we can encode runs of 256 bytes.
2019-03-04 23:09:00 +00:00
a_pixels = byte_to_colour_string(a, is_odd_offset)
b_pixels = byte_to_colour_string(b, is_odd_offset)
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
dist = weighted_levenshtein.dam_lev(
a_pixels, b_pixels,
insert_costs=insert_costs,
delete_costs=delete_costs,
substitute_costs=substitute_costs,
)
return np.int64(dist)
Introduction more general notion of update priority used to increase weight of diffs that persist across multiple frames. For each frame, zero out update priority of bytes that no longer have a pending diff, and add the edit distance of the remaining diffs. Zero these out as opcodes are retired. Replace hamming distance with Damerau-Levenshtein distance of the encoded pixel colours in the byte, e.g. 0x2A --> GGG0 (taking into account the half-pixel) This has a couple of benefits over hamming distance of the bit patterns: - transposed pixels are weighted less (edit distance 1, not 2+ for Hamming) - coloured pixels are weighted equally as white pixels (not half as much) - weighting changes in palette bit that flip multiple pixel colours While I'm here, the RLE opcode should emit run_length - 1 so that we can encode runs of 256 bytes.
2019-03-04 23:09:00 +00:00
@functools.lru_cache(None)
def byte_to_colour_string(b: int, is_odd_offset: bool) -> str:
pixels = []
idx = 0
if is_odd_offset:
pixels.append("01"[b & 0x01])
idx += 1
# K = black
# G = green
# V = violet
# W = white
palettes = (
(
"K", # 0x00
"V", # 0x01
"G", # 0x10
"W" # 0x11
), (
"K", # 0x00
Add some tests for edit_weight and byte_to_colour_string
2019-03-10 23:07:44 +00:00
"B", # 0x01
"O", # 0x10
Introduction more general notion of update priority used to increase weight of diffs that persist across multiple frames. For each frame, zero out update priority of bytes that no longer have a pending diff, and add the edit distance of the remaining diffs. Zero these out as opcodes are retired. Replace hamming distance with Damerau-Levenshtein distance of the encoded pixel colours in the byte, e.g. 0x2A --> GGG0 (taking into account the half-pixel) This has a couple of benefits over hamming distance of the bit patterns: - transposed pixels are weighted less (edit distance 1, not 2+ for Hamming) - coloured pixels are weighted equally as white pixels (not half as much) - weighting changes in palette bit that flip multiple pixel colours While I'm here, the RLE opcode should emit run_length - 1 so that we can encode runs of 256 bytes.
2019-03-04 23:09:00 +00:00
"W" # 0x11
)
)
Add some tests for edit_weight and byte_to_colour_string
2019-03-10 23:07:44 +00:00
palette = palettes[(b & 0x80) != 0]
Introduction more general notion of update priority used to increase weight of diffs that persist across multiple frames. For each frame, zero out update priority of bytes that no longer have a pending diff, and add the edit distance of the remaining diffs. Zero these out as opcodes are retired. Replace hamming distance with Damerau-Levenshtein distance of the encoded pixel colours in the byte, e.g. 0x2A --> GGG0 (taking into account the half-pixel) This has a couple of benefits over hamming distance of the bit patterns: - transposed pixels are weighted less (edit distance 1, not 2+ for Hamming) - coloured pixels are weighted equally as white pixels (not half as much) - weighting changes in palette bit that flip multiple pixel colours While I'm here, the RLE opcode should emit run_length - 1 so that we can encode runs of 256 bytes.
2019-03-04 23:09:00 +00:00
for _ in range(3):
pixel = palette[(b >> idx) & 0b11]
pixels.append(pixel)
idx += 2
if not is_odd_offset:
pixels.append("01"[b & 0x40 != 0])
idx += 1
return "".join(pixels)
Refactor the world
2019-01-05 23:31:56 +00:00
class Video:
"""Apple II screen memory map encoding a bitmapped frame."""
Video() is now aware of target frame rate, and will continue to emit opcodes until the cycle budget for the frame is exhausted. Output stream is also now aware of TCP framing, and schedules an ACK opcode every 2048 output bytes to instruct the client to perform TCP ACK and buffer management. Fixes several serious bugs in RLE encoding, including: - we were emitting the RLE opcode with the next content byte after the run completed! - we were looking at the wrong field for the start offset! - handle the case where the entire page is a single run - stop trying to allow accumulating error when RLE -- this does not respect the Apple II colour encoding, i.e. may introduce colour fringing. - also because of this we're unlikely to actually be able to find many runs because odd and even columns are encoded differently. In a followup we should start encoding odd and even columns separately Optimize after profiling -- encoder is now about 2x faster Add tests.
2019-02-23 23:52:25 +00:00
CLOCK_SPEED = 1024 * 1024
Integrated audio + video player! - Introduce a new Movie() class that multiplexes audio and video. - Every N audio frames we grab a new video frame and begin pulling opcodes from the audio and video streams - Grab frames from the input video using bmp2dhr if the .BIN file does not already exist. Run bmp2dhr in a background thread to not block encoding - move the output byte streaming from Video to Movie - For now, manually clip updates to pages > 56 since the client doesn't support them yet The way we encode video is now: - iterate in descending order over update_priority - begin a new (page, content) opcode - for all of the other offset bytes in that page, compute the error between the candidate content byte and the target content byte - iterate over offsets in order of increasing error and decreasing update_priority to fill out the remaining opcode
2019-03-07 23:07:24 +00:00
def __init__(
self,
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
filename: str):
self.filename = filename # type: str
self._reader = skvideo.io.FFmpegReader(filename)
# Compute frame rate from input video
data = skvideo.io.ffprobe(self.filename)['video']
rate_data = data['@r_frame_rate'].split("/") # e.g. 12000/1001
self._input_frame_rate = float(rate_data[0]) / float(rate_data[1])
self.cycles_per_frame = 1024. * 1024 / self._input_frame_rate
self.frame_number = 0
Refactor the world
2019-01-05 23:31:56 +00:00
# Initialize empty
Refactor the various representations of screen memory (bitmap, (x,y) bytemap, (page,offset) memory map) - add a FlatMemoryMap that is a linear 8K array - add converter methods and default constructors that allow converting between them - use MemoryMap as the central representation used by the video encoder
2019-03-03 22:21:28 +00:00
self.memory_map = screen.MemoryMap(
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
screen_page=1) # type: screen.MemoryMap
Video() is now aware of target frame rate, and will continue to emit opcodes until the cycle budget for the frame is exhausted. Output stream is also now aware of TCP framing, and schedules an ACK opcode every 2048 output bytes to instruct the client to perform TCP ACK and buffer management. Fixes several serious bugs in RLE encoding, including: - we were emitting the RLE opcode with the next content byte after the run completed! - we were looking at the wrong field for the start offset! - handle the case where the entire page is a single run - stop trying to allow accumulating error when RLE -- this does not respect the Apple II colour encoding, i.e. may introduce colour fringing. - also because of this we're unlikely to actually be able to find many runs because odd and even columns are encoded differently. In a followup we should start encoding odd and even columns separately Optimize after profiling -- encoder is now about 2x faster Add tests.
2019-02-23 23:52:25 +00:00
Introduction more general notion of update priority used to increase weight of diffs that persist across multiple frames. For each frame, zero out update priority of bytes that no longer have a pending diff, and add the edit distance of the remaining diffs. Zero these out as opcodes are retired. Replace hamming distance with Damerau-Levenshtein distance of the encoded pixel colours in the byte, e.g. 0x2A --> GGG0 (taking into account the half-pixel) This has a couple of benefits over hamming distance of the bit patterns: - transposed pixels are weighted less (edit distance 1, not 2+ for Hamming) - coloured pixels are weighted equally as white pixels (not half as much) - weighting changes in palette bit that flip multiple pixel colours While I'm here, the RLE opcode should emit run_length - 1 so that we can encode runs of 256 bytes.
2019-03-04 23:09:00 +00:00
# Accumulates pending edit weights across frames
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
self.update_priority = np.zeros((32, 256), dtype=np.int64)
def tick(self, cycles) -> bool:
# print(cycles, self.cycles_per_frame, self.cycles_per_frame *
# self.frame_number)
if cycles > (self.cycles_per_frame * self.frame_number):
self.frame_number += 1
return True
return False
Introduction more general notion of update priority used to increase weight of diffs that persist across multiple frames. For each frame, zero out update priority of bytes that no longer have a pending diff, and add the edit distance of the remaining diffs. Zero these out as opcodes are retired. Replace hamming distance with Damerau-Levenshtein distance of the encoded pixel colours in the byte, e.g. 0x2A --> GGG0 (taking into account the half-pixel) This has a couple of benefits over hamming distance of the bit patterns: - transposed pixels are weighted less (edit distance 1, not 2+ for Hamming) - coloured pixels are weighted equally as white pixels (not half as much) - weighting changes in palette bit that flip multiple pixel colours While I'm here, the RLE opcode should emit run_length - 1 so that we can encode runs of 256 bytes.
2019-03-04 23:09:00 +00:00
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
def _frame_grabber(self):
for frame_array in self._reader.nextFrame():
yield Image.fromarray(frame_array)
Refactor the world
2019-01-05 23:31:56 +00:00
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
def frames(self) -> Iterator[screen.MemoryMap]:
"""Encode frame to HGR using bmp2dhr.
Refactor the world
2019-01-05 23:31:56 +00:00
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
We do the encoding in a background thread to parallelize.
Refactor the world
2019-01-05 23:31:56 +00:00
"""
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
frame_dir = self.filename.split(".")[0]
try:
os.mkdir(frame_dir)
except FileExistsError:
pass
q = queue.Queue(maxsize=10)
def worker():
for _idx, _frame in enumerate(self._frame_grabber()):
outfile = "%s/%08dC.BIN" % (frame_dir, _idx)
bmpfile = "%s/%08d.bmp" % (frame_dir, _idx)
try:
os.stat(outfile)
except FileNotFoundError:
_frame = _frame.resize((280, 192))
_frame.save(bmpfile)
subprocess.call(
["/usr/local/bin/bmp2dhr", bmpfile, "hgr", "D9"])
os.remove(bmpfile)
_frame = np.fromfile(outfile, dtype=np.uint8)
q.put(_frame)
q.put(None)
t = threading.Thread(target=worker)
t.start()
while True:
frame = q.get()
if frame is None:
break
yield screen.FlatMemoryMap(
screen_page=1, data=frame).to_memory_map()
q.task_done()
t.join()
def encode_frame(
self, target: screen.MemoryMap
) -> Iterator[opcodes.Opcode]:
"""Update to match content of frame within provided budget."""
print("Similarity %f" % (self.update_priority.mean()))
Integrated audio + video player! - Introduce a new Movie() class that multiplexes audio and video. - Every N audio frames we grab a new video frame and begin pulling opcodes from the audio and video streams - Grab frames from the input video using bmp2dhr if the .BIN file does not already exist. Run bmp2dhr in a background thread to not block encoding - move the output byte streaming from Video to Movie - For now, manually clip updates to pages > 56 since the client doesn't support them yet The way we encode video is now: - iterate in descending order over update_priority - begin a new (page, content) opcode - for all of the other offset bytes in that page, compute the error between the candidate content byte and the target content byte - iterate over offsets in order of increasing error and decreasing update_priority to fill out the remaining opcode
2019-03-07 23:07:24 +00:00
yield from self._index_changes(self.memory_map, target)
Video() is now aware of target frame rate, and will continue to emit opcodes until the cycle budget for the frame is exhausted. Output stream is also now aware of TCP framing, and schedules an ACK opcode every 2048 output bytes to instruct the client to perform TCP ACK and buffer management. Fixes several serious bugs in RLE encoding, including: - we were emitting the RLE opcode with the next content byte after the run completed! - we were looking at the wrong field for the start offset! - handle the case where the entire page is a single run - stop trying to allow accumulating error when RLE -- this does not respect the Apple II colour encoding, i.e. may introduce colour fringing. - also because of this we're unlikely to actually be able to find many runs because odd and even columns are encoded differently. In a followup we should start encoding odd and even columns separately Optimize after profiling -- encoder is now about 2x faster Add tests.
2019-02-23 23:52:25 +00:00
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
def _diff_weights(
Video() is now aware of target frame rate, and will continue to emit opcodes until the cycle budget for the frame is exhausted. Output stream is also now aware of TCP framing, and schedules an ACK opcode every 2048 output bytes to instruct the client to perform TCP ACK and buffer management. Fixes several serious bugs in RLE encoding, including: - we were emitting the RLE opcode with the next content byte after the run completed! - we were looking at the wrong field for the start offset! - handle the case where the entire page is a single run - stop trying to allow accumulating error when RLE -- this does not respect the Apple II colour encoding, i.e. may introduce colour fringing. - also because of this we're unlikely to actually be able to find many runs because odd and even columns are encoded differently. In a followup we should start encoding odd and even columns separately Optimize after profiling -- encoder is now about 2x faster Add tests.
2019-02-23 23:52:25 +00:00
self,
Refactor the various representations of screen memory (bitmap, (x,y) bytemap, (page,offset) memory map) - add a FlatMemoryMap that is a linear 8K array - add converter methods and default constructors that allow converting between them - use MemoryMap as the central representation used by the video encoder
2019-03-03 22:21:28 +00:00
source: screen.MemoryMap,
target: screen.MemoryMap
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
):
diff_weights = np.zeros((32, 256), dtype=np.int64)
Introduction more general notion of update priority used to increase weight of diffs that persist across multiple frames. For each frame, zero out update priority of bytes that no longer have a pending diff, and add the edit distance of the remaining diffs. Zero these out as opcodes are retired. Replace hamming distance with Damerau-Levenshtein distance of the encoded pixel colours in the byte, e.g. 0x2A --> GGG0 (taking into account the half-pixel) This has a couple of benefits over hamming distance of the bit patterns: - transposed pixels are weighted less (edit distance 1, not 2+ for Hamming) - coloured pixels are weighted equally as white pixels (not half as much) - weighting changes in palette bit that flip multiple pixel colours While I'm here, the RLE opcode should emit run_length - 1 so that we can encode runs of 256 bytes.
2019-03-04 23:09:00 +00:00
it = np.nditer(
Integrated audio + video player! - Introduce a new Movie() class that multiplexes audio and video. - Every N audio frames we grab a new video frame and begin pulling opcodes from the audio and video streams - Grab frames from the input video using bmp2dhr if the .BIN file does not already exist. Run bmp2dhr in a background thread to not block encoding - move the output byte streaming from Video to Movie - For now, manually clip updates to pages > 56 since the client doesn't support them yet The way we encode video is now: - iterate in descending order over update_priority - begin a new (page, content) opcode - for all of the other offset bytes in that page, compute the error between the candidate content byte and the target content byte - iterate over offsets in order of increasing error and decreasing update_priority to fill out the remaining opcode
2019-03-07 23:07:24 +00:00
source.page_offset ^ target.page_offset, flags=['multi_index'])
Introduction more general notion of update priority used to increase weight of diffs that persist across multiple frames. For each frame, zero out update priority of bytes that no longer have a pending diff, and add the edit distance of the remaining diffs. Zero these out as opcodes are retired. Replace hamming distance with Damerau-Levenshtein distance of the encoded pixel colours in the byte, e.g. 0x2A --> GGG0 (taking into account the half-pixel) This has a couple of benefits over hamming distance of the bit patterns: - transposed pixels are weighted less (edit distance 1, not 2+ for Hamming) - coloured pixels are weighted equally as white pixels (not half as much) - weighting changes in palette bit that flip multiple pixel colours While I'm here, the RLE opcode should emit run_length - 1 so that we can encode runs of 256 bytes.
2019-03-04 23:09:00 +00:00
while not it.finished:
Integrated audio + video player! - Introduce a new Movie() class that multiplexes audio and video. - Every N audio frames we grab a new video frame and begin pulling opcodes from the audio and video streams - Grab frames from the input video using bmp2dhr if the .BIN file does not already exist. Run bmp2dhr in a background thread to not block encoding - move the output byte streaming from Video to Movie - For now, manually clip updates to pages > 56 since the client doesn't support them yet The way we encode video is now: - iterate in descending order over update_priority - begin a new (page, content) opcode - for all of the other offset bytes in that page, compute the error between the candidate content byte and the target content byte - iterate over offsets in order of increasing error and decreasing update_priority to fill out the remaining opcode
2019-03-07 23:07:24 +00:00
# If no diff, don't need to bother
if not it[0]:
it.iternext()
continue
Introduction more general notion of update priority used to increase weight of diffs that persist across multiple frames. For each frame, zero out update priority of bytes that no longer have a pending diff, and add the edit distance of the remaining diffs. Zero these out as opcodes are retired. Replace hamming distance with Damerau-Levenshtein distance of the encoded pixel colours in the byte, e.g. 0x2A --> GGG0 (taking into account the half-pixel) This has a couple of benefits over hamming distance of the bit patterns: - transposed pixels are weighted less (edit distance 1, not 2+ for Hamming) - coloured pixels are weighted equally as white pixels (not half as much) - weighting changes in palette bit that flip multiple pixel colours While I'm here, the RLE opcode should emit run_length - 1 so that we can encode runs of 256 bytes.
2019-03-04 23:09:00 +00:00
diff_weights[it.multi_index] = edit_weight(
source.page_offset[it.multi_index],
target.page_offset[it.multi_index],
it.multi_index[1] % 2 == 1
)
it.iternext()
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
return diff_weights
Introduction more general notion of update priority used to increase weight of diffs that persist across multiple frames. For each frame, zero out update priority of bytes that no longer have a pending diff, and add the edit distance of the remaining diffs. Zero these out as opcodes are retired. Replace hamming distance with Damerau-Levenshtein distance of the encoded pixel colours in the byte, e.g. 0x2A --> GGG0 (taking into account the half-pixel) This has a couple of benefits over hamming distance of the bit patterns: - transposed pixels are weighted less (edit distance 1, not 2+ for Hamming) - coloured pixels are weighted equally as white pixels (not half as much) - weighting changes in palette bit that flip multiple pixel colours While I'm here, the RLE opcode should emit run_length - 1 so that we can encode runs of 256 bytes.
2019-03-04 23:09:00 +00:00
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
def _heapify_priorities(self) -> List:
Integrated audio + video player! - Introduce a new Movie() class that multiplexes audio and video. - Every N audio frames we grab a new video frame and begin pulling opcodes from the audio and video streams - Grab frames from the input video using bmp2dhr if the .BIN file does not already exist. Run bmp2dhr in a background thread to not block encoding - move the output byte streaming from Video to Movie - For now, manually clip updates to pages > 56 since the client doesn't support them yet The way we encode video is now: - iterate in descending order over update_priority - begin a new (page, content) opcode - for all of the other offset bytes in that page, compute the error between the candidate content byte and the target content byte - iterate over offsets in order of increasing error and decreasing update_priority to fill out the remaining opcode
2019-03-07 23:07:24 +00:00
priorities = []
it = np.nditer(self.update_priority, flags=['multi_index'])
while not it.finished:
priority = it[0]
if not priority:
it.iternext()
continue
page, offset = it.multi_index
# Don't use deterministic order for page, offset
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
nonce = random.random()
Integrated audio + video player! - Introduce a new Movie() class that multiplexes audio and video. - Every N audio frames we grab a new video frame and begin pulling opcodes from the audio and video streams - Grab frames from the input video using bmp2dhr if the .BIN file does not already exist. Run bmp2dhr in a background thread to not block encoding - move the output byte streaming from Video to Movie - For now, manually clip updates to pages > 56 since the client doesn't support them yet The way we encode video is now: - iterate in descending order over update_priority - begin a new (page, content) opcode - for all of the other offset bytes in that page, compute the error between the candidate content byte and the target content byte - iterate over offsets in order of increasing error and decreasing update_priority to fill out the remaining opcode
2019-03-07 23:07:24 +00:00
heapq.heappush(priorities, (-priority, nonce, page, offset))
it.iternext()
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
return priorities
@functools.lru_cache(None)
def _compute_delta(self, content, target, old, is_odd):
return edit_weight(content, target, is_odd, error=True) - old
def _compute_error(self, page, content, target, old_error):
offsets = []
old_error_page = old_error[page]
tpo = target.page_offset[page]
page_priorities = [(-p, random.random(), o) for o, p in enumerate(
self.update_priority[page]) if p]
heapq.heapify(page_priorities)
# Iterate in descending priority order and take first 3 offsets with
# negative delta
while page_priorities:
_, _, o = heapq.heappop(page_priorities)
# If we store content at this offset, what is the difference
# between this edit distance and the ideal target edit distance?
delta = self._compute_delta(
content, tpo[o], o % 2 == 1, old_error_page[o])
# Getting further away from goal, no thanks!
if delta >= 0:
continue
#
# # print("Offset %d prio %d: %d -> %d = %d" % (
# # o, p, content,
# # target.page_offset[page, o],
# # delta
# # ))
offsets.append(o)
if len(offsets) == 3:
break
return offsets
def _index_changes(
self,
source: screen.MemoryMap,
target: screen.MemoryMap
) -> Iterator[Tuple[int, int, int, int, int]]:
"""Transform encoded screen to sequence of change tuples.
Change tuple is (update_priority, page, offset, content, run_length)
"""
diff_weights = self._diff_weights(source, target)
# Clear any update priority entries that have resolved themselves
# with new frame
self.update_priority[diff_weights == 0] = 0
self.update_priority += diff_weights
priorities = self._heapify_priorities()
while priorities:
_, _, page, offset = heapq.heappop(priorities)
# Check whether we've already cleared this diff while processing
# an earlier opcode
if self.update_priority[page, offset] == 0:
Integrated audio + video player! - Introduce a new Movie() class that multiplexes audio and video. - Every N audio frames we grab a new video frame and begin pulling opcodes from the audio and video streams - Grab frames from the input video using bmp2dhr if the .BIN file does not already exist. Run bmp2dhr in a background thread to not block encoding - move the output byte streaming from Video to Movie - For now, manually clip updates to pages > 56 since the client doesn't support them yet The way we encode video is now: - iterate in descending order over update_priority - begin a new (page, content) opcode - for all of the other offset bytes in that page, compute the error between the candidate content byte and the target content byte - iterate over offsets in order of increasing error and decreasing update_priority to fill out the remaining opcode
2019-03-07 23:07:24 +00:00
continue
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
Integrated audio + video player! - Introduce a new Movie() class that multiplexes audio and video. - Every N audio frames we grab a new video frame and begin pulling opcodes from the audio and video streams - Grab frames from the input video using bmp2dhr if the .BIN file does not already exist. Run bmp2dhr in a background thread to not block encoding - move the output byte streaming from Video to Movie - For now, manually clip updates to pages > 56 since the client doesn't support them yet The way we encode video is now: - iterate in descending order over update_priority - begin a new (page, content) opcode - for all of the other offset bytes in that page, compute the error between the candidate content byte and the target content byte - iterate over offsets in order of increasing error and decreasing update_priority to fill out the remaining opcode
2019-03-07 23:07:24 +00:00
offsets = [offset]
content = target.page_offset[page, offset]
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
# print("Priority %d: page %d offset %d content %d" % (
Integrated audio + video player! - Introduce a new Movie() class that multiplexes audio and video. - Every N audio frames we grab a new video frame and begin pulling opcodes from the audio and video streams - Grab frames from the input video using bmp2dhr if the .BIN file does not already exist. Run bmp2dhr in a background thread to not block encoding - move the output byte streaming from Video to Movie - For now, manually clip updates to pages > 56 since the client doesn't support them yet The way we encode video is now: - iterate in descending order over update_priority - begin a new (page, content) opcode - for all of the other offset bytes in that page, compute the error between the candidate content byte and the target content byte - iterate over offsets in order of increasing error and decreasing update_priority to fill out the remaining opcode
2019-03-07 23:07:24 +00:00
# priority, page, offset, content))
# Clear priority for the offset we're emitting
self.update_priority[page, offset] = 0
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
self.memory_map.page_offset[page, offset] = content
Integrated audio + video player! - Introduce a new Movie() class that multiplexes audio and video. - Every N audio frames we grab a new video frame and begin pulling opcodes from the audio and video streams - Grab frames from the input video using bmp2dhr if the .BIN file does not already exist. Run bmp2dhr in a background thread to not block encoding - move the output byte streaming from Video to Movie - For now, manually clip updates to pages > 56 since the client doesn't support them yet The way we encode video is now: - iterate in descending order over update_priority - begin a new (page, content) opcode - for all of the other offset bytes in that page, compute the error between the candidate content byte and the target content byte - iterate over offsets in order of increasing error and decreasing update_priority to fill out the remaining opcode
2019-03-07 23:07:24 +00:00
# Need to find 3 more offsets to fill this opcode
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
for o in self._compute_error(
page,
Integrated audio + video player! - Introduce a new Movie() class that multiplexes audio and video. - Every N audio frames we grab a new video frame and begin pulling opcodes from the audio and video streams - Grab frames from the input video using bmp2dhr if the .BIN file does not already exist. Run bmp2dhr in a background thread to not block encoding - move the output byte streaming from Video to Movie - For now, manually clip updates to pages > 56 since the client doesn't support them yet The way we encode video is now: - iterate in descending order over update_priority - begin a new (page, content) opcode - for all of the other offset bytes in that page, compute the error between the candidate content byte and the target content byte - iterate over offsets in order of increasing error and decreasing update_priority to fill out the remaining opcode
2019-03-07 23:07:24 +00:00
content,
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
target,
diff_weights
):
offsets.append(o)
# Clear priority for the offset we're emitting
self.update_priority[page, o] = 0
self.memory_map.page_offset[page, o] = content
# Pad to 4 if we didn't find enough
Integrated audio + video player! - Introduce a new Movie() class that multiplexes audio and video. - Every N audio frames we grab a new video frame and begin pulling opcodes from the audio and video streams - Grab frames from the input video using bmp2dhr if the .BIN file does not already exist. Run bmp2dhr in a background thread to not block encoding - move the output byte streaming from Video to Movie - For now, manually clip updates to pages > 56 since the client doesn't support them yet The way we encode video is now: - iterate in descending order over update_priority - begin a new (page, content) opcode - for all of the other offset bytes in that page, compute the error between the candidate content byte and the target content byte - iterate over offsets in order of increasing error and decreasing update_priority to fill out the remaining opcode
2019-03-07 23:07:24 +00:00
for _ in range(len(offsets), 4):
offsets.append(offsets[0])
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
# print("Page %d, content %d: offsets %s" % (page+32, content,
Integrated audio + video player! - Introduce a new Movie() class that multiplexes audio and video. - Every N audio frames we grab a new video frame and begin pulling opcodes from the audio and video streams - Grab frames from the input video using bmp2dhr if the .BIN file does not already exist. Run bmp2dhr in a background thread to not block encoding - move the output byte streaming from Video to Movie - For now, manually clip updates to pages > 56 since the client doesn't support them yet The way we encode video is now: - iterate in descending order over update_priority - begin a new (page, content) opcode - for all of the other offset bytes in that page, compute the error between the candidate content byte and the target content byte - iterate over offsets in order of increasing error and decreasing update_priority to fill out the remaining opcode
2019-03-07 23:07:24 +00:00
# offsets))
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
yield (page + 32, content, offsets)
Video() is now aware of target frame rate, and will continue to emit opcodes until the cycle budget for the frame is exhausted. Output stream is also now aware of TCP framing, and schedules an ACK opcode every 2048 output bytes to instruct the client to perform TCP ACK and buffer management. Fixes several serious bugs in RLE encoding, including: - we were emitting the RLE opcode with the next content byte after the run completed! - we were looking at the wrong field for the start offset! - handle the case where the entire page is a single run - stop trying to allow accumulating error when RLE -- this does not respect the Apple II colour encoding, i.e. may introduce colour fringing. - also because of this we're unlikely to actually be able to find many runs because odd and even columns are encoded differently. In a followup we should start encoding odd and even columns separately Optimize after profiling -- encoder is now about 2x faster Add tests.
2019-02-23 23:52:25 +00:00
Read video frame rate and encode a new frame when the cycle count has ticked past the appropriate time. - optimize the frame encoding a bit - use int64 consistently to avoid casting Fix a bug - when retiring an offset, also update our memory map with the new content, oops If we run out of changes to index, keep emitting stores for content at page=32,offset=0 forever Switch to a weighted D-L implementation so we can weight e.g. different substitutions differently (e.g. weighting diffs to/from black pixels differently than color errors)
2019-03-10 22:42:31 +00:00
# If we run out of things to do, pad forever
content = target.page_offset[(0, 0)]
while True:
yield (32, content, [0, 0, 0, 0])
Reference in New Issue Copy Permalink