I'm not sure using a hexdump makes sense for str() here, but I don't know what
else does yet. I also know that I want a buffer to be hexdumpable, although I
think I'd prefer something that allows a hexdump not to require a huge amount
of memory. Until we start processing HFS, I don't need to dwell on that too
much. Let's get it working first.
The nex gen_hexdump returns an Iterator to give you a hex dump one line at a
time. This is the thing to use (probably) if you want to do something useful
with a hexdump other than print it, say. The hexdump function now takes
another argument, func, a callable meant to take a string, defaulting to print.
It just uses the Iterator.
I think I'm done messing with the API to this and can soon start actually
committing some code that uses it now and then. ;)
Type hinting in Python is just that: Hinting. Which means if you do it wrong,
as long as the syntax is right, Python itself will just let you do it. Python
is intended to be a duck-typed language, so you're really just documenting you
intent/expectations. The feature is somewhat new, and its conventions are not
completely understood by certain tools that encounter it. This apparently
applies to myself as a developer trying to use it. ;)
I explained in the comments on BufferType why I'm doing this, but the nutshell
version is that I anticipate having bigger files to deal with at some point we
won't want to keep in memory. Otherwise we could just use bytearrays.
The way this is meant to be used (which admittedly isn't clear--would someone
like to submit a patch that improves the docs to clarify?) is that this is
intended to be used as a context. In other words, using Python's with
statement. This isn't all that different for a ByteBuffer, but it would be for
a FileBuffer (which doesn't exist yet and won't for awhile.)
Implementation hint for FileBuffer when I get there: If the file is not
explicitly opened read-only, I intend for read-modify-write to be standard
practice from the start. That'll mean duplicating a file to a temporary one we
can manipulate safely and then at flush time, copying the changes over the
original. That way you'd always be able to undo your changes by quitting
without saving. This seems important as blocksfree is likely to serve a lot of
archival duty and you may only get one shot at trying to save an image from a
damaged floppy. It would be awful if that image were then destroyed by an
accidental exception somewhere in the middle of other operations. So let's not
go there.
What this is missing is a flush method. It's not implemented because:
a. cppo is all we've got so far and it doesn't need write at all
b. I'm not 100% sure how I'm doing files yet
c. I'm hoping the way I ought to implement this will make itself apparent when
the code gets used.
Python3 already has a means of turning numerical data into a binary string with
format(). The only place it was used was with ProDOS case masks as it was, so
it's an easy call to replace the specialty function with Python's internals.
The util functions consist entirely of hexdump and its helper function right
now, both of which are completely unused at the moment. I don't intend for
legacy to ever call these functions, but I should start using them soon. :)
I may not have done this 100% "properly"--this is really the first full
application thingy I've ever tried to write in Python. I learned circular
imports are possible and the error messages are not obvious when you do that.
I've also learned that importing a package doesn't necessarily import the
modules within that package--if you want that, the package needs to import its
submodules into itself. That was not obvious either. It does explain why
sometimes you must import subpackages directly and other times importing the
package is all you need. This is probably obvious to Python developers who
actually work on big projects, but ... I've only ever done single-file scripts
before now so it wasn't obvious to me.
For now, blocksfree is importing legacy. We don't have enough outside of
legacy yet to make the alternative even remotely useful at this time.
Eventually though the goal is to stop doing that.
Basically cleaning out my stash stack--the stash this came from had been mostly
applied elsewhere already, leaving only a few stray ()s. Figured it was a good
time to PEP 8 the end-of-line comments just so there was something here to
actually commit.
You now simply stuff g with the appropriate options and run the thing. You
could even modify the function to take those things as arguments now, but I
didn't do that for now.
At present we actually have a reasonably high Python minimum of 3.5. We could
back that off to 3.3 without much effort or perhaps even 3.2. Once we're a
little closer to something to release, we should consider doing that.
Not quite finished with this, but the goal here is not have args being passed
in to the legacy cppo at all. The g namespace is not ideal, but it does work
and it isolates the legacy code from needing to understand what's going on in
the shell level. So we'll take advantage of that for the time being.
The bigger problem was that when I moved the arg parsing out of cppo, I failed
to move all of it--a block that checked the number of args got lost. Restored.
The point is to separate the CLI interface to cppo from the inner logic so that
we can begin replacing the legacy code with proper implementations thereof.
This isn't 100% complete in that regard--we still need to pass args to the
legacy main function--but the part of cppo that will survive this whole process
is now functionally isolated from the part that's likely to get replaced to a
large degree.
The history document is kind of a mishmash of explanation about what decisions
have lead to what this project is trying to do and why this rather than other
things, such as improving AppleCommander. (Ohh, it has the reason for that
believe me--die in the cash-consuming fire of the Internet's rage, Oracle!)
More importantly, there are Copyright notices and the GNU GPL v2.
This is kind of an expensive thing to do unconditionally, but it lets us make
multi-line strings fit into the code with less ugliness. Basically, if you're
four levels in, you can do something like this:
log.warn("""\
There was a problem.
It was probably wasn't fatal because this
is only a warning, but it is enough to have
a multiline string.
""")
This will print without the indentation. It's not quite as clean as how
docutils handles docstrings (allowing the first line to be unindented, hence
the line-continuation), but it's still an improvement. If you can improve upon
this, please feel free to PR it!
The section of cppo containing the logging code has been moved to its own very
short module inside a (bare) Python package. This is messily done for now, but
I wanted this to be a minimal commit.
CiderPress checks that these optional areas are actually as long as the header
says they are and discards them if they're not. This seems like a useful
heuristic. It happens to be the only one we can perform. :)
Rwrite of the 2mg parsing code which saves the comment and creator blocks in
memory by putting the code into the Disk class.
Doesn't yet attempt to parse out 2mg image format, and the question is open as
to whether or not I'll try to follow cppo's historical method of handling this
(chop the header and go), AppleCommander's method (consider 2mg its own unique
image format which happens to contain one of the others) or CiderPress's (use
one of several possible wrappers independent of the image orders. CP's is
probably the most flexible.
So basically this code is probably far from final, but it works for what it
does so far.
What's wrong with b2a_hex() or hex()? Well, hex() only converts integers. And
while a2b_hex() ignores whitespace, b2a_hex() doesn't provide any, making for
difficult to read output for anything longer than about 8 bytes or so.
In the basic case, it seems like you want a classic hexdump. I chose the xxd
format:
xxxxxxxx: xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx |cccccccccccccccc|
Rather than hardcode all of the integers and strings (as I started doing), I
decided that I might as well use variables for these things if only for
readability. And if they're locals, you might as well be able to override
them.
The knobs you have to play with are therefore these:
- wordsize=2, how many bytes are grouped together
- sep=' ', the spacing between words
- sep2=' ', the midpoint spacing
I suppose I could've made everything else configurable too, but YAGNI.
Finally! The Disk class doesn't actually serve as much more than a slightly
improved Globals class at the moment holding every splitting of the source path
and filename that we use in legacy code, as well as a copy of the disk image
itself that gets used long enough to read the a2mg header.
The idea I have here is to begin building the module-based code in parallel.
Then I'll just modify the linear code to compare doing it the old way to doing
it the new. That'll let me verify that the new code does what the old should.
When it's all done, we can just modify main to use the new modular code and
look at splitting the modular code into a package with cppo as a runner. At
that point the code should begin being able to do things cppo cannot. We could
continue to extend cppo at that point, but my inclination is to maintain the
cppo runner as a compatibility layer and begin building a more modern image
tool. Essentially to begin building the CiderPress for Linux or the Java-free
AppleCommander.
The methods int.to_bytes() and int.from_bytes() are ... both messy and ugly,
contrary to the Zen of Python. Right now, we're just writing out single ints
and words, but eventually we'll be reading and writing whole structures, so it
makes sense to begin moving the codebase in that direction.
I've written only the functions I either use or see a use for immediately, save
for pack_u32be() which I wrote for date conversions, and then immediately
realized we don't want to use for that purpose yet. We can remove it if we
don't ultimately need it.
Replaced some of the commented out print lines with calls to a logger object.
These are currently always turned on, which we don't want, but they don't hurt
anything and are short.
Oh, and new logging system! The setup is a bit more verbose than it could be
because logging predates str.format and uses the str % tuple syntax that now
exists mostly to avoid breaking older code. You can override this, and I did.
It's not done yet because we might want to actually make some of the existing
print calls into log.info's. Y'know, just as soon as I set that up so ONLY
logging.INFO goes to stdout, unadorned, and everything higher than that goes to
stderr, depending on your logging level, with pretty formatting.
Yeah, logging can do all of that and chew bubblegum at the same time, I just
haven't set it up yet because I want to do it right.
A little more unStudlyCapping of things. I'm going to have to start actually
creating classes soon which is going to bring back the capitals, but I've been
working to get rid of them so that it becomes less confusing when we get there.
I dunno if it's helped any.
I also added a few comments about our imports and checked that we actually used
everything we imported. No, we don't. But we maybe should switch what we are
using for what we aren't at some point?
Function now takes raw bytes containing two little-endian 16-bit words right
out of a disk image. It extracts timestamp components using bit shifting and
bitwise operators and asks datetime.datetime to give us a timestamp from the
result. Caller need not catch exceptions for this process anymore. Either it
works or you get None back.
The full summary:
**ADDED**
- New dopo_swap swaps 140k images between DOS order and ProDOS order. This
function replaces code in the main program body which does the same thing.
- Add optional zfill parameter to to_bin. Considered this for to_hex, but
nowhere would that be used currently and I'd rather get rid of these lower
level support functions that mainly are there for Bashbyter (but for now
have larger direct use now that Bashbyter is gone.)
**CHANGED**
- In getFileLength for DOS 3.3 T/other files, initialize prevTSpair to [0,0]
which, when combined with the removal of Bashbyter and vestiges of Python2
support, makes `The Correspondent 4.4.dsk` both -cat and extract. (Noted
previously, we currently do nothing about the control characters in the
filenames on this disk. They extract as non-printables and they don't show
up properly in -cat.)
- Replaced 4294967296 (2**32) with 1<<32 for use in turning negative integers
into unsigned 32 bit integers. It's possible to int.to_bytes() in a way
that does this the way we want, and we ought to do that. The syntax is
longer than it needs to be though.
- Strip high bit of DOS 3.3 filenames more efficiently
- Replaced type("".encode().decode()) with str. That wasn't necessary, and
you might think otherwise is an example of why dropping Python 2 is a very
good idea.
- Use int.from_bytes() calls to replace reading several bytes by hand,
multiplying them by bit offsets, and adding them together.
- Made unixDateToADDate return four bytes instead of a hex-ustr because it
only had one caller which just converted the value to that format anyway.
- Misc slight changes like unStudlyCapping identifiers, saving a return value
rather than calling the function that creates it multiple times, tuple
assignment, and coding style
**REMOVED**
- slyce functions: {,bin_,dec_,hex_}slyce
- aToB conversions: binTo{Dec,Hex}, charTo{Dec,Hex}, decTo{Char,Hex},
hexTo{Bin,Char,Dec} (just use to_thing or do it better than that.)
- Removed: readchar{s,Dec,Hex}, writechar{s,sHex,Dec,Hex}
We're using the latest community GSOS release as a convenient example of a
large ProDOS volume with lots of directory structure, forks, and at least a few
files that can be identified by eyeball as having been extracted correctly.
Then I went looking for something DOS 3.3 with T type files on the disk to
verify those were being extracted correctly. Randomly stumbled across The
Correspondent 4.4 which has them, but also had some surprises for us. It
crashed cppo! A combination of removing bashbyter functions and ensuring a
variable gets initialized caused cppo to be able to cat and dump it. We do
nothing special about filenames made up entirely of control characters,
however, so they'll print incorrectly and extract as non-printable characters
in your filesystem. It's legal (if crazy) on most UNIX filesystems. I have
_no idea_ if or how to handle these things on macOS or Windows.
The old way involved a lot more sequence duplication. Now just turn the
bytes object into a mutable bytearray, iterate through the mask and
change what we need, then change it back.
A few clumsy if statements got rewritten with truth tables to verify
that the somewhat simplified conditions still evaluated the same. Other
changes are mostly cosmetic.
Originally cppo was written as a shell script and was never intended to
be a library of functions to be used by anyone else. Single-file Python
modules are often written to be run as standalone programs either to do
what they do from the command line or for testing purposes.
Theoretically you do this if your code provides useful stuff for other
programs to use, and it's hard to argue that cppo does that yet, but it
is intended to do so in the future. Let's start working toward that.
A few of my local copies of cppo have some/most of the code reformatted
in a more "pythonic" coding style. I still use hard tabs for
indentation because even us diehard console-using developers have
editors that can have whatever tabstop we want on a per-file basis, and
we have editorconfig. It's 2017, get with the times, even for a program
made for accessing files for a 1977 computer! ;) No functional changes
here, save for an if statement processing extensions replaces multiple
conditionals with an if x in tuple construct.
