mirror of
https://github.com/fadden/nulib2.git
synced 2024-09-06 16:54:23 +00:00
207 lines
10 KiB
Plaintext
207 lines
10 KiB
Plaintext
![]() |
NufxLib NOTES
|
||
|
Last revised: 2000/01/23
|
||
|
|
||
|
|
||
|
The interface is documented in "nufxlibapi.html", available from the
|
||
|
www.nulib.com web site. This discusses some of the internal design that
|
||
|
may be of interest.
|
||
|
|
||
|
Some familiarity with the NuFX file format is assumed.
|
||
|
|
||
|
|
||
|
Read-Write Data Structures
|
||
|
==========================
|
||
|
|
||
|
For both read-only and read-write files (but not streaming read-only files),
|
||
|
the archive is represented internally as a linked list of Records, each
|
||
|
of which has an array of Threads attached. No attempt is made to
|
||
|
optimize searches by filename, so use of the "replace existing entry when
|
||
|
filenames match" option should be restricted to situations where it is
|
||
|
necessary. Otherwise, O(N^2) behavior can result.
|
||
|
|
||
|
Modifications, such as deletions, changes to filename threads, and
|
||
|
additions of new records, are queued up in a separate list until a NuFlush
|
||
|
call is issued. The list works much the same way as the temporary file:
|
||
|
when the operation completes, the "new" list becomes the "original" list.
|
||
|
If the operation is aborted, the "new" list is scrubbed, and the "original"
|
||
|
list remains unmodified.
|
||
|
|
||
|
Just as it is inefficient to write data to the temp file when it's not
|
||
|
necessary to do so, it is inefficient to allocate a complete copy of the
|
||
|
records from the original list if none are changed. As a result, there are
|
||
|
actually two "new" lists, one with a copy of the original record list, and
|
||
|
one with new additions. The "copy" list starts out uninitialized, and
|
||
|
remains that way until one of the entries from the original list is
|
||
|
modified. When that happens, the entire original list is duplicated, and
|
||
|
the changes are made directly to members of the "copy" list. (This is
|
||
|
important for really large archives, like a by-file archive with the
|
||
|
entire contents of a hard drive, where the record index could be several
|
||
|
megabytes in size.)
|
||
|
|
||
|
It would be more *memory* efficient to simply maintain a list of what
|
||
|
has changed. However, we can't disturb the "original" list in any way or
|
||
|
we lose the ability to roll back quickly if the operation is aborted.
|
||
|
Consequently, we need to create a new list of records that reflects
|
||
|
the state of the new archive, so that when we rename the temp file over
|
||
|
the original, we can simply "rename" the new record list over the original.
|
||
|
Since we're going to need the new list eventually, we might as well create
|
||
|
it as soon as it is needed, and deal with memory allocation failures up
|
||
|
front rather than during the update process. (Some items, such as the
|
||
|
record's file offset in the archive, have to be updated even for records
|
||
|
that aren't themselves changing... which means we potentially need to
|
||
|
modify all existing record structures, so we need a complete copy of the
|
||
|
record list regardless of how little or how much has changed.)
|
||
|
|
||
|
This also ties into the "modify original archive file directly if possible"
|
||
|
option, which avoids the need for creating and renaming a temp file. If
|
||
|
the only changes are updates to pre-sized records (e.g. renaming a file
|
||
|
inside the archive, or updating a comment), or adding new records onto the
|
||
|
end, there is little risk and possibly a huge efficiency gain in just
|
||
|
modifying the archive in place. If none of the operations caused the
|
||
|
"copy" list to be initialized, then clearly there's no need to write to a
|
||
|
temp file. (It's not actually this simple, because updates to pre-sized
|
||
|
threads are annotated in the "copy" list.)
|
||
|
|
||
|
One of the goals was to be able to execute a sequence of operations like:
|
||
|
|
||
|
- open original archive
|
||
|
- read original archive
|
||
|
- modify archive
|
||
|
- flush (success)
|
||
|
- modify archive
|
||
|
- flush (failure, rollback)
|
||
|
- modify archive
|
||
|
- flush (success)
|
||
|
- close archive
|
||
|
|
||
|
The archive is opened at the start and held open across many operations.
|
||
|
There is never a need to re-read the entire archive. We could avoid the
|
||
|
need to allocate two complete Record lists by requiring that the archive be
|
||
|
re-scanned after changes are aborted; if we did that, we could just modify
|
||
|
the original record list in place, and let the changes become "permanent"
|
||
|
after a successful write. In many ways, though, its cleaner to have two
|
||
|
lists.
|
||
|
|
||
|
Archives with several thousand entries should be sufficiently rare, and
|
||
|
virtual memory should be sufficiently plentiful, that this won't be a
|
||
|
problem for anyone. Scanning repeatedly through a 15MB archive stored on a
|
||
|
CD-ROM is likely to be very annoying though, so the design makes every
|
||
|
attempt to avoid repeated scans of the archive. And in any event, this
|
||
|
only applies to archive updates. The memory requirements for simple file
|
||
|
extraction are minimal.
|
||
|
|
||
|
In summary:
|
||
|
|
||
|
"orig" list has original set of records, and is not disturbed until
|
||
|
the changes are committed.
|
||
|
"copy" list is created on first add/update/delete operation, and
|
||
|
initially contains a complete copy of "orig".
|
||
|
"new" list contains all new additions to the archive, including
|
||
|
new additions that replace existing entries (the existing entry
|
||
|
is deleted from "copy" and then added to "new").
|
||
|
|
||
|
|
||
|
Each Record in the list has a "thread modification" list attached to it.
|
||
|
Any changes to the record header or additions to the thread mod list are
|
||
|
made in the "copy" set; the "original" set remains untouched. The thread
|
||
|
mod list can have the following items in it:
|
||
|
|
||
|
- delete thread (NuThreadIdx)
|
||
|
- add thread (type, otherSize, format, +contents)
|
||
|
- update pre-sized thread (NuThreadIdx, +contents)
|
||
|
|
||
|
Contents are specified with a NuDataSource, which allows the application
|
||
|
to indicate that the data is already compressed. This is useful for
|
||
|
copying parts of records between archives without having to expand and
|
||
|
recompress the data.
|
||
|
|
||
|
Some interactions and concepts that are important to understand:
|
||
|
|
||
|
When a file is added, the file type information will be placed in the
|
||
|
"new" Record immediately (subject to some restrictions: adding a data
|
||
|
fork always causes the type info to be updated, adding a rsrc fork only
|
||
|
updates the type info if a data fork is not already present).
|
||
|
|
||
|
Deleting a record results in the Record being removed from the "copy"
|
||
|
list immediately. Future modify operations on that NuRecordIdx will
|
||
|
fail. Future read operations will work just fine until the next
|
||
|
NuFlush is issued, because read operations use the "original" list.
|
||
|
|
||
|
Deleting all threads from a record results in the record being
|
||
|
deleted, but not until the NuFlush call is issued. It is possible to
|
||
|
delete all the existing threads and then add new ones.
|
||
|
|
||
|
It is *not* allowed to delete a modified thread, modify a deleted thread,
|
||
|
or delete a record that has been modified. This limitation was added to
|
||
|
keep the system simple. Note this does not mean you can't delete a data
|
||
|
fork and add a data fork; doing so results in operations on two threads
|
||
|
with different NuThreadIdx values. What you can't do is update the
|
||
|
filename thread and then delete it, or vice-versa. (If anyone can think
|
||
|
of a reason why you'd want to rename a file and then delete it with the
|
||
|
same NuFlush call, I'll figure out a way to support it.)
|
||
|
|
||
|
Updating a filename thread is intercepted, and causes the Record's
|
||
|
filename cache to be updated as well. Adding a filename thread for
|
||
|
records where the filename is stored in the record itself cause the
|
||
|
"in-record" filename to be zeroed. Adding a filename thread to a
|
||
|
record that already has one isn't allowed; nufxlib restricts you to
|
||
|
a single filename thread per record.
|
||
|
|
||
|
Some actions on an archive are allowed but strongly discouraged. For
|
||
|
example, deleting a filename thread but leaving the data threads behind
|
||
|
is a valid thing to do, but leaves most archivers in a state of mild
|
||
|
confusion. Deleting the data threads but leaving the filename thread is
|
||
|
similarly perplexing.
|
||
|
|
||
|
You can't call "update thread" on a thread that doesn't yet exist,
|
||
|
even if an "add thread" call has been made. You can, however, call
|
||
|
"add thread" on a newly created Record.
|
||
|
|
||
|
When a new record is created because of a "create record" call, a filename
|
||
|
thread is created automatically. It is not necessary to explicitly add the
|
||
|
filename.
|
||
|
|
||
|
Failures encountered while committing changes to a record cause all
|
||
|
operations on that record to be rolled back. If, during a NuFlush, a
|
||
|
file add fails, the user is given the option of aborting the entire
|
||
|
operation or skipping the file in question (and perhaps retrying or other
|
||
|
options as well). Aborting the flush causes a complete rollback. If only
|
||
|
the thread mod operation is canceled, then all thread mods for that record
|
||
|
are ignored. The temp file (or archive file) will have its file pointer
|
||
|
reset to the original start of the record, and if the record already
|
||
|
existed in the original archive, the full original record will be copied
|
||
|
over. This may seem drastic, but it helps ensure that you don't end up
|
||
|
with a record in a partially created state.
|
||
|
|
||
|
If a failure occurs during an "update in place", it isn't possible to
|
||
|
roll back all changes. If the failure was due to a bug in NufxLib, it
|
||
|
is possible that the archive could be unrecoverably damaged. NufxLib
|
||
|
tries to identify such situations, and will leave the archive open in
|
||
|
read-only mode after rolling back any new file additions.
|
||
|
|
||
|
|
||
|
Updating Filenames
|
||
|
==================
|
||
|
|
||
|
Updating filenames is a small nightmare, because the filename can be
|
||
|
either in the record header or in a filename thread. It's possible,
|
||
|
but illogical, to have a single record with a filename in the record
|
||
|
header and two or more filenames in threads.
|
||
|
|
||
|
NufxLib will not automatically "fix" broken records, but it will prevent
|
||
|
applications from creating situations that should not exist.
|
||
|
|
||
|
When reading an archive, NufxLib will use the filename from the
|
||
|
first filename thread found. If no filename threads are found, the
|
||
|
filename from the record header will be used.
|
||
|
|
||
|
If you add a filename thread to a record that has a filename in the
|
||
|
record header, the header name will be removed.
|
||
|
|
||
|
If you update a filename thread in a record that has a filename in
|
||
|
the record header, the header name will be left untouched.
|
||
|
|
||
|
Adding a filename thread is only allowed if no filename thread exists,
|
||
|
or all existing filename threads have been deleted.
|
||
|
|