Initial commit

.gitignore

@@ -0,0 +1,5 @@
+TestBed/System Folder/System
+TestBed/System Folder/System.idump
+TestBed/System Folder/System.rdump
+*.tmp
+.DS_Store

README.md

@@ -0,0 +1,53 @@
+Backporting HFS+ to Mac OS pre-8.1
+==================================
+
+This is a project to patch early versions of the "classic" Mac OS to support the [HFS+ ("Mac OS Extended")](https://en.wikipedia.org/wiki/HFS_Plus) filesystem. HFS+ succeeded Apple's previous Hierarchical File System (plain "HFS" or "Mac OS Standard"). It was salvaged from the failed Copland OS project and shipped with Mac OS version 8.1. Its main practical benefit to users was to save space and increase the maximum file count on large volumes, by allowing disk sectors to divided among >64k "allocation blocks".
+
+Mac OS 8.1 and later provided equal support for HFS and HFS+, until the release of Mac OS X 10.6 over ten years later. Floppy disks were never formatted with HFS+, allowing easy data exchange in most cases (the overhead with HFS+ was also too large for a floppy disk). Unfortunately, no HFS+ support was ever provided for Mac OS 8.0 and earlier. On unsupported systems, HFS+ volumes would seem to contain only one document: "Where_have_all_my_files_gone?"
+
+HFS+ was only relatively recently deprecated by macOS 10, and can still easily be read and written on a modern Mac. HFS, by contrast, was relegated to read-only status years ago, and likely will soon be removed altogether.
+
+
+Inspiration
+-----------
+*Disk Tools PPC*
+
+This remarkable Disk Copy image was included with the Mac OS 8.1 install CD, to allow a user to make her own minimal bootable rescue disk. It can read, write, format and repair HFS+ volumes.
+
+Lacking the Appearace Manager and Finder 8, it looks and feels more like System 7. On closer inspection, I found that it actually is System 7! The contents of the resource fork reveal that it is based on System 7.5.5. It must then be possible for pre-8.1 Systems to support HFS+ without a complete rewrite. This is our goal.
+
+"Disk Tools 1", on the same CD, contains a similar minimal System for 68k Macs only. This restriction is enforced by the "Disk Tools 1/2 Enabler" file on each Disk Tools floppy. The file is misnamed: it comprises minimal code to *disable* boot on opposite-architecture Macs, by complaining in a Deep Shit error box, "This Disk Tools disk will not work with this computer. Use the disk labelled Disk Tools 2/1".
+
+"Disk Tools 1" and "Disk Tools PPC" are versioned "8.0DT" and "8.1DT" respectively, as if to avoid confusion about which System version will read HFS+ volumes. 
+
+
+Tooling
+-------
+These the prerequisites to work on this project:
+
+- Pretty much any Unix-like system
+- Python 3
+- My `macresources` and `machfs` Python packages: `python3 -m pip install macresources machfs`
+- Mac OS emulators: [SheepShaver](https://www.emaculation.com/doku.php/sheepshaver) (PowerPC) and [Basilisk II](https://www.emaculation.com/doku.php/basilisk_ii)
+
+The Python packages are used to manipulate Mac OS [resource forks](https://en.wikipedia.org/wiki/Resource_fork), which contain most of the code in the Mac OS System file, and to create bootable disk images to test our work.
+
+This repository contains an anthology of Mac OS System files under `SampleSystems`, including the Disk Tools discussed above. Systems have been dumped into the Rez-like `.rdump` format preferred by `macresources`. All resources have been decompressed (an essay in itself) and placed in a canonical order for easy browsing/diffing. `resreport.txt` shows the evolution of the resources across these System versions, and clearly illustrates the similarity between System 7.5.5 and Disk Tools PPC.
+
+Binary-level comparisions between resources are possible by combining `rfx` (installed with `macresources`) with C.J. Madsen's `vbindiff`. E.g.: `rfx vbindiff SampleSystems/7.6.1//boot/3 SampleSystems/8.1.0//boot/3`
+
+`make.py`, heavily commented, combines resources from various System versions to make a a series of patched Systems under `TestImages.tmp`.
+
+This project has, so far, not required any deep exploration of the HFS or HFS+ on-disk formats.
+
+
+How HFS+ is implemented on Mac OS
+---------------------------------
+
+Essentially, HFS+ support is provided by the 68k 'ptch' -20217 resource. This patch uses the "File System Manager" hooks provided by the Mac Plus and later ROMs. Mac OS 8.1 executes the patch fairly early in the boot process from the 'boot' 3 resource. (At some point before Mac OS 9.2.2, this was moved even earlier into the 'boot' 2 resource.) Additional support comes from:
+
+- 'ptch' 41, the Disk Cache
+- 'boot' 22460
+- 'PACK' 2, the Disk Initialization code, and related resources
+
+The Text Encoding Converter extension is required on at least some systems, I suppose because HFS+ does encode Unicode filenames with UTF-16.

SampleSystems/7.1.0.idump

@@ -0,0 +1 @@
+????????

SampleSystems/7.1.1.idump

@@ -0,0 +1 @@
+zsysMACS

SampleSystems/7.5.0.idump

@@ -0,0 +1 @@
+zsysMACS

SampleSystems/7.5.3.idump

@@ -0,0 +1 @@
+zsysMACS

SampleSystems/7.5.5.idump

@@ -0,0 +1 @@
+zsysMACS

SampleSystems/7.6.0.idump

@@ -0,0 +1 @@
+zsysMACS

SampleSystems/7.6.1.idump

@@ -0,0 +1 @@
+zsysMACS

SampleSystems/8.0.0.idump

@@ -0,0 +1 @@
+zsysMACS

SampleSystems/8.1.0.idump

@@ -0,0 +1 @@
+zsysMACS

SampleSystems/9.2.2.idump

@@ -0,0 +1 @@
+zsysMACS

SampleSystems/DT_8.1_1.idump

@@ -0,0 +1 @@
+zsysMACS

SampleSystems/DT_8.1_PPC.idump

@@ -0,0 +1 @@
+zsysMACS

TestBed/System Folder/Extensions/Text Encoding Converter

@@ -0,0 +1 @@
+

TestBed/System Folder/Extensions/Text Encoding Converter.idump

@@ -0,0 +1 @@
+shlbencv

TestBed/System Folder/Finder.idump

@@ -0,0 +1 @@
+FNDRMACS

make.py

@@ -0,0 +1,179 @@
+#!/usr/bin/env python3
+
+from functools import lru_cache
+from os import path
+from shutil import copyfile
+import machfs
+import machfs
+import macresources
+import os
+import re
+import shutil
+
+
+@lru_cache()
+def get_resource_fork(version):
+    with open(path.join(SampleSystems, version) + '.rdump', 'rb') as f:
+        return list(macresources.parse_rez_code(f.read()))
+
+def get_resource(version, rtype, rid):
+    if isinstance(rtype, str): rtype = rtype.encode('mac_roman')
+
+    for resource in get_resource_fork(version):
+        if (resource.type, resource.id) == (rtype, rid):
+            return resource
+
+    raise ValueError('not found: %r %r %r' % (version, rtype, rid))
+
+# Some resource types can "own" other resources, which should be moved with them
+def does_x_own_y(xtype, xid, yid):
+    types = [b'DRVR', b'WDEF', b'MDEF', b'CDEF', b'PDEF', b'PACK']
+    if yid & (1 << 15) and yid & (1 << 14):
+        if xtype in types and types.index(xtype) == (yid >> 11) & 7:
+            if (yid >> 5) & 0x3F == xid:
+                return True
+    return False
+
+
+
+parent = path.dirname(__file__)
+SampleSystems = path.join(parent, 'SampleSystems')
+TestBed = path.join(parent, 'TestBed')
+
+TestImages = path.join(parent, 'TestImages.tmp')
+os.makedirs(TestImages, exist_ok=True)
+
+
+with open(path.join(TestImages, 'Test-Blank.dsk'), 'wb') as f:
+    for i in range(50):
+        f.write(bytes(1024 * 1024))
+
+
+base_versions = sorted(os.listdir(SampleSystems))
+base_versions = [v for v in base_versions if re.match(r'^\d.\d.\d$', v)]
+
+base_versions = [v for v in base_versions if int(v.replace('.', '')) < 810]
+
+
+for base_version in base_versions:
+    print('\n=== Patching System ' + base_version + ' ===')
+
+    src_system = path.join(SampleSystems, base_version)
+
+    dest_dir = path.join(TestImages, 'Test-' + base_version)
+    dest_disk = dest_dir + '.dsk'
+    dest_system = path.join(dest_dir, 'System Folder', 'System')
+
+    try:
+        shutil.rmtree(dest_dir)
+    except FileNotFoundError:
+        pass
+    shutil.copytree(TestBed, dest_dir)
+
+    # Copy the sample system, except the data fork
+    for suffix in ['', '.idump']:
+        copyfile(src_system + suffix, dest_system + suffix)
+
+    # Get its resource fork, and copy it to another list so we don't ruin our cache
+    the_resources = get_resource_fork(base_version)
+
+
+    def place_resource(resource):
+        the_len = len(the_resources)
+        the_resources[:] = [r for r in the_resources if (r.type, r.id) != (resource.type, resource.id)]
+        # if len(the_resources) < the_len:
+        #     print(' replaced %s %s' % (repr(resource.type)[1:], resource.id))
+        the_resources.append(resource)
+
+
+    def copy_resource_and_subresources(version, rtype, rid):
+        if isinstance(rtype, str): rtype = rtype.encode('mac_roman')
+
+        for resource in get_resource_fork(version):
+            if (resource.type, resource.id) == (rtype, rid):
+                print('  Copying %s\'s %s %s' % (version, repr(resource.type)[1:], resource.id))
+                place_resource(resource)
+                break
+        else:
+            raise ValueError('not found: %r %r %r' % (version, rtype, rid))
+
+        n_extra = 0
+        for resource in get_resource_fork(version):
+            if does_x_own_y(rtype, rid, resource.id):
+                # print('+ sub-resource %s %s' % (repr(resource.type)[1:], resource.id))
+                place_resource(resource)
+                n_extra += 1
+        if n_extra: print('    + %d owned resources' % n_extra)
+
+
+
+
+
+    # Use ALL of these:
+    print('These are the main HFS+ resources:')
+    copy_resource_and_subresources('8.1.0', 'ptch', -20217)     # HFS+ patch
+    copy_resource_and_subresources('8.1.0', 'boot', 22460)      # Seems to be a gatekeeper/bootloader??
+
+    print('Disk Cache patch:')
+    copy_resource_and_subresources('8.1.0', 'ptch', 41)         # Disk Cache patch
+
+    print('Disk Init pack and things it uses:')
+    copy_resource_and_subresources('8.1.0', 'PACK', 2)          # Disk Init (brings in several "owned" resources)
+    copy_resource_and_subresources('8.1.0', 'p2u#', 0)          # Referenced by ptch, "Text Encodings"...
+    copy_resource_and_subresources('8.1.0', 'STR#', -20574)     # "also known as" for various formats
+    copy_resource_and_subresources('8.1.0', 'STR#', -20573)     # "Where_have_all_my_files_gone?"
+    copy_resource_and_subresources('8.1.0', 'STR#', -20483)     # "There is a problem with the disk"
+    copy_resource_and_subresources('8.1.0', 'TEXT', -20574)     # "This is the localized version of the HFSPlus Wrapper Read Me."
+    copy_resource_and_subresources('8.1.0', 'TEXT', -20573)     # Wrapper readme contents
+
+
+    # Use ONE of these to chain load 'ptch' -20217 (not required on 7.6.0 and later):
+    MOVEW_B107_D0 = b'\x30\x3C\xB1\x07'
+    if MOVEW_B107_D0 not in get_resource(base_version, 'boot', 2) and MOVEW_B107_D0 not in get_resource(base_version, 'boot', 3):
+        # Mac OS 9 actually loads ptch -20217 in boot 2, which is more elegant, because unlike boot 3,
+        # boot 2 is guaranteed not to be run from a separate gibbly
+
+        print('Pre-7.6 needs some help to load ptch -20217:')
+        copy_resource_and_subresources('9.2.2', 'boot', 2)
+        # copy_resource_and_subresources('8.1.0', 'boot', 3)
+        # copy_resource_and_subresources('DT_8.1_PPC', 'boot', 3)
+
+
+    print('Rudimentary 68k support on Basilisk II\'s Quadra 900:')
+
+    # These 3 resources have been identified by bisecting the different resources between 8.0 and 8.1
+    # Use all of these for 68k, currently only Basilisk II's Quadra 900 (only works on 7.6.0/7.6.1/8.0):
+    # (Still unclear what these do -- next step is to reverse-engineer the gusd/gtbl/gpch mechanism)
+
+    # HFS+ not loaded if it is not changed (causes a crash on its own -- requires BOTH the below resources)
+    copy_resource_and_subresources('8.1.0', 'gtbl', 6) # diff contents
+
+    # Crash if it is not changed: illegal instruction (harmless on its own)
+    copy_resource_and_subresources('8.1.0', 'gpch', 750) # diff contents
+
+    # Error Type 41 if not present (harmless on its own)
+    copy_resource_and_subresources('8.1.0', 'ptch', 42) # diff contents
+
+
+
+    # Lastly, don't forget the Text Encoding Converter extension is essential to make this work
+
+
+
+
+
+    the_resources.sort(key=lambda r: (r.type.decode('mac_roman'), r.id))
+    with open(dest_system + '.rdump', 'wb') as f:
+        f.write(macresources.make_rez_code(the_resources, ascii_clean=True))
+
+
+
+
+    vol = machfs.Volume()
+    vol.name = 'Test-' + base_version
+    vol.read_folder(TestBed, date=0xC0000000)
+    vol = vol.write(10 * 1024 * 1024)
+
+
+    with open(dest_disk, 'wb') as f:
+        f.write(vol)

resreport.bash

@@ -0,0 +1,20 @@
+#!/bin/bash
+
+# cd to own containing directory
+cd "${0%/*}"
+cd SampleSystems
+
+~/Documents/mac/ToolboxToolbox/fsnek -r \
+    7.1.0.rdump \
+    7.1.1.rdump \
+    7.5.0.rdump \
+    7.5.3.rdump \
+    DT_8.1_1.rdump \
+    7.5.5.rdump \
+    DT_8.1_PPC.rdump \
+    7.6.0.rdump \
+    7.6.1.rdump \
+    8.0.0.rdump \
+    8.1.0.rdump \
+    9.2.2.rdump \
+    > ../resreport.txt