commit eceb8f7038b289fb0a40eb8098e9c8837f2a35cc Author: Andy McFadden Date: Sun Dec 21 10:37:29 2014 -0800 Transfer nulib.com web site to github The full site, in all its FrontPage-generated glory. diff --git a/README.md b/README.md new file mode 100644 index 0000000..9612608 --- /dev/null +++ b/README.md @@ -0,0 +1,2 @@ +The gh-pages branch contains the NuLib2 web site. This is accessible as +https://fadden.github.io/nulib2/, or http://nulib.com/. diff --git a/_borders/_vti_cnf/top.htm b/_borders/_vti_cnf/top.htm new file mode 100644 index 0000000..4d7b697 --- /dev/null +++ b/_borders/_vti_cnf/top.htm @@ -0,0 +1,23 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|06 Feb 2000 09:07:24 -0000 +vti_extenderversion:SR|4.0.2.7802 +vti_nexttolasttimemodified:TR|06 Feb 2000 09:06:47 -0000 +vti_author:SR|fadden +vti_modifiedby:SR|fadden +vti_timecreated:TR|23 Dec 1999 21:42:47 -0000 +vti_shadowfiles:VX| +vti_filesize:IR|825 +vti_title:SR|Shared Top Border +vti_metatags:VR|HTTP-EQUIV=Content-Type text/html;\\ charset=windows-1252 GENERATOR Microsoft\\ FrontPage\\ 4.0 ProgId FrontPage.Editor.Document +vti_progid:SR|FrontPage.Editor.Document +vti_generator:SR|Microsoft FrontPage 4.0 +vti_backlinkinfo:VX| +vti_cacheddtm:TX|06 Feb 2000 08:07:24 -0000 +vti_cachedlinkinfo:VX| +vti_cachedsvcrellinks:VX| +vti_cachedtitle:SR|Shared Top Border +vti_cachedbodystyle:SR| +vti_cachedhasbots:BR|true +vti_cachedhastheme:BR|false +vti_cachedhasborder:BR|false +vti_botnavbits:SW|SHUB diff --git a/_borders/top.htm b/_borders/top.htm new file mode 100644 index 0000000..fcc70f2 --- /dev/null +++ b/_borders/top.htm @@ -0,0 +1,21 @@ + + + + +Shared Top Border + + + + + + + +


+

+
+ + + + diff --git a/_vti_cnf/bugs.htm b/_vti_cnf/bugs.htm new file mode 100644 index 0000000..7f72283 --- /dev/null +++ b/_vti_cnf/bugs.htm @@ -0,0 +1,24 @@ +vti_encoding:SR|utf8-nl +vti_author:SR|fadden +vti_timecreated:TR|15 Jan 2000 06:19:33 -0000 +vti_timelastmodified:TR|31 Mar 2004 17:37:51 -0000 +vti_shadowfiles:VX| +vti_filesize:IR|3954 +vti_title:SR|Bugs & Features +vti_metatags:VR|HTTP-EQUIV=Content-Language en-us HTTP-EQUIV=Content-Type text/html;\\ charset=windows-1252 GENERATOR Microsoft\\ FrontPage\\ 4.0 ProgId FrontPage.Editor.Document +vti_progid:SR|FrontPage.Editor.Document +vti_generator:SR|Microsoft FrontPage 4.0 +vti_extenderversion:SR|4.0.2.8912 +vti_backlinkinfo:VX|index.htm bugs.htm +vti_nexttolasttimemodified:TR|19 Mar 2003 17:33:03 -0000 +vti_modifiedby:SR|fadden +vti_cacheddtm:TX|31 Mar 2004 16:37:52 -0000 +vti_cachedlinkinfo:VX|K|bugs.htm H|mailto:fadden@fadden.com +vti_cachedsvcrellinks:VX|FKUS|bugs.htm NHUS|mailto:fadden@fadden.com +vti_cachedtitle:SR|Bugs & Features +vti_cachedbodystyle:SR| +vti_cachedhasbots:BR|true +vti_cachedhastheme:BR|false +vti_cachedhasborder:BR|true +vti_botnavbits:SW|SHUB +vti_borderaggregate:SR|default diff --git a/_vti_cnf/index.htm b/_vti_cnf/index.htm new file mode 100644 index 0000000..c532b7a --- /dev/null +++ b/_vti_cnf/index.htm @@ -0,0 +1,22 @@ +vti_encoding:SR|utf8-nl +vti_author:SR|fadden +vti_timecreated:TR|22 Dec 1999 22:47:10 -0000 +vti_timelastmodified:TR|19 Feb 2007 23:13:28 -0000 +vti_filesize:IR|9646 +vti_title:SR|NuLib Home Page +vti_metatags:VR|HTTP-EQUIV=Content-Type text/html;\\ charset=windows-1252 HTTP-EQUIV=Content-Language en-us GENERATOR Microsoft\\ FrontPage\\ 4.0 ProgId FrontPage.Editor.Document description Home\\ page\\ for\\ NuLib,\\ NuLib2,\\ NufxLib,\\ ShrinkIt,\\ and\\ NuFX\\ (SHK)\\ archives keywords nulib,\\ nulib2,\\ nufxlib,\\ shk,\\ sdk,\\ bxy,\\ bse,\\ shrinkit,\\ nufx,\\ apple,\\ apple2,\\ emulator +vti_progid:SR|FrontPage.Editor.Document +vti_generator:SR|Microsoft FrontPage 4.0 +vti_extenderversion:SR|4.0.2.8912 +vti_backlinkinfo:VX| +vti_nexttolasttimemodified:TR|19 Sep 2005 05:42:45 -0000 +vti_shadowfiles:VX| +vti_modifiedby:SR|fadden +vti_cacheddtm:TX|19 Feb 2007 23:13:28 -0000 +vti_cachedlinkinfo:VX|H|downloads/index.htm H|nulib2-manual.htm H|nufxlibapi.htm H|bugs.htm H|library/index.htm H|nulib2-manual.htm H|nufxlibapi.htm H|http://www.faddensoft.com/ciderpress/ H|http://www.faddensoft.com/ H|http://www.fadden.com/ H|http://sourceforge.net S|http://sourceforge.net/sflogo.php +vti_cachedsvcrellinks:VX|FHUS|downloads/index.htm FHUS|nulib2-manual.htm FHUS|nufxlibapi.htm FHUS|bugs.htm FHUS|library/index.htm FHUS|nulib2-manual.htm FHUS|nufxlibapi.htm NHHS|http://www.faddensoft.com/ciderpress/ NHHS|http://www.faddensoft.com/ NHHS|http://www.fadden.com/ NHHS|http://sourceforge.net NSHS|http://sourceforge.net/sflogo.php +vti_cachedtitle:SR|NuLib Home Page +vti_cachedbodystyle:SR| +vti_cachedhasbots:BR|false +vti_cachedhastheme:BR|false +vti_cachedhasborder:BR|false diff --git a/_vti_cnf/nufxlibapi.htm b/_vti_cnf/nufxlibapi.htm new file mode 100644 index 0000000..63fccd0 --- /dev/null +++ b/_vti_cnf/nufxlibapi.htm @@ -0,0 +1,24 @@ +vti_encoding:SR|utf8-nl +vti_author:SR|fadden +vti_timecreated:TR|24 Dec 1999 08:11:12 -0000 +vti_timelastmodified:TR|19 Feb 2007 23:15:03 -0000 +vti_shadowfiles:VX| +vti_filesize:IR|110697 +vti_title:SR|NufxLib API +vti_metatags:VR|HTTP-EQUIV=Content-Language en-us HTTP-EQUIV=Content-Type text/html;\\ charset=windows-1252 GENERATOR Microsoft\\ FrontPage\\ 4.0 ProgId FrontPage.Editor.Document +vti_progid:SR|FrontPage.Editor.Document +vti_generator:SR|Microsoft FrontPage 4.0 +vti_extenderversion:SR|4.0.2.8912 +vti_backlinkinfo:VX|nufxlibapi.htm index.htm +vti_nexttolasttimemodified:TR|19 Feb 2007 22:35:33 -0000 +vti_modifiedby:SR|fadden +vti_cacheddtm:TX|19 Feb 2007 23:15:03 -0000 +vti_cachedlinkinfo:VX|K|nufxlibapi.htm K|nufxlibapi.htm K|nufxlibapi.htm K|nufxlibapi.htm K|nufxlibapi.htm K|nufxlibapi.htm K|nufxlibapi.htm K|nufxlibapi.htm K|nufxlibapi.htm K|nufxlibapi.htm K|nufxlibapi.htm K|nufxlibapi.htm K|nufxlibapi.htm K|nufxlibapi.htm K|nufxlibapi.htm K|nufxlibapi.htm K|nufxlibapi.htm K|nufxlibapi.htm K|nufxlibapi.htm K|nufxlibapi.htm K|nufxlibapi.htm K|nufxlibapi.htm H|library/FTN.e08002.htm H|library/nufx-addendum.htm H|library/FTN.e08002.htm K|nufxlibapi.htm H|library/nufx-addendum.htm K|nufxlibapi.htm K|nufxlibapi.htm K|nufxlibapi.htm K|nufxlibapi.htm K|nufxlibapi.htm K|nufxlibapi.htm K|nufxlibapi.htm H|http://www.fadden.com/ H|http://www.nulib.com/ +vti_cachedsvcrellinks:VX|FKUS|nufxlibapi.htm FKUS|nufxlibapi.htm FKUS|nufxlibapi.htm FKUS|nufxlibapi.htm FKUS|nufxlibapi.htm FKUS|nufxlibapi.htm FKUS|nufxlibapi.htm FKUS|nufxlibapi.htm FKUS|nufxlibapi.htm FKUS|nufxlibapi.htm FKUS|nufxlibapi.htm FKUS|nufxlibapi.htm FKUS|nufxlibapi.htm FKUS|nufxlibapi.htm FKUS|nufxlibapi.htm FKUS|nufxlibapi.htm FKUS|nufxlibapi.htm FKUS|nufxlibapi.htm FKUS|nufxlibapi.htm FKUS|nufxlibapi.htm FKUS|nufxlibapi.htm FKUS|nufxlibapi.htm FHUS|library/FTN.e08002.htm FHUS|library/nufx-addendum.htm FHUS|library/FTN.e08002.htm FKUS|nufxlibapi.htm FHUS|library/nufx-addendum.htm FKUS|nufxlibapi.htm FKUS|nufxlibapi.htm FKUS|nufxlibapi.htm FKUS|nufxlibapi.htm FKUS|nufxlibapi.htm FKUS|nufxlibapi.htm FKUS|nufxlibapi.htm NHHS|http://www.fadden.com/ NHHS|http://www.nulib.com/ +vti_cachedtitle:SR|NufxLib API +vti_cachedbodystyle:SR| +vti_cachedhasbots:BR|true +vti_cachedhastheme:BR|false +vti_cachedhasborder:BR|true +vti_botnavbits:SW|SHUB +vti_borderaggregate:SR|default diff --git a/_vti_cnf/nulib2-manual.htm b/_vti_cnf/nulib2-manual.htm new file mode 100644 index 0000000..dca2300 --- /dev/null +++ b/_vti_cnf/nulib2-manual.htm @@ -0,0 +1,24 @@ +vti_encoding:SR|utf8-nl +vti_author:SR|fadden +vti_timecreated:TR|16 Jan 2000 22:42:13 -0000 +vti_timelastmodified:TR|19 Feb 2006 02:02:53 -0000 +vti_shadowfiles:VX| +vti_filesize:IR|33618 +vti_title:SR|NuLib2 Manual +vti_metatags:VR|HTTP-EQUIV=Content-Type text/html;\\ charset=windows-1252 GENERATOR Microsoft\\ FrontPage\\ 4.0 ProgId FrontPage.Editor.Document +vti_progid:SR|FrontPage.Editor.Document +vti_generator:SR|Microsoft FrontPage 4.0 +vti_extenderversion:SR|4.0.2.8912 +vti_backlinkinfo:VX|nulib2-manual.htm index.htm +vti_nexttolasttimemodified:TR|19 Feb 2006 02:02:22 -0000 +vti_modifiedby:SR|fadden +vti_cacheddtm:TX|19 Feb 2006 02:02:53 -0000 +vti_cachedlinkinfo:VX|K|nulib2-manual.htm K|nulib2-manual.htm K|nulib2-manual.htm K|nulib2-manual.htm K|nulib2-manual.htm K|nulib2-manual.htm K|nulib2-manual.htm K|nulib2-manual.htm K|nulib2-manual.htm K|nulib2-manual.htm K|nulib2-manual.htm H|http://www.nulib.com/ H|http://www.zlib.org/ K|nulib2-manual.htm H|library/nulib2-preserve.htm H|library/nulib2-preserve.htm K|nulib2-manual.htm H|http://www.fadden.com/ H|http://www.nulib.com/ +vti_cachedsvcrellinks:VX|FKUS|nulib2-manual.htm FKUS|nulib2-manual.htm FKUS|nulib2-manual.htm FKUS|nulib2-manual.htm FKUS|nulib2-manual.htm FKUS|nulib2-manual.htm FKUS|nulib2-manual.htm FKUS|nulib2-manual.htm FKUS|nulib2-manual.htm FKUS|nulib2-manual.htm FKUS|nulib2-manual.htm NHHS|http://www.nulib.com/ NHHS|http://www.zlib.org/ FKUS|nulib2-manual.htm FHUS|library/nulib2-preserve.htm FHUS|library/nulib2-preserve.htm FKUS|nulib2-manual.htm NHHS|http://www.fadden.com/ NHHS|http://www.nulib.com/ +vti_cachedtitle:SR|NuLib2 Manual +vti_cachedbodystyle:SR| +vti_cachedhasbots:BR|true +vti_cachedhastheme:BR|false +vti_cachedhasborder:BR|true +vti_botnavbits:SW|SHUB +vti_borderaggregate:SR|default diff --git a/_vti_pvt/_vti_cnf/_x_todo.htm b/_vti_pvt/_vti_cnf/_x_todo.htm new file mode 100644 index 0000000..4ee1687 --- /dev/null +++ b/_vti_pvt/_vti_cnf/_x_todo.htm @@ -0,0 +1,18 @@ +vti_encoding:SR|utf8-nl +vti_author:SR|fadden +vti_modifiedby:SR|fadden +vti_timecreated:TR|31 Jan 2000 05:53:17 -0000 +vti_timelastmodified:TR|31 Jan 2000 05:53:17 -0000 +vti_filesize:IR|578 +vti_title:SR|Active To Do List +vti_metatags:VR|HTTP-EQUIV=Content-Type text/html;\\ charset=utf-8 +vti_extenderversion:SR|4.0.2.5526 +vti_backlinkinfo:VX| +vti_cacheddtm:TX|31 Jan 2000 04:53:18 -0000 +vti_cachedlinkinfo:VX| +vti_cachedsvcrellinks:VX| +vti_cachedtitle:SR|Active To Do List +vti_cachedbodystyle:SR| +vti_cachedhasbots:BR|false +vti_cachedhastheme:BR|false +vti_cachedhasborder:BR|false diff --git a/_vti_pvt/_vti_cnf/_x_todoh.htm b/_vti_pvt/_vti_cnf/_x_todoh.htm new file mode 100644 index 0000000..b09b930 --- /dev/null +++ b/_vti_pvt/_vti_cnf/_x_todoh.htm @@ -0,0 +1,18 @@ +vti_encoding:SR|utf8-nl +vti_author:SR|fadden +vti_modifiedby:SR|fadden +vti_timecreated:TR|31 Jan 2000 05:53:17 -0000 +vti_timelastmodified:TR|31 Jan 2000 05:53:17 -0000 +vti_cacheddtm:TX|31 Jan 2000 05:53:18 -0000 +vti_filesize:IR|580 +vti_cachedlinkinfo:VX| +vti_cachedsvcrellinks:VX| +vti_cachedtitle:SR|To Do List History +vti_title:SR|To Do List History +vti_cachedbodystyle:SR| +vti_cachedhasbots:BR|false +vti_cachedhastheme:BR|false +vti_cachedhasborder:BR|false +vti_metatags:VR|HTTP-EQUIV=Content-Type text/html;\\ charset=utf-8 +vti_extenderversion:SR|4.0.2.2717 +vti_backlinkinfo:VX| diff --git a/_vti_pvt/_x_todo.htm b/_vti_pvt/_x_todo.htm new file mode 100644 index 0000000..2b925a0 --- /dev/null +++ b/_vti_pvt/_x_todo.htm @@ -0,0 +1,23 @@ + + +Active To Do List + +

Active To Do List

+ + + + + + + + + + + + + + + + +
NumberTaskPriAuthorCreated By ToolCreated OnModified OnCompletedModified ByAssigned ToLink ToMagicDescription
+ diff --git a/_vti_pvt/_x_todoh.htm b/_vti_pvt/_x_todoh.htm new file mode 100644 index 0000000..ae0ef3b --- /dev/null +++ b/_vti_pvt/_x_todoh.htm @@ -0,0 +1,23 @@ + + +To Do List History + +

To Do List History

+ + + + + + + + + + + + + + + + +
NumberTaskPriAuthorCreated By ToolCreated OnModified OnCompletedModified ByAssigned ToLink ToMagicDescription
+ diff --git a/_vti_pvt/botinfs.cnf b/_vti_pvt/botinfs.cnf new file mode 100644 index 0000000..2d0c750 --- /dev/null +++ b/_vti_pvt/botinfs.cnf @@ -0,0 +1,2 @@ +vti_encoding:SR|utf8-nl +C\:\\Program Files\\Common Files\\Microsoft Shared\\Web Server Extensions\\40\\bots\\vinavbar\\vinavbar.inf:VW|vinavbar diff --git a/_vti_pvt/bots.cnf b/_vti_pvt/bots.cnf new file mode 100644 index 0000000..3af039b --- /dev/null +++ b/_vti_pvt/bots.cnf @@ -0,0 +1,2 @@ +vti_encoding:SR|utf8-nl +vinavbar:VW|C:\\\\Program\\ Files\\\\Common\\ Files\\\\Microsoft\\ Shared\\\\Web\\ Server\\ Extensions\\\\40\\\\bots\\\\vinavbar\\\\vinavbar.inf vinavbar E I info N C:\\\\Program\\ Files\\\\Common\\ Files\\\\Microsoft\\ Shared\\\\Web\\ Server\\ Extensions\\\\40\\\\bots\\\\vinavbar\\\\fp4Avnb.dll diff --git a/_vti_pvt/deptodoc.btr b/_vti_pvt/deptodoc.btr new file mode 100644 index 0000000..de677cc Binary files /dev/null and b/_vti_pvt/deptodoc.btr differ diff --git a/_vti_pvt/doctodep.btr b/_vti_pvt/doctodep.btr new file mode 100644 index 0000000..c7b5f53 Binary files /dev/null and b/_vti_pvt/doctodep.btr differ diff --git a/_vti_pvt/linkinfo.cnf b/_vti_pvt/linkinfo.cnf new file mode 100644 index 0000000..e6b3e74 --- /dev/null +++ b/_vti_pvt/linkinfo.cnf @@ -0,0 +1,14 @@ +vti_encoding:SR|utf8-nl +http\://www.zlib.org/:nulib2-manual.htm library/nufx-addendum.htm +http\://sources.redhat.com/bzip2/:library/nufx-addendum.htm +downloads/nulibdist.tar.gz:downloads/index.htm +http\://www.faddensoft.com/:index.htm +http\://www.fadden.com/:nulib2-manual.htm library/nulib2-preserve.htm nufxlibapi.htm index.htm library/nufx-addendum.htm +http\://www.faddensoft.com/ciderpress/:downloads/index.htm index.htm +mailto\:fadden@fadden.com:bugs.htm +http\://www.nulib.com/:nulib2-manual.htm library/nulib2-preserve.htm nufxlibapi.htm library/nufx-addendum.htm +http\://www.fadden.com/dl-apple2/:library/index.htm +http\://sourceforge.net/projects/nulib2/:downloads/index.htm +http\://sourceforge.net/sflogo.php:downloads/index.htm index.htm +http\://sourceforge.net:index.htm +downloads/nulib2_win32.zip:downloads/index.htm diff --git a/_vti_pvt/service.cnf b/_vti_pvt/service.cnf new file mode 100644 index 0000000..49d2c9a --- /dev/null +++ b/_vti_pvt/service.cnf @@ -0,0 +1,28 @@ +vti_encoding:SR|utf8-nl +vti_casesensitiveurls:IX|0 +vti_textextensions:SX|.txt. +vti_featurelist:VX|vti_ACAll vti_ServiceMarkUrlDirBrowse vti_ServiceMarkUrlDirExec vti_ServiceMarkUrlDirScript vti_ServerEmailTransport vti_ServerIndexServer vti_ServerASP +vti_httpdversion:SX|FrontPage DBW +vti_ignorekeyboard:IR|0 +vti_navbuttonuplabel:SR|Up +vti_dependenciesood:IR|1 +vti_webservertype:SR|diskweb +vti_categories:VR|Travel Expense\\ Report Business Competition Goals/Objectives Ideas Miscellaneous Waiting VIP In\\ Process Planning Schedule +vti_navbuttonnextlabel:SR|Next +vti_approvallevels:VR|Content\\ Review Legal\\ Review Code\\ Review Manager\\ Review +vti_timecreated:TR|22 Dec 1999 22:47:08 -0000 +vti_extenderversion:SR|4.0.2.2717 +vti_navbuttonprevlabel:SR|Back +vti_borderdefault:SR|t +vti_longfilenames:IX|1 +vti_welcomenames:VX|index.htm index.html default.htm default.html welcome.htm welcome.html home.htm home.html +vti_insecureserverurl:SR|file:// +vti_disableautoimgsizeexts:SX|.asp +vti_oldestcompatibleversion:SR|2.0.0.0 +vti_restartmanual:IX|0 +vti_defaultcharset:SR|windows-1252 +vti_defaultlanguage:SR|en +vti_publishmetainfokeys:VR|vti_assignedto vti_approvallevel vti_categories vti_description +vti_autorecalc:IX|1 +vti_htmlextensions:SX|.htm.html.shtml.shtm.stm.htt.htx.asp.alx.asa. +vti_navbuttonhomelabel:SR|Home diff --git a/_vti_pvt/service.lck b/_vti_pvt/service.lck new file mode 100644 index 0000000..e69de29 diff --git a/_vti_pvt/services.cnf b/_vti_pvt/services.cnf new file mode 100644 index 0000000..d83d55f --- /dev/null +++ b/_vti_pvt/services.cnf @@ -0,0 +1 @@ +/ diff --git a/_vti_pvt/structure.cnf b/_vti_pvt/structure.cnf new file mode 100644 index 0000000..7016449 --- /dev/null +++ b/_vti_pvt/structure.cnf @@ -0,0 +1,11 @@ +3.0.0.507 +1008 +1000,index.htm,0,NuLib Home Page,0,945985133,1 +vti_globalpage:BW|true +1001,downloads/index.htm,0,NuLib Downloads,1000,947902101,0 +1003,library/index.htm,0,NuLib Library,1000,949192642,0 +1007,nulib2-manual.htm,0,NuLib2 Manual,1000,948062548,0 +1002,nufxlibapi.htm,0,NufxLib API,1000,946023114,0 +1004,bugs.htm,0,Bugs & Features,1000,950147319,0 +1005,library/nufx-addendum.htm,0,NuFX Addendum,1003,948051434,0 +1006,library/nulib2-preserve.htm,0,ProDOS Attribute Preservation,1003,948076779,0 \ No newline at end of file diff --git a/bugs.htm b/bugs.htm new file mode 100644 index 0000000..5dd6ed8 --- /dev/null +++ b/bugs.htm @@ -0,0 +1,80 @@ + + + + + + + +Bugs & Features + + + +
+ +

Bugs & Features
+[ Home ] [ NuLib Downloads ] [ NuLib Library ] [ NuLib2 Manual ] [ NufxLib API ] [ Bugs & Features ]

+
+ +
+ +

 

+

Skip down to the bottom for the list of suggested +features.

+

NuLib2 and NufxLib will always have the same version number for +the time being, because they're being distributed together.  It's less +confusing that way.

+

 

+

Known Bugs - NufxLib v2.0

+

None so far.

+

 

+

Known Bugs - NuLib2 v2.0

+

None so far.

+

 

+

Reporting New Bugs

+

Some good things to include in a bug report:

+
    +
  • +

    What version of NuLib2 and NufxLib are you using?

    • +
  • +

    What compile options were used (NuLib2 shows this at the top + of the usage screen; just type "nulib2")?

    • +
  • +

    What operating system are you running?  What version?

    • +
  • +

    Did you build it yourself or get a binary from a web + site?  Did you make any changes?  What compiler did you use?

    • +
  • +

    What exactly do you have to do to make the bug occur?  + (If it's a NuLib2 problem, list the command, and preferably where I can find + a copy of the archive or the files involved.  If it's a NufxLib + problem, all of the above, plus a sequence of Exerciser commands that make + the bug happen.)

    • +
+

It's difficult to fix a bug if it isn't repeatable or if it only +occurs on an operating system I don't have access to.  The more obscure +your circumstances, the more details you will need to provide.

+

Please use the github issue tracker to file bugs.

+

 

+

Product "To Do" List

+

For the NuLib2 application:

+
    +
  • Extract files to AppleDouble format.  This would allow UNIX + AppleShare servers like netatalk to serve up extracted files immediately, + and preserve all file attributes.
    • +
  • Support .2MG disk images automatically when adding and extracting.
    • +
  • Add BinSCII support.
    • +
  • Need some regression tests.
    • +
+

For NufxLib:

+
    +
  • Add native resource fork support (for systems like Mac OS and GS/OS).
    • +
  • Ought to store empty directories.
    • +
  • Make name-comparison sensitivity (for get-by-name and duplicate checking) + configurable.
    • +
  • Need a complete set of regression tests.
    • +
+

  + +

+ + diff --git a/downloads/_vti_cnf/gsnulib.shk b/downloads/_vti_cnf/gsnulib.shk new file mode 100644 index 0000000..0bbd123 --- /dev/null +++ b/downloads/_vti_cnf/gsnulib.shk @@ -0,0 +1,4 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|31 Jan 2000 05:35:38 -0000 +vti_extenderversion:SR|4.0.2.2717 +vti_backlinkinfo:VX|downloads/index.htm diff --git a/downloads/_vti_cnf/index.htm b/downloads/_vti_cnf/index.htm new file mode 100644 index 0000000..0fea906 --- /dev/null +++ b/downloads/_vti_cnf/index.htm @@ -0,0 +1,24 @@ +vti_encoding:SR|utf8-nl +vti_author:SR|fadden +vti_timecreated:TR|23 Dec 1999 21:39:46 -0000 +vti_timelastmodified:TR|19 Feb 2007 23:57:54 -0000 +vti_filesize:IR|4569 +vti_title:SR|NuLib Downloads +vti_metatags:VR|HTTP-EQUIV=Content-Language en-us HTTP-EQUIV=Content-Type text/html;\\ charset=windows-1252 GENERATOR Microsoft\\ FrontPage\\ 4.0 ProgId FrontPage.Editor.Document +vti_progid:SR|FrontPage.Editor.Document +vti_generator:SR|Microsoft FrontPage 4.0 +vti_extenderversion:SR|4.0.2.8912 +vti_backlinkinfo:VX|index.htm +vti_nexttolasttimemodified:TR|19 Feb 2007 23:53:31 -0000 +vti_shadowfiles:VX| +vti_modifiedby:SR|fadden +vti_cacheddtm:TX|19 Feb 2007 23:57:54 -0000 +vti_cachedlinkinfo:VX|H|http://www.faddensoft.com/ciderpress/ H|nulibdist.tar.gz H|http://sourceforge.net/projects/nulib2/ S|http://sourceforge.net/sflogo.php H|nulib2_win32.zip H|nulib325.tar.gz H|nulib_win32.zip H|nulib-tb.zip H|gsnulib.shk H|../library/index.htm +vti_cachedsvcrellinks:VX|NHHS|http://www.faddensoft.com/ciderpress/ NHUS|downloads/nulibdist.tar.gz NHHS|http://sourceforge.net/projects/nulib2/ NSHS|http://sourceforge.net/sflogo.php NHUS|downloads/nulib2_win32.zip FHUS|downloads/nulib325.tar.gz FHUS|downloads/nulib_win32.zip FHUS|downloads/nulib-tb.zip FHUS|downloads/gsnulib.shk FHUS|library/index.htm +vti_cachedtitle:SR|NuLib Downloads +vti_cachedbodystyle:SR| +vti_cachedhasbots:BR|true +vti_cachedhastheme:BR|false +vti_cachedhasborder:BR|true +vti_botnavbits:SW|SHUB +vti_borderaggregate:SR|default diff --git a/downloads/_vti_cnf/nufxdll-dev-202.zip b/downloads/_vti_cnf/nufxdll-dev-202.zip new file mode 100644 index 0000000..7c1981d --- /dev/null +++ b/downloads/_vti_cnf/nufxdll-dev-202.zip @@ -0,0 +1,3 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|10 Mar 2004 21:56:34 -0000 +vti_extenderversion:SR|4.0.2.7802 diff --git a/downloads/_vti_cnf/nufxdll-dev-203.zip b/downloads/_vti_cnf/nufxdll-dev-203.zip new file mode 100644 index 0000000..c855f75 --- /dev/null +++ b/downloads/_vti_cnf/nufxdll-dev-203.zip @@ -0,0 +1,3 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|11 Oct 2004 22:43:46 -0000 +vti_extenderversion:SR|4.0.2.7802 diff --git a/downloads/_vti_cnf/nufxdll-dev-210.zip b/downloads/_vti_cnf/nufxdll-dev-210.zip new file mode 100644 index 0000000..bffec3d --- /dev/null +++ b/downloads/_vti_cnf/nufxdll-dev-210.zip @@ -0,0 +1,4 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|19 Sep 2005 05:34:35 -0000 +vti_extenderversion:SR|4.0.2.8912 +vti_backlinkinfo:VX| diff --git a/downloads/_vti_cnf/nulib-tb.zip b/downloads/_vti_cnf/nulib-tb.zip new file mode 100644 index 0000000..a3b6226 --- /dev/null +++ b/downloads/_vti_cnf/nulib-tb.zip @@ -0,0 +1,4 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|16 Jan 2000 20:55:18 -0000 +vti_extenderversion:SR|4.0.2.2717 +vti_backlinkinfo:VX|downloads/index.htm diff --git a/downloads/_vti_cnf/nulib2_101_win32.zip b/downloads/_vti_cnf/nulib2_101_win32.zip new file mode 100644 index 0000000..c193ca9 --- /dev/null +++ b/downloads/_vti_cnf/nulib2_101_win32.zip @@ -0,0 +1,3 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|23 May 2000 00:54:24 -0000 +vti_extenderversion:SR|4.0.2.5526 diff --git a/downloads/_vti_cnf/nulib2_110_win32.zip b/downloads/_vti_cnf/nulib2_110_win32.zip new file mode 100644 index 0000000..b11caeb --- /dev/null +++ b/downloads/_vti_cnf/nulib2_110_win32.zip @@ -0,0 +1,4 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|11 Oct 2002 23:01:26 -0000 +vti_extenderversion:SR|4.0.2.5526 +vti_backlinkinfo:VX| diff --git a/downloads/_vti_cnf/nulib2_200_win32.zip b/downloads/_vti_cnf/nulib2_200_win32.zip new file mode 100644 index 0000000..8b5ece4 --- /dev/null +++ b/downloads/_vti_cnf/nulib2_200_win32.zip @@ -0,0 +1,4 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|19 Mar 2003 02:47:06 -0000 +vti_extenderversion:SR|4.0.2.6513 +vti_backlinkinfo:VX| diff --git a/downloads/_vti_cnf/nulib2_201_win32.zip b/downloads/_vti_cnf/nulib2_201_win32.zip new file mode 100644 index 0000000..26a85e9 --- /dev/null +++ b/downloads/_vti_cnf/nulib2_201_win32.zip @@ -0,0 +1,4 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|16 Oct 2003 22:59:04 -0000 +vti_extenderversion:SR|4.0.2.7802 +vti_backlinkinfo:VX| diff --git a/downloads/_vti_cnf/nulib2_202_win32.zip b/downloads/_vti_cnf/nulib2_202_win32.zip new file mode 100644 index 0000000..d19c5f8 --- /dev/null +++ b/downloads/_vti_cnf/nulib2_202_win32.zip @@ -0,0 +1,4 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|10 Mar 2004 21:43:56 -0000 +vti_extenderversion:SR|4.0.2.7802 +vti_backlinkinfo:VX| diff --git a/downloads/_vti_cnf/nulib2_203_win32.zip b/downloads/_vti_cnf/nulib2_203_win32.zip new file mode 100644 index 0000000..3f051de --- /dev/null +++ b/downloads/_vti_cnf/nulib2_203_win32.zip @@ -0,0 +1,3 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|11 Oct 2004 22:30:28 -0000 +vti_extenderversion:SR|4.0.2.7802 diff --git a/downloads/_vti_cnf/nulib2_210_win32.zip b/downloads/_vti_cnf/nulib2_210_win32.zip new file mode 100644 index 0000000..2fac406 --- /dev/null +++ b/downloads/_vti_cnf/nulib2_210_win32.zip @@ -0,0 +1,3 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|19 Sep 2005 05:31:47 -0000 +vti_extenderversion:SR|4.0.2.8912 diff --git a/downloads/_vti_cnf/nulib2_211_win32.zip b/downloads/_vti_cnf/nulib2_211_win32.zip new file mode 100644 index 0000000..23d3236 --- /dev/null +++ b/downloads/_vti_cnf/nulib2_211_win32.zip @@ -0,0 +1,3 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|19 Feb 2006 01:54:21 -0000 +vti_extenderversion:SR|4.0.2.8912 diff --git a/downloads/_vti_cnf/nulib2_220_win32.zip b/downloads/_vti_cnf/nulib2_220_win32.zip new file mode 100644 index 0000000..4fe9f90 --- /dev/null +++ b/downloads/_vti_cnf/nulib2_220_win32.zip @@ -0,0 +1,3 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|20 Feb 2007 00:45:59 -0000 +vti_extenderversion:SR|4.0.2.8912 diff --git a/downloads/_vti_cnf/nulib325.tar.gz b/downloads/_vti_cnf/nulib325.tar.gz new file mode 100644 index 0000000..dfc0d17 --- /dev/null +++ b/downloads/_vti_cnf/nulib325.tar.gz @@ -0,0 +1,4 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|07 Jun 1998 05:06:02 -0000 +vti_extenderversion:SR|4.0.2.2717 +vti_backlinkinfo:VX|downloads/index.htm library/index.htm diff --git a/downloads/_vti_cnf/nulib_win32.zip b/downloads/_vti_cnf/nulib_win32.zip new file mode 100644 index 0000000..9fbb551 --- /dev/null +++ b/downloads/_vti_cnf/nulib_win32.zip @@ -0,0 +1,4 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|16 Jan 2000 21:01:54 -0000 +vti_extenderversion:SR|4.0.2.2717 +vti_backlinkinfo:VX|downloads/index.htm diff --git a/downloads/_vti_cnf/nulibdist-100.tar.gz b/downloads/_vti_cnf/nulibdist-100.tar.gz new file mode 100644 index 0000000..6880b5e --- /dev/null +++ b/downloads/_vti_cnf/nulibdist-100.tar.gz @@ -0,0 +1,3 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|19 May 2000 01:04:30 -0000 +vti_extenderversion:SR|4.0.2.5526 diff --git a/downloads/_vti_cnf/nulibdist-101.tar.gz b/downloads/_vti_cnf/nulibdist-101.tar.gz new file mode 100644 index 0000000..f1ea9c0 --- /dev/null +++ b/downloads/_vti_cnf/nulibdist-101.tar.gz @@ -0,0 +1,3 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|23 May 2000 00:46:16 -0000 +vti_extenderversion:SR|4.0.2.5526 diff --git a/downloads/_vti_cnf/nulibdist-110.tar.gz b/downloads/_vti_cnf/nulibdist-110.tar.gz new file mode 100644 index 0000000..04ffd23 --- /dev/null +++ b/downloads/_vti_cnf/nulibdist-110.tar.gz @@ -0,0 +1,4 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|11 Oct 2002 22:59:56 -0000 +vti_extenderversion:SR|4.0.2.5526 +vti_backlinkinfo:VX| diff --git a/downloads/_vti_cnf/nulibdist-200.tar.gz b/downloads/_vti_cnf/nulibdist-200.tar.gz new file mode 100644 index 0000000..817600d --- /dev/null +++ b/downloads/_vti_cnf/nulibdist-200.tar.gz @@ -0,0 +1,4 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|19 Mar 2003 02:47:00 -0000 +vti_extenderversion:SR|4.0.2.6513 +vti_backlinkinfo:VX| diff --git a/downloads/_vti_cnf/nulibdist-201.tar.gz b/downloads/_vti_cnf/nulibdist-201.tar.gz new file mode 100644 index 0000000..26a85e9 --- /dev/null +++ b/downloads/_vti_cnf/nulibdist-201.tar.gz @@ -0,0 +1,4 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|16 Oct 2003 22:59:04 -0000 +vti_extenderversion:SR|4.0.2.7802 +vti_backlinkinfo:VX| diff --git a/downloads/_vti_cnf/nulibdist-202.tar.gz b/downloads/_vti_cnf/nulibdist-202.tar.gz new file mode 100644 index 0000000..9f36f92 --- /dev/null +++ b/downloads/_vti_cnf/nulibdist-202.tar.gz @@ -0,0 +1,4 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|10 Mar 2004 21:38:18 -0000 +vti_extenderversion:SR|4.0.2.7802 +vti_backlinkinfo:VX| diff --git a/downloads/_vti_cnf/nulibdist-203.tar.gz b/downloads/_vti_cnf/nulibdist-203.tar.gz new file mode 100644 index 0000000..f41dc97 --- /dev/null +++ b/downloads/_vti_cnf/nulibdist-203.tar.gz @@ -0,0 +1,3 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|11 Oct 2004 22:25:22 -0000 +vti_extenderversion:SR|4.0.2.7802 diff --git a/downloads/_vti_cnf/nulibdist-210.tar.gz b/downloads/_vti_cnf/nulibdist-210.tar.gz new file mode 100644 index 0000000..a0b85ba --- /dev/null +++ b/downloads/_vti_cnf/nulibdist-210.tar.gz @@ -0,0 +1,3 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|19 Sep 2005 05:31:43 -0000 +vti_extenderversion:SR|4.0.2.8912 diff --git a/downloads/_vti_cnf/nulibdist-211.tar.gz b/downloads/_vti_cnf/nulibdist-211.tar.gz new file mode 100644 index 0000000..77465a9 --- /dev/null +++ b/downloads/_vti_cnf/nulibdist-211.tar.gz @@ -0,0 +1,3 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|19 Feb 2006 01:54:39 -0000 +vti_extenderversion:SR|4.0.2.8912 diff --git a/downloads/_vti_cnf/nulibdist-220.tar.gz b/downloads/_vti_cnf/nulibdist-220.tar.gz new file mode 100644 index 0000000..8cdd7fb --- /dev/null +++ b/downloads/_vti_cnf/nulibdist-220.tar.gz @@ -0,0 +1,3 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|19 Feb 2007 23:46:00 -0000 +vti_extenderversion:SR|4.0.2.8912 diff --git a/downloads/gsnulib.shk b/downloads/gsnulib.shk new file mode 100644 index 0000000..06ad72a Binary files /dev/null and b/downloads/gsnulib.shk differ diff --git a/downloads/index.htm b/downloads/index.htm new file mode 100644 index 0000000..2183e4f --- /dev/null +++ b/downloads/index.htm @@ -0,0 +1,84 @@ + + + + + + + +NuLib Downloads + + + +
+ +

NuLib Downloads
+[ Home ] [ NuLib Downloads ] [ NuLib Library ] [ NuLib2 Manual ] [ NufxLib API ] [ Bugs & Features ]

+
+ +
+ +

 

+ +

Hey!  If you're looking for a GUI "ShrinkIt for Windows", take a look at CiderPress, which uses NufxLib to access ShrinkIt archives.

+

NuLib2 and NufxLib

+

Source code and Win32 binaries are available for download from +github. Click on +the "releases" link.

+

NuLib2 and NufxLib are packaged together.  Instructions for +building them are included with the sources.  Version 2.2.2 has been built +and tested on the following systems:

+
    +
  • [x86 MSVC++2013] Windows 7
    • +
  • [x86] Ubuntu 14.04 (gcc 4.8.2)
    • +
+Earlier versions have also been tested on: +
    +
  • [x86 MSVC++6.0] Windows XP SP2
    • +
  • [x86] Fedora Core 4 (gcc 4.0)
    • +
  • [x86] Darwin (Mac OS X)
    • +
  • [x86 MSVC++6.0] Windows 2000 Professional SP4
    • +
  • [x86 MSVC++6.0] Windows XP Professional SP1, SP2
    • +
  • [x86 MSVC++6.0] Windows 98SE
    • +
  • [x86 DJGPP] Windows 98SE
    • +
  • [x86 MSVC++6.0] Windows NT 4 Server
    • +
  • [x86] Linux (Debian 2.2)
    • +
  • [x86] Linux (Red Hat 5.x, 6.0. 6.1, 7.3, 8.0, 9.0, FC3, FC4)
    • +
  • [x86] Solaris 7, 8
    • +
  • [x86] FreeBSD (4.7-RC and 4.8)
    • +
  • [x86] OpenBSD 2.5 and 2.6
    • +
  • [x86] BeOS 4.5.2
    • +
  • [PPC - G4] Mac OS X 10.2 SERVER Edition
    • +
  • [PPC - RS/6000] Linux 2.2 (Debian 3.0)
    • +
  • [PPC - xlc] AIX 4.3
    • +
  • [PPC - mwerks] BeOS 4.5.2
    • +
  • [Alpha] Linux 2.2 (Debian 3.0)
    • +
  • [AMD64] Linux 2.4 (SuSE 8 ES on Opteron)
    • +
  • [Sparc - R220] Sun Solaris 8
    • +
  • [Sparc] SunOS 4.1.4
    • +
  • [Sparc - SunPro C] Solaris 2.6
    • +
  • [Sparc] Solaris 7
    • +
  • [CerfCube SA1110] Linux 2.4 (Debian 3.0)
    • +
+

NuLib "Classic"

+

The original NuLib has been ported to many different systems.  Version +3.2.5 is functionally similar to v3.2.4, but the source code and documentation +were cleaned up by Devin Reade, making it easier to build.  NuLib2 is +generally superior, but NuLib does have the advantage of running under 16-bit MS-DOS +and GS/OS.

+

Documentation is included with the sources and binaries.

+

Source code:

+ +

Binaries:

+ +Source code for older versions of the original NuLib are available in the library. +

 

+ + diff --git a/downloads/nufxdll-dev-202.zip b/downloads/nufxdll-dev-202.zip new file mode 100644 index 0000000..16054da Binary files /dev/null and b/downloads/nufxdll-dev-202.zip differ diff --git a/downloads/nufxdll-dev-203.zip b/downloads/nufxdll-dev-203.zip new file mode 100644 index 0000000..bf63a82 Binary files /dev/null and b/downloads/nufxdll-dev-203.zip differ diff --git a/downloads/nufxdll-dev-210.zip b/downloads/nufxdll-dev-210.zip new file mode 100644 index 0000000..a1b18e4 Binary files /dev/null and b/downloads/nufxdll-dev-210.zip differ diff --git a/downloads/nulib-tb.zip b/downloads/nulib-tb.zip new file mode 100644 index 0000000..f9dcead Binary files /dev/null and b/downloads/nulib-tb.zip differ diff --git a/downloads/nulib2_101_win32.zip b/downloads/nulib2_101_win32.zip new file mode 100644 index 0000000..2e3d6f8 Binary files /dev/null and b/downloads/nulib2_101_win32.zip differ diff --git a/downloads/nulib2_110_win32.zip b/downloads/nulib2_110_win32.zip new file mode 100644 index 0000000..4788f48 Binary files /dev/null and b/downloads/nulib2_110_win32.zip differ diff --git a/downloads/nulib2_200_win32.zip b/downloads/nulib2_200_win32.zip new file mode 100644 index 0000000..0f1628e Binary files /dev/null and b/downloads/nulib2_200_win32.zip differ diff --git a/downloads/nulib2_201_win32.zip b/downloads/nulib2_201_win32.zip new file mode 100644 index 0000000..78ea93b Binary files /dev/null and b/downloads/nulib2_201_win32.zip differ diff --git a/downloads/nulib2_202_win32.zip b/downloads/nulib2_202_win32.zip new file mode 100644 index 0000000..611036d Binary files /dev/null and b/downloads/nulib2_202_win32.zip differ diff --git a/downloads/nulib2_203_win32.zip b/downloads/nulib2_203_win32.zip new file mode 100644 index 0000000..71a35ce Binary files /dev/null and b/downloads/nulib2_203_win32.zip differ diff --git a/downloads/nulib2_210_win32.zip b/downloads/nulib2_210_win32.zip new file mode 100644 index 0000000..1e94a42 Binary files /dev/null and b/downloads/nulib2_210_win32.zip differ diff --git a/downloads/nulib2_211_win32.zip b/downloads/nulib2_211_win32.zip new file mode 100644 index 0000000..c2a38a2 Binary files /dev/null and b/downloads/nulib2_211_win32.zip differ diff --git a/downloads/nulib2_220_win32.zip b/downloads/nulib2_220_win32.zip new file mode 100644 index 0000000..ec3dc14 Binary files /dev/null and b/downloads/nulib2_220_win32.zip differ diff --git a/downloads/nulib325.tar.gz b/downloads/nulib325.tar.gz new file mode 100644 index 0000000..8a96378 Binary files /dev/null and b/downloads/nulib325.tar.gz differ diff --git a/downloads/nulib_win32.zip b/downloads/nulib_win32.zip new file mode 100644 index 0000000..98eab81 Binary files /dev/null and b/downloads/nulib_win32.zip differ diff --git a/downloads/nulibdist-100.tar.gz b/downloads/nulibdist-100.tar.gz new file mode 100644 index 0000000..1255beb Binary files /dev/null and b/downloads/nulibdist-100.tar.gz differ diff --git a/downloads/nulibdist-101.tar.gz b/downloads/nulibdist-101.tar.gz new file mode 100644 index 0000000..a6c0f12 Binary files /dev/null and b/downloads/nulibdist-101.tar.gz differ diff --git a/downloads/nulibdist-110.tar.gz b/downloads/nulibdist-110.tar.gz new file mode 100644 index 0000000..6a78ffa Binary files /dev/null and b/downloads/nulibdist-110.tar.gz differ diff --git a/downloads/nulibdist-200.tar.gz b/downloads/nulibdist-200.tar.gz new file mode 100644 index 0000000..9c7b227 Binary files /dev/null and b/downloads/nulibdist-200.tar.gz differ diff --git a/downloads/nulibdist-201.tar.gz b/downloads/nulibdist-201.tar.gz new file mode 100644 index 0000000..4588080 Binary files /dev/null and b/downloads/nulibdist-201.tar.gz differ diff --git a/downloads/nulibdist-202.tar.gz b/downloads/nulibdist-202.tar.gz new file mode 100644 index 0000000..0cbeb16 Binary files /dev/null and b/downloads/nulibdist-202.tar.gz differ diff --git a/downloads/nulibdist-203.tar.gz b/downloads/nulibdist-203.tar.gz new file mode 100644 index 0000000..dec5403 Binary files /dev/null and b/downloads/nulibdist-203.tar.gz differ diff --git a/downloads/nulibdist-210.tar.gz b/downloads/nulibdist-210.tar.gz new file mode 100644 index 0000000..4a34178 Binary files /dev/null and b/downloads/nulibdist-210.tar.gz differ diff --git a/downloads/nulibdist-211.tar.gz b/downloads/nulibdist-211.tar.gz new file mode 100644 index 0000000..582287a Binary files /dev/null and b/downloads/nulibdist-211.tar.gz differ diff --git a/downloads/nulibdist-220.tar.gz b/downloads/nulibdist-220.tar.gz new file mode 100644 index 0000000..dda8647 Binary files /dev/null and b/downloads/nulibdist-220.tar.gz differ diff --git a/index.htm b/index.htm new file mode 100644 index 0000000..2b4b11d --- /dev/null +++ b/index.htm @@ -0,0 +1,167 @@ + + + + + + +NuLib Home Page + + + + + + + + + +

NuLib Home Page

+

Shortcuts
+go to the downloads area
+read the NuLib2 manual
+see the NufxLib library API documentation
+report a bug in NufxLib or NuLib2
+check the library for old programs and misc docs

+

 

+

This is the home page for NuLib, NuLib2, NufxLib, and various items related +to ShrinkIt and NuFX archives.

+

 

+

What is NuLib?

+

NuLib is a disk and file archive program, similar in principle to PKZIP.  Instead of ZIP +archives, it manipulates NuFX archives, which are usually identified with +".SHK", ".SDK", or ".BXY".

+

The ".SHK" file extension is derived from ShrinkIt, the de facto +archiving standard for Apple II computers.  Both NuFX and ShrinkIt were developed by Andy Nicholas, +and were initially released in January of 1989.

+

In mid-1989, while a sophomore in college, I started playing around +with the file format.  My goal was to write a simple program that could +list the contents of a NuFX file.  I had recently finished my first class +on C programming, and wanted a small project to play around with.  I never +intended to go any further with it than just being able to list files.

+

Three name changes (NuView to NuARC to CShrink to NuLib) and more than three +years later, NuLib was a full-featured archiver, and the code had finally settled down to where it could be left +alone.  Except for a major overhaul by Devin Reade in late 1996, NuLib +hasn't changed much since.

+

What is NuLib2?

+

NuLib was functional, but had a number of flaws.  It couldn't handle +resource forks, it was clumsy to use with BXY (ShrinkIt wrapped in Binary II) +archives, it used more memory than it needed to, and some features -- notably +the archive integrity test -- were entirely broken.  Due to generally poor +architecture, it was difficult to fix problems and add features.

+

NuLib2 is a replacement for NuLib.  It does pretty much everything the +original NuLib did, and adds a number of new features.  NuLib2 is +distributed as source code, under the terms of the BSD License.

+

The NuLib2 manual has a quick comparison of +the two programs.

+ +

One additional "feature" of NuLib2: it was built on top of NufxLib.

+

What is NufxLib?

+

NufxLib is a NuFX file archive manipulation library.  Unlike most other +compression library products, NufxLib goes beyond extracting and listing files.  +Full support for additions, deletions, and renaming of archived files is +supported, with a transaction-oriented interface for maximum efficiency and +reliability.

+

A thorough description of the library's features and interface is available here.  +NufxLib is distributed as source code under the terms of the BSD License.

+

Possible uses for NufxLib:

+ +

Why did I do this?

+

In late 1997, I cranked up my Apple IIgs for the first time in a long +while.  It failed to boot.  The old 100MB hard drive had seized up +from "stiction".  A friend of mine and I managed to resurrect +that drive and a second 80MB hard drive that had also seized up, but it was +clear the drives' days were numbered.

+

I replaced the old ones with a new 2GB drive, which was the smallest I could +find in retail stores at the time.  It occurred to me then that it would be +useful, as well as prudent, to have a complete file archive of the contents of +my hard drive.  GS/ShrinkIt was capable of constructing such a thing, but +couldn't extract from it because the archive was too large.  YankIt and NuLib could do the extraction, but +were too clumsy to be useful.

+

I resolved to write a program that could handle these archives.  I +thought it would be best to write it as a Win32 GUI application.  However, +I also wanted a better version of NuLib for UNIX.  The NuLib sources are a +pile of dung, so adding the features I wanted would be most easily accomplished +by rewriting the whole thing from scratch.

+

Around the middle of 1998, I started fleshing out the API for something I +called "NufxLib".  The library would do all the hard work for +both command line and GUI applications, and perhaps would be embedded into Apple +II emulators as well, allowing seamless access to ShrinkIt-compressed disk +images.

+

For the next 1.5 years, I would occasionally pick up the project and then +leave it alone for weeks at a time.  This continued until I left a big +company for a small startup, and knew that my free time was about to evaporate +entirely.  I decided to finish up what I could and make it available.  +Version 1.0 was released in May of 2000.

+

In December of 2002, I decided it was time to learn how to write Windows +software.  Learning a new system is easier when you're working with +something you know, so I decided to use NufxLib as the foundation of a Win32 +application.  The result, CiderPress, is available from the faddenSoft +web site.

+

Visible Changes in v1.1

+

New stuff in NufxLib v1.1:

+ +

New stuff in NuLib2 v1.1:

+ +

Visible Changes in v2.0

+

Version 2.0 is actually a rather small release.  Some changes in NufxLib broke +binary compatibility, so it was necessary to increment the major version.

+

New stuff in NufxLib v2.0:

+ +

New stuff in NuLib2 v2.0:

+ +

Future Directions

+

The code appears to be stable, so I'm going to leave it alone for a while.  +If you're interested in developing applications with NufxLib, send a message to +fadden -at- fadden.com.

+
+

NuLib, NuLib2, and NufxLib were written by Andy +McFadden.  Please see the documentation for each product to +see a list of contributors.

+ + + + diff --git a/library/Crc16.c.txt b/library/Crc16.c.txt new file mode 100644 index 0000000..2f06f2e --- /dev/null +++ b/library/Crc16.c.txt @@ -0,0 +1,107 @@ +/* + * Compute 16-bit CRCs. + * + * Adapted from code in NuLib, http://www.nulib.com/. + */ +#include + +/* define this if you want a table-based lookup */ +#define CRC_TAB + +#ifdef CRC_TAB +/* + * updcrc macro derived from article Copyright (C) 1986 Stephen Satchell. + * NOTE: First srgument must be in range 0 to 255. + * Second argument is referenced twice. + * + * Programmers may incorporate any or all code into their programs, + * giving proper credit within the source. Publication of the + * source routines is permitted so long as proper credit is given + * to Stephen Satchell, Satchell Evaluations and Chuck Forsberg, + * Omen Technology. + */ + +/*#define updcrc(cp, crc) ( crctab[((crc >> 8) & 255)] ^ (crc << 8) ^ cp)*/ +#define updcrc(cp, crc) ( (crctab[((crc >> 8) & 0xFF) ^ cp] ^ (crc << 8)) & 0xFFFF) + +/* crctab calculated by Mark G. Mendel, Network Systems Corporation */ +const unsigned short crctab[256] = { + 0x0000, 0x1021, 0x2042, 0x3063, 0x4084, 0x50a5, 0x60c6, 0x70e7, + 0x8108, 0x9129, 0xa14a, 0xb16b, 0xc18c, 0xd1ad, 0xe1ce, 0xf1ef, + 0x1231, 0x0210, 0x3273, 0x2252, 0x52b5, 0x4294, 0x72f7, 0x62d6, + 0x9339, 0x8318, 0xb37b, 0xa35a, 0xd3bd, 0xc39c, 0xf3ff, 0xe3de, + 0x2462, 0x3443, 0x0420, 0x1401, 0x64e6, 0x74c7, 0x44a4, 0x5485, + 0xa56a, 0xb54b, 0x8528, 0x9509, 0xe5ee, 0xf5cf, 0xc5ac, 0xd58d, + 0x3653, 0x2672, 0x1611, 0x0630, 0x76d7, 0x66f6, 0x5695, 0x46b4, + 0xb75b, 0xa77a, 0x9719, 0x8738, 0xf7df, 0xe7fe, 0xd79d, 0xc7bc, + 0x48c4, 0x58e5, 0x6886, 0x78a7, 0x0840, 0x1861, 0x2802, 0x3823, + 0xc9cc, 0xd9ed, 0xe98e, 0xf9af, 0x8948, 0x9969, 0xa90a, 0xb92b, + 0x5af5, 0x4ad4, 0x7ab7, 0x6a96, 0x1a71, 0x0a50, 0x3a33, 0x2a12, + 0xdbfd, 0xcbdc, 0xfbbf, 0xeb9e, 0x9b79, 0x8b58, 0xbb3b, 0xab1a, + 0x6ca6, 0x7c87, 0x4ce4, 0x5cc5, 0x2c22, 0x3c03, 0x0c60, 0x1c41, + 0xedae, 0xfd8f, 0xcdec, 0xddcd, 0xad2a, 0xbd0b, 0x8d68, 0x9d49, + 0x7e97, 0x6eb6, 0x5ed5, 0x4ef4, 0x3e13, 0x2e32, 0x1e51, 0x0e70, + 0xff9f, 0xefbe, 0xdfdd, 0xcffc, 0xbf1b, 0xaf3a, 0x9f59, 0x8f78, + 0x9188, 0x81a9, 0xb1ca, 0xa1eb, 0xd10c, 0xc12d, 0xf14e, 0xe16f, + 0x1080, 0x00a1, 0x30c2, 0x20e3, 0x5004, 0x4025, 0x7046, 0x6067, + 0x83b9, 0x9398, 0xa3fb, 0xb3da, 0xc33d, 0xd31c, 0xe37f, 0xf35e, + 0x02b1, 0x1290, 0x22f3, 0x32d2, 0x4235, 0x5214, 0x6277, 0x7256, + 0xb5ea, 0xa5cb, 0x95a8, 0x8589, 0xf56e, 0xe54f, 0xd52c, 0xc50d, + 0x34e2, 0x24c3, 0x14a0, 0x0481, 0x7466, 0x6447, 0x5424, 0x4405, + 0xa7db, 0xb7fa, 0x8799, 0x97b8, 0xe75f, 0xf77e, 0xc71d, 0xd73c, + 0x26d3, 0x36f2, 0x0691, 0x16b0, 0x6657, 0x7676, 0x4615, 0x5634, + 0xd94c, 0xc96d, 0xf90e, 0xe92f, 0x99c8, 0x89e9, 0xb98a, 0xa9ab, + 0x5844, 0x4865, 0x7806, 0x6827, 0x18c0, 0x08e1, 0x3882, 0x28a3, + 0xcb7d, 0xdb5c, 0xeb3f, 0xfb1e, 0x8bf9, 0x9bd8, 0xabbb, 0xbb9a, + 0x4a75, 0x5a54, 0x6a37, 0x7a16, 0x0af1, 0x1ad0, 0x2ab3, 0x3a92, + 0xfd2e, 0xed0f, 0xdd6c, 0xcd4d, 0xbdaa, 0xad8b, 0x9de8, 0x8dc9, + 0x7c26, 0x6c07, 0x5c64, 0x4c45, 0x3ca2, 0x2c83, 0x1ce0, 0x0cc1, + 0xef1f, 0xff3e, 0xcf5d, 0xdf7c, 0xaf9b, 0xbfba, 0x8fd9, 0x9ff8, + 0x6e17, 0x7e36, 0x4e55, 0x5e74, 0x2e93, 0x3eb2, 0x0ed1, 0x1ef0 +}; +#endif + + +/* + * Calculate CRC on a region. Pass in seed (which can be an initial + * value -- typically zero -- or the result of an earlier computation), + * and the pointer and length of the buffer to compute on. + * + * A CRC is the result of a mathematical operation based on the + * coefficients of a polynomial when multiplied by X^16 then divided by + * the generator polynomial (X^16 + X^12 + X^5 + 1) using modulo two + * arithmetic. + * + * This routine is a slightly modified verison of one found in: + * _Advanced Programming Techniques for the Apple //gs Toolbox_ + * By Morgan Davis and Dan Gookin (Compute! Publications, Inc.) + * + * This can either calculate the CRC bit-by-bit or use a table. + * Depending on CPU architecture, one may be dramatically faster than + * the other. + */ +unsigned short +CalcCRC16(unsigned short seed, const unsigned char* ptr, int count) +{ + unsigned short CRC = seed; +#ifndef CRC_TAB + int x; + assert(sizeof(unsigned short) == 2); /* I think this is assumed */ +#endif + + do { +#ifndef CRC_TAB + CRC ^= *ptr++ << 8; /* XOR hi-byte of CRC w/dat */ + for (x = 8; x; --x) /* Then, for 8 bit shifts... */ + if (CRC & 0x8000) /* Test hi order bit of CRC */ + CRC = CRC << 1 ^ 0x1021; /* if set, shift & XOR w/$1021 */ + else + CRC <<= 1; /* Else, just shift left once. */ +#else + CRC = updcrc(*ptr++, CRC); /* look up new value in table */ +#endif + } while (--count); + + return (CRC); +} + diff --git a/library/FTN.e00001.htm b/library/FTN.e00001.htm new file mode 100644 index 0000000..f476900 --- /dev/null +++ b/library/FTN.e00001.htm @@ -0,0 +1,344 @@ + + + + + + +FTN.e00001 + + + +
+ +


+

+
+ +
+

Apple II FTN - AppleSingle File

+

Back to nulib.com library

+
+Apple II
+File Type Notes
+_____________________________________________________________________________
+                                                  Developer Technical Support
+
+File Type:         $E0 (224)
+Auxiliary Type:    $0001
+
+Full Name:     AppleSingle File
+Short Name:    AppleSingle File
+
+Revised by:    Matt Deatherage                                   January 1991
+Written by:    Matt Deatherage                                     March 1989
+
+Files of this type and auxiliary type contain a file in AppleSingle format.
+Changes since March 1990:  Added information about AppleSingle version 2.0.
+_____________________________________________________________________________
+
+AppleSingle is one of two standards (the other is AppleDouble) put forth by
+Apple Computer, Inc. for representing files on foreign file systems while
+preserving all attributes of the file's home system on file systems that do
+not support the same attributes.
+
+Experience indicated that a single format would be inadequate to cover all
+cases.  Two closely related formats, however, can serve most needs.  Although
+the primary impetus for developing these formats is storing extended files
+(files with both resource and data forks) on file systems that do not support
+the notion of two forks, the proposed formats are general enough that theycan be
+used to represent a file from any file system on any other file system.
+
+AppleSingle keeps all attributes and the contents of both forks in a single file
+in the foreign file system, and this Note describes this file format.
+AppleDouble keeps the data fork as a separate file from the file attributes and
+the resource fork, and is described in the File Type Notes for File Type $E0,
+Auxiliary Types $0002 and $0003.
+
+AppleSingle is intended to be used primarily as a storage format, especially for
+cases where you must store an extended file on a foreign file system and later
+reconstruct the extended file.  AppleDouble is more appropriate for applications
+where the users of the foreign file system might want to modify the contents of
+the file.  Since most applications keep file data in the data fork, AppleDouble
+format saves the contents of the data fork in one file.All other file
+attributes, including the resource fork, are kept in a separate file.
+
+
+Reasons for Using AppleSingle
+
+There are several reasons for supporting an interchange format between file
+systems.  Perhaps the most germane is one of the least obvious:  handling
+extended files on foreign file systems which do not support extended files.
+
+For example, the ProDOS FST in GS/OS can create an extended file on a ProDOS
+disk.  However, ProDOS 8 is unable to operate on the file, since it sees itas
+having an unsupported storage type.  If a telecommunications program or other
+utility capable of transferring files is operating under ProDOS 8 andattempts to
+receive an extended file, it is unable to create the file.
+
+At this point, the application could use READ_BLOCK and WRITE_BLOCK commands,
+along with a knowledge of the ProDOS file system, to create the file on its own.
+However, this is strongly discouraged.  The ProDOS file system format for
+extended files is not documented and could change in the future.  In addition,
+the program could be running on a eight-bit system.  If the disk is only used on
+an eight-bit system, the extended files would not only be unwanted, but also
+unremovable without using the disk on an Apple IIGS or later system running
+GS/OS.
+
+However, if the application is aware of the AppleSingle format, it canquickly
+store an extended file in AppleSingle, leaving the conversion back to the
+extended file to GS/OS, or another operating system.  This is the recommended
+way for ProDOS 8 applications to create and handle extended files.  Useeither
+AppleSingle or AppleDouble.
+
+
+AppleSingle Format
+
+An AppleSingle file contains a header followed by data.  The header consists of
+several fixed fields and a list of entry descriptors, each pointing to an entry.
+Apple defines the following standard entries:  Data Fork, Resource Fork, Real
+Name (name in the home file system), Comment, Icon and File Info. Each entry is
+optional, so it may not appear in the file.
+
+Note:  All numeric entries, including entries representing ProDOS data
+       structures (such as file type and auxiliary type) are Reverse
+       ordered.  This is provided so any host CPU can attempt to
+       interpret entries in the header without having to know the
+       standard byte-ordering of the home file system.  Therefore, in
+       this Note you see descriptive entries like "Rev. 4 Bytes."  This
+       serves as a reminder that all header fields are stored high byte
+       first, even though the notation Bytes does not imply any specific
+       ordering in other File Type Notes.
+
+Also note that ASCII strings are not stored in reverse order, just non-ASCII
+constants.
+
+The Header:
+
+Magic Number         Rev. Long    The Magic Number field is modeled after
+                                  the feature in UNIX.  It is intended to be
+                                  used in whatever way the foreign file
+                                  system distinguishes a file as AppleSingle
+                                  format.  See the section "Identifying
+                                  AppleSingle Files."  The Magic Number for
+                                  AppleSingle format is $00051600, which is
+                                  stored reverse as $00 $05 $16 $00 (reverse
+                                  of normal 65816/6502 order).
+Version Number       Rev. Long    The version of AppleSingle format, in case
+                                  the format evolves (more fields may be
+                                  added to the header).  The version
+                                  described here is $00010000, stored
+                                  (reverse) as $00 $01 $00 $00.
+Home File System     16 Bytes     A fixed-length, 16-byte ASCII string not
+                                  preceded by a length byte, but possibly
+                                  padded with blanks.  Apple has defined
+                                  these values:
+                                  ProDOS    $50726F444F5320202020202020202020
+                                  Macintosh $4D6163696E746F736820202020202020
+                                  MS-DOS    $4D532D444F5320202020202020202020
+                                  Unix      $556E9878202020202020202020202020
+                                  VAX VMS   $56415820564D53202020202020202020
+                                  Apple welcomes suggestions for other file
+                                  systems that should be included in this
+                                  list.
+Number of entries    Rev. Word    Tells how many different entries are
+                                  included in the file.  This unsigned
+                                  reverse word may be zero.  If it is non-
+                                  zero, then that number of entry
+                                  descriptors immediately follows this
+                                  field.
+
+
+For Each Entry:
+
+Entry ID             Rev. Long    Identifies the entry.  Apple has defined
+                                  the following Entry IDs and their values:
+                                  1 = Data Fork
+                                  2 = Resource Fork
+                                  3 = Real Name (The file's name in the home
+                                      file system)
+                                  4 = Comment* (standard Macintosh comment)
+                                  5 = Icon, B&W* (standard Macintosh black
+                                      and white icon)
+                                  6 = Icon, Color* (reserved for Macintosh
+                                      color icon)
+                                  7 = File Info (file attributes,dates, etc.)
+                                  9 = Finder Info* (standard Macintosh
+                                      Finder Info)
+                                  Entry IDs marked with asterisks (*) are
+                                  not used for most files created under
+                                  ProDOS or GS/OS.  Furthermore, icon
+                                  entries probably do not appear in most
+                                  files since they are typically stored as a
+                                  bundle in the application file's resource
+                                  fork on the Macintosh.  Apple reserves the
+                                  range of Entry IDs from $0 to $7FFFFFFF
+                                  for future use.  The rest of the range is
+                                  available for other systems to define
+                                  their own entries.  Apple does not
+                                  arbitrate the use of the rest of the
+                                  range.
+                                  Descriptions of the standard entries are
+                                  given below.
+Offset               Rev. Long    An unsigned reverse long which indicates
+                                  the byte offset from the start of the file
+                                  to the start of the entry.
+Entry Length         Rev. Long    An unsigned reverse long which indicates
+                                  the length of the entry in bytes.  The
+                                  length may be zero.
+
+
+Standard Entries:
+
+The Real Name Entry:
+
+The Real Name entry indicates the file's original filename in the host file
+system.  This is not a Pascal or C string; it is just ASCII data.  The length is
+indicated by the Entry Length field for the Real Name entry.
+
+The File Info Entry:
+
+The File Info entry (Entry ID = 7) is different for each home file system. For
+ProDOS files, the entry is 16 bytes long and consists of the creationdate and
+time and the modification date and time in ProDOS 8 (ProDOS 16/class zero GS/OS)
+form, the access word, a two-byte file type and four-byte auxiliary type.  This
+is detailed in standard format below, along with defined FileInfo entries for
+some other file systems.
+
+ProDOS:
+
+Create Date        Rev. 2 Bytes  Creation date packed into standard
+                                 ProDOS 8 format.
+Create Time        Rev. 2 Bytes  Creation time packed into standard
+                                 ProDOS 8 format.
+Modification Date  Rev. 2 Bytes  Modification date packed into
+                                 standard ProDOS 8 format.
+Modification Time  Rev. 2 Bytes  Modification time packed into
+                                 standard ProDOS 8 format.
+Access             Rev. Word     The file's access.  This may be used
+                                 directly in ProDOS 16 or GS/OS calls; only
+                                 the low byte is significant to ProDOS 8.
+File Type          Rev. Word     The file type of the original file.  Only
+                                 the low byte is significant to ProDOS 8.
+Auxiliary Type     Rev. Long     The auxiliary type of the original file.
+                                 Only the low word is significant to ProDOS
+                                 8.
+
+Note:  Although the ProDOS Access field, File Type and Auxiliary Type are
+       the same length as found in ProDOS 16 and GS/OS structures, the
+       Create and Modification Dates and Times are stored in two-byte
+       (albeit byte-reversed) ProDOS 8 format, not eight-byte Apple IIGS
+       format.
+
+Macintosh:
+
+Create Date          Rev. Long    Unsigned number of seconds between
+                                  January 1, 1904, and the creation time of
+                                  this file.
+Modification Date    Rev. Long    Unsigned number of seconds between
+                                  January 1, 1904, and the last modification
+                                  of this file.
+Last Backup Date     Rev. Long    Unsigned number of seconds between
+                                  January 1, 1904, and the last backup time
+                                  of this file.
+Attributes           Rev. Long    32 boolean flags.  Once the bytes are
+                                  unreversed, bit zero is the locked bit and
+                                  bit one is the protected bit.
+
+MS-DOS:
+
+Modification Date    Rev. 4 Bytes    MS-DOS format modification date.
+Attributes           Rev. 2 Bytes    MS-DOS attributes.
+
+Unix:
+
+Create Date/Time       Rev. 4 Bytes    Unix creation date and time.
+Last Use Date/Time     Rev. 4 Bytes    Unix time for the last
+                                       time this file was used.
+Last Mod. Date/Time    Rev. 4 Bytes    Unix time for the last
+                                       time this file was modified.
+
+
+The Finder Info Entry:
+
+The Finder Info entry (Entry ID = 9) is for files where the host file system is
+Macintosh.  It consists of 16 bytes of Finder Info followed by 16 bytes of
+Extended Finder Info.  These are the fields ioFlFndrInfo followed by
+ioFlXFndrInfo, as described in Inside Macintosh, Volume IV-183.  Newlycreated
+files have zeroes in all Finder Info subfields.  If you are creating an
+AppleSingle file whose home system is Macintosh, you may zero all unknown
+fields, but you may want to set the fdType and fdCreator subfields.
+
+
+The Entries:
+
+The entries themselves follow the header field and the entry descriptors.The
+actual data representing each entry must be in a single, contiguous block. The
+offset field in that entry's descriptor points to it.  The entries could appear
+in any order, but since the data fork is the entry that is most commonly
+extended, Apple strongly recommends that the data fork always bekept last in the
+file to facilitate its extension.  Apple also recommends that those entries that
+are most often read, such as Real Name, File Info (and Finder Info if present)
+be kept as close as possible to the header tomaximize the probability that a
+read of the first few blocks of the file retrieves these entries.
+
+It is possible to have holes in the file (unused space between entries).  To
+find the holes, you must take the list of entry descriptors and sort theminto
+increasing offset order.  If the offset field of an entry is greater than the
+offset plus the length of the previous entry (sorted), then a hole exists
+between the entries.  You can make use of such holes; for example, if afile's
+comment is ten bytes long, you could create a hole of 190 bytes after the
+comment field to easily allow for the comment to later expand to its maximum
+length of 200 bytes.  Because an AppleSingle file may contain holes, you must
+find each entry by getting its offset from its entry descriptor, not by assuming
+that it begins after the previous entry.
+
+Byte ordering in file header fields follows 68000 convention, and each header
+field has been so noted by the Reverse operator.
+
+
+Identifying AppleSingle files
+
+As this is an interchange format, from a ProDOS directory entry there is no way
+to guarantee which files are AppleSingle files.  Apple has allocated File Type
+$E0, Auxiliary Type $0001 for files which are AppleSingle files.  We strongly
+encourage ProDOS 8 and GS/OS applications to use this file type and auxiliary
+type assignment when creating AppleSingle files.
+
+AppleSingle files which do not have file type $E0 and auxiliary type $0001can
+most easily be identified by opening them and attempting to interpret them. If
+they are not AppleSingle files, the Magic Number is not contained in the first
+four bytes of the file.  The chances that the file would begin with those four
+bytes and not be an AppleSingle file, on a purely random basis,are 4,294,967,295
+to 1.  The chances that both the Magic Number and the Version bytes would be the
+same in a non-AppleSingle file are roughly 1.8 x 10^19 to 1.
+
+
+About AppleSingle 2.0
+
+AppleSingle 2.0 is a revision to the original AppleSingle specification
+described in this Note.  AppleSingle 2.0 comes closer to the ideal of an
+interchange format by allowing file information for multiple file systems in the
+same AppleSingle file.
+
+AppleSingle 2.0 basically replaces the File Info entry (ID = 7) with a File
+Dates entry (ID = 8) and one or more host file system entries, such as a
+Macintosh File Info entry (ID = 10), a ProDOS File Info entry (ID = 11), or an
+MS-DOS File Info entry (ID = 12).  Information on these entries and AppleSingle
+2.0 can be found in the AppleSingle/AppleDouble Formats for Foreign Files
+Developer's Note, available from APDA, AppleLink, and the Developer CD series.
+
+
+Further Reference
+_____________________________________________________________________________
+  o  Inside Macintosh, Volume IV
+  o  ProDOS 8 Technical Reference Manual
+  o  GS/OS Reference
+  o  AppleSingle/AppleDouble Formats for Foreign Files Developer's Note
+
+
+ +
This document is Copyright by Apple Computer, Inc.
+ +
+ + diff --git a/library/FTN.e000023.htm b/library/FTN.e000023.htm new file mode 100644 index 0000000..bcf6471 --- /dev/null +++ b/library/FTN.e000023.htm @@ -0,0 +1,462 @@ + + + + + + +FTN.e000023 + + + +
+ +


+

+
+ +
+

Apple II FTN - AppleDouble File

+

Back to nulib.com library

+
+Apple II
+File Type Notes
+_____________________________________________________________________________
+                                                  Developer Technical Support
+
+File Type:         $E0 (224)
+Auxiliary Types:   $0002 & $0003
+
+Full Name:     AppleDouble Header File (Auxiliary Type $0002)
+               AppleDouble Data File (Auxiliary Type $0003)
+Short Name:    AppleDouble Header (Auxiliary Type $0002)
+               AppleDouble Data (Auxiliary Type $0003)
+
+Revised by:    Matt Deatherage                                  November 1990
+Written by:    Matt Deatherage                                     March 1989
+
+Files of these types and auxiliary types contain file data in AppleDouble
+format.
+Changes since March 1990:  Added information about AppleDouble 2.0.
+_____________________________________________________________________________
+
+AppleDouble is one of two standards (the other is AppleSingle) put forth by
+Apple Computer, Inc. for representing files on foreign file systems while
+preserving all attributes of the file's home system on file systems that do not
+support the same attributes.
+
+Experience indicated that a single format would be inadequate to cover all
+cases.  Two closely related formats, however, can serve most needs.  Although
+the primary impetus for developing these formats is storing extended files
+(files with both resource and data forks) on file systems that do not support
+the notion of two forks, the proposed formats are general enough that theycan be
+used to represent a file from any file system on any other file system.
+
+AppleDouble keeps the data fork as a separate file from the file attributes and
+the resource fork, and this Note describes this file format.  AppleSingle keeps
+all attributes and the contents of both forks in a single file in the foreign
+file system, and is described in the File Type Note for File Type$E0, Auxiliary
+Type $0001.
+
+AppleSingle is intended to be used primarily as a storage format, especially for
+cases where you must store an extended file on a foreign file system and later
+reconstruct the extended file.  AppleDouble is more appropriate for applications
+where the users of the foreign file system might want to modify the contents of
+the file.  Since most applications keep file data in the data fork, AppleDouble
+format saves the contents of the data fork in one file.All other file
+attributes, including the resource fork, are kept in a separate file.
+
+
+Reasons for Using AppleDouble
+
+There are several reasons for supporting an interchange format between file
+systems.  Perhaps the most germane is one of the least obvious:  handling
+extended files on foreign file systems which do not support extended files.
+
+For example, the ProDOS FST in GS/OS can create an extended file on a ProDOS
+disk.  However, ProDOS 8 is unable to operate on the file, since it sees itas
+having an unsupported storage type.  If a telecommunications program or other
+utility capable of transferring files is operating under ProDOS 8 andattempts to
+receive an extended file, it is unable to create the file.
+
+At this point, the application could use READ_BLOCK and WRITE_BLOCK commands,
+along with a knowledge of the ProDOS file system, to create the file on its own.
+However, this is strongly discouraged.  The ProDOS file system format for
+extended files is not documented and could change in the future.  In addition,
+the program could be running on a eight-bit system.  If the disk is only used on
+an eight-bit system, the extended files would not only be unwanted, but also
+unremovable without using the disk on an Apple IIGS or later system running
+GS/OS.
+
+However, if the application is aware of the AppleDouble format, it canquickly
+store an extended file in AppleDouble, leaving the conversion back to the
+extended file to GS/OS, or another operating system.  This is the recommended
+way for ProDOS 8 applications to create and handle extended files.  Useeither
+AppleSingle or AppleDouble.
+
+
+AppleDouble Format
+
+AppleDouble consists of two files, an AppleDouble Header File and an AppleDouble
+Data File.  The AppleDouble Header file contains a headerfollowed by data.  The
+header consists of several fixed fields and a list of entry descriptors, each
+pointing to an entry.  Apple defines these standardentries: Resource Fork, Real
+Name (name in the home file system), Comment, Icon and File Info.  Each entry is
+optional, so it may not appear in the file.  Wealso define the new entry Data
+Pathname, pointing to the pathname of the AppleDouble Data File.  The Header
+File has exactly the same format as an AppleSingle file, except it has no data
+fork entry.  The AppleDouble DataFile consists of just the data fork of the
+file, with no extra header at all.
+
+Note:  All numeric entries, including entries representing ProDOS data
+       structures (such as file type and auxiliary type) are Reverse
+       ordered.  This is provided so any host CPU can attempt to
+       interpret entries in the header without having to know the
+       standard byte-ordering of the home file system.  Therefore, in
+       this Note you see descriptive entries like "Rev. 4 Bytes."  This
+       serves as a reminder that all header fields are stored high byte
+       first, even though the notation Bytes does not imply any specific
+       ordering in other File Type Notes.
+
+Also note that ASCII strings are not stored in reverse order, just non-ASCII
+constants.
+
+The Header in the Header File:
+
+Magic Number         Rev. Long    The Magic Number field is modeled after
+                                  the feature in UNIX.  It is intended to be
+                                  used in whatever way the foreign file
+                                  system distinguishes a file as AppleDouble
+                                  format.  See the section "Identifying
+                                  AppleDouble Files."  The Magic Number for
+                                  AppleDouble format is $00051607, which is
+                                  stored reverse as $00 $05 $16 $07 (reverse
+                                  of normal 65816/6502 order).
+Version Number       Rev. Long    The version of AppleDouble format, in case
+                                  the format evolves (more fields may be
+                                  added to the header).  The version
+                                  described here is $00010000, stored
+                                  (reverse) as $00 $01 $00 $00.
+
+Home File System     16 Bytes     A fixed-length, 16-byte ASCII string not
+                                  preceded by a length byte, but possibly
+                                  padded with blanks.  Apple has defined
+                                  these values:
+                                  ProDOS    $50726F444F5320202020202020202020
+                                  Macintosh $4D6163696E746F736820202020202020
+                                  MS-DOS    $4D532D444F5320202020202020202020
+                                  Unix      $556E9878202020202020202020202020
+                                  VAX VMS   $56415820564D53202020202020202020
+                                  Apple welcomes suggestions for other file
+                                  systems that should be included in this
+                                  list.
+Number of entries    Rev. Word    Tells how many different entries are
+                                  included in the file.  This unsigned
+                                  reverse word may be zero.  If it is non-
+                                  zero, then that number of entry
+                                  descriptors immediately follows this
+                                  field.
+
+
+For Each Entry:
+
+Entry ID             Rev. Long    Identifies the entry.  Apple has defined
+                                  the following Entry IDs and their values:
+                                  1 = Data Fork
+                                  2 = Resource Fork
+                                  3 = Real Name (The file's name in the home
+                                      file system)
+                                  4 = Comment* (standard Macintosh comment)
+                                  5 = Icon, B&W* (standard Macintosh black
+                                      and white icon)
+                                  6 = Icon, Color* (reserved for Macintosh
+                                      color icon)
+                                  7 = File Info (file attributes,dates, etc.)
+                                  9 = Finder Info* (standard Macintosh
+                                      Finder Info)
+                                  Entry IDs marked with asterisks (*) are
+                                  not used for most files created under
+                                  ProDOS or GS/OS.  Furthermore, icon
+                                  entries probably do not appear in most
+                                  files since they are typically stored as a
+                                  bundle in the application file's resource
+                                  fork on the Macintosh.  Apple reserves the
+                                  range of Entry IDs from $0 to $7FFFFFFF
+                                  for future use.  The rest of the range is
+                                  available for other systems to define
+                                  their own entries.  Apple does not
+                                  arbitrate the use of the rest of the
+                                  range.
+                                  Descriptions of the standard entries are
+                                  given below.
+Offset               Rev. Long    An unsigned reverse long which indicates
+                                  the byte offset from the start of the file
+                                  to the start of the entry.
+Entry Length         Rev. Long    An unsigned reverse long which indicates
+                                  the length of the entry in bytes.  The
+                                  length may be zero.
+
+
+Standard Entries:
+
+The Real Name Entry:
+
+The Real Name entry indicates the file's original filename in the host file
+system.  This is not a Pascal or C string; it is just ASCII data.  The length is
+indicated by the Entry Length field for the Real Name entry.
+
+The File Info Entry:
+
+The File Info entry (Entry ID = 7) is different for each home file system. For
+ProDOS files, the entry is 16 bytes long and consists of the creationdate and
+time and the modification date and time in ProDOS 8 (ProDOS 16/class zero GS/OS)
+form, the access word, a two-byte file type and four-byte auxiliary type.  This
+is detailed in standard format below, along with defined FileInfo entries for
+some other file systems.
+
+ProDOS:
+
+Create Date         Rev. 2 Bytes  Creation date packed into standard
+                                  ProDOS 8 format.
+Create Time         Rev. 2 Bytes  Creation time packed into standard
+                                  ProDOS 8 format.
+Modification Date   Rev. 2 Bytes  Modification date packed into
+                                  standard ProDOS 8 format.
+Modification Time   Rev. 2 Bytes  Modification time packed into
+                                  standard ProDOS 8 format.
+Access              Rev. Word     The file's access.  This may be used
+                                  directly in ProDOS 16 or GS/OS calls; only
+                                  the low byte is significant to ProDOS 8.
+File Type           Rev. Word     The file type of the original file.  Only
+                                  the low byte is significant to ProDOS 8.
+Auxiliary Type      Rev. Long     The auxiliary type of the original file.
+                                  Only the low word is significant to ProDOS
+                                  8.
+
+Note:  Although the ProDOS Access field, File Type and Auxiliary Type are
+       the same length as found in ProDOS 16 and GS/OS structures, the
+       Create and Modification Dates and Times are stored in two-byte
+       (albeit byte-reversed) ProDOS 8 format, not eight-byte Apple IIGS
+       format.
+
+Macintosh:
+
+Create Date         Rev. Long     Unsigned number of seconds between
+                                  January 1, 1904, and the creation time of
+                                  this file.
+Modification Date   Rev. Long     Unsigned number of seconds between
+                                  January 1, 1904, and the last modification
+                                  of this file.
+Last Backup Date    Rev. Long     Unsigned number of seconds between
+                                  January 1, 1904, and the last backup time
+                                  of this file.
+Attributes          Rev. Long     32 boolean flags.  Once the bytes are
+                                  unreversed, bit zero is the locked bit and
+                                  bit one is the protected bit.
+
+MS-DOS:
+
+Modification Date    Rev. 4 Bytes  MS-DOS format modification date.
+Attributes           Rev. 2 Bytes  MS-DOS attributes.
+
+Unix:
+
+Create Date/Time       Rev. 4 Bytes  Unix creation date and time.
+Last Use Date/Time     Rev. 4 Bytes  Unix time for the last
+                                     time this file was used.
+Last Mod. Date/Time    Rev. 4 Bytes  Unix time for the last
+                                     time this file was modified.
+
+
+The Finder Info Entry:
+
+The Finder Info entry (Entry ID = 9) is for files where the host file system is
+Macintosh.  It consists of 16 bytes of Finder Info followed by 16 bytes of
+Extended Finder Info.  These are the fields ioFlFndrInfo followed by
+ioFlXFndrInfo, as described in Inside Macintosh, Volume IV-183.  Newlycreated
+files have zeroes in all Finder Info subfields.  If you are creating an
+AppleDouble file whose home system is Macintosh, you may zero all unknown
+fields, but you may want to set the fdType and fdCreator subfields.
+
+
+The Data Pathname Entry:
+
+The Data Pathname entry (Entry ID = 100) is defined for the first time inthis
+Note.  It consists of a class one GS/OS input string noting the pathname of the
+AppleDouble Data File as originally created:
+
+Path Length    Rev. Word    The length of the pathname.
+Pathname       Bytes        ASCII pathname of the AppleDouble Data File
+when created.
+
+ For strategies on using this segment (or not using it) to find theAppleDouble
+Data File, see the section "Finding the AppleDouble Data File."
+
+
+The Entries in the Header File:
+
+The entries themselves follow the header field and the entry descriptors.The
+actual data representing each entry must be in a single, contiguous block. The
+offset field in that entry's descriptor points to it.  The entries could appear
+in any order, but since the data fork is the entry that is most commonly
+extended, Apple strongly recommends that the data fork always bekept last in the
+file to facilitate its extension.  Apple also recommends that those entries that
+are most often read, such as Real Name, File Info (and Finder Info if present)
+be kept as close as possible to the header tomaximize the probability that a
+read of the first few blocks of the file retrieves these entries.
+
+It is possible to have holes in the file (unused space between entries).  To
+find the holes, you must take the list of entry descriptors and sort theminto
+increasing offset order.  If the offset field of an entry is greater than the
+offset plus the length of the previous entry (sorted), then a hole exists
+between the entries.  You can make use of such holes; for example, if afile's
+comment is ten bytes long, you could create a hole of 190 bytes after the
+comment field to easily allow for the comment to later expand to its maximum
+length of 200 bytes.  Because an AppleDouble file may contain holes, you must
+find each entry by getting its offset from its entry descriptor, not by assuming
+that it begins after the previous entry.
+
+Byte ordering in file header fields follows 68000 convention, and each header
+field has been so noted by the Reverse operator.
+
+
+The AppleDouble Data File
+
+The AppleDouble Data File is simply the data fork of the original file contained
+in a file of its own.  You may create it with a File Type and Auxiliary Type
+assignment best suited to it, if desired.  For example, if the program creating
+the AppleDouble Data File knows that the data fork contains strictly ASCII text,
+it could create the file with File Type $04 (Text File) so that other
+applications can deal with it accordingly.
+
+If the creating program wishes to make no assumptions about the content ofthe
+data fork, it is encouraged to create the AppleDouble Data File with filetype
+$E0 and auxiliary type $0003.  This identifies the file as an AppleDoubleData
+File.
+
+
+Identifying AppleDouble Files
+
+As this is an interchange format, from a ProDOS directory entry there is no way
+to guarantee which files are AppleDouble files.  Apple has allocated File Type
+$E0, Auxiliary Type $0002 for files which are AppleDouble Header Files, and File
+Type $E0, Auxiliary Type $0003 for files which are AppleDouble Data Files.  We
+strongly encourage ProDOS 8 and GS/OS applications to use these file type and
+auxiliary type assignments when creating AppleDouble files.
+
+AppleDouble files which do not have file type $E0 and auxiliary type $0002 or
+$0003 can most easily be identified by opening them and attempting to interpret
+them.  If it is not an AppleDouble Header File, the Magic Number is not
+contained in the first four bytes of the file.  The chances that the file would
+begin with those four bytes and not be an AppleDouble Header File, on a purely
+random basis, are 4,294,967,295 to 1.  The chances that both the Magic Number
+and the Version bytes would be the same in a non-AppleSingle file are roughly
+1.8 x 10^19 to 1.
+
+
+Finding the AppleDouble Data File
+
+Since the AppleDouble Data File can be stored anywhere, with any file typeand
+auxiliary type, a program may have to make an effort to find it.  Werecommend
+the following steps:
+
+1.  If the Data Pathname segment exists, use that pathname.  If the
+    path specified in the segment does not exist, extract the file
+    name from the end of the pathname and look in the current
+    directory for that file name.
+2.  If Step 1 does not find the file (or if the Data Pathname segment
+    does not exist), perform the appropriate Home File System
+    algorithm (described below) to generate the name of the
+    AppleDouble Data File from the AppleDouble Header File.
+3.  If none of the file names generated in Step 2 are found, ask the
+    user where the AppleDouble Data File is located.
+
+
+Filename Conventions:
+
+Apple proposes the following standard for identifying AppleDouble Header File
+names and AppleDouble Data File names from the file's real name.
+
+ProDOS:
+
+To generate the AppleDouble Data File name, use character substitution or
+deletion to remove illegal characters, and use truncation if necessary to
+reduce the length of the name to two characters less than the maximum file
+name length.  This would be a maximum of 13, since the maximum file name
+length is 15.
+
+To generate the AppleDouble Header File name, prefix the AppleDouble DataFile
+name with the characters "R." (uppercase R period).
+
+For example, the file name "This is a Foo File" could translate to an
+AppleDouble Data File Name of "THIS.IS.A.FOO."  The AppleDouble Header File
+name would then be "R.THIS.IS.A.FOO."
+
+Unix:
+
+To generate the AppleDouble Data File name, use character substitution to
+replace any illegal characters with an underscore (_).  Since different Unix
+systems have different requirements on maximum file name length, do not
+explicitly truncate the name to a specific length.  Rather, allow the
+truncation to be done by the Unix functions create(), open(), etc.
+
+To generate the AppleDouble Header File name, A/UX (Apple's implementation of
+Unix for Macintosh computers) prefixes a percent sign (%) to the AppleDouble
+Data File name.  If necessary, truncate the last character to keep the file name
+within the legal length range.  Other Unix systems may prefix adirectory name
+(e.g., ".AppleDouble/") to the AppleDouble Data File name to create the name of
+the AppleDouble Header File.  In this scheme, all AppleDouble Header Files
+corresponding to AppleDouble Data files are kept together in a single
+subdirectory.
+
+MS-DOS:
+
+To generate the AppleDouble Data File name, use character substitution or
+deletion to remove illegal characters, and use truncation if necessary to
+reduce the length of the name to eight characters.  Then add the MS-DOS
+extension that is most appropriate to the file (such as "TXT" for a pure text
+file).
+
+To generate the AppleDouble Header File name, add the extension ".ADF" to the
+eight-character file name.
+
+In any instance, most programs probably wish to display the names being used
+for both AppleDouble files, so that the user may keep track of them on disk.
+
+AppleDouble name derivations will be defined for all other file systems of
+interest.  This allows applications running on the foreign file system (and
+users as well) to see easily which files are AppleDouble pairs.Knowledgeable
+users, if they know the derivation, could rename or move the files so as to
+preserve the connection between the two.  However, there is no guaranteed way to
+prevent one file of the pair from being inconsistently renamed, moved, or
+deleted.
+
+
+About AppleDouble 2.0
+
+AppleDouble 2.0 is a revision to the original AppleDouble specification
+described in this Note.  AppleDouble 2.0 comes closer to the ideal of an
+interchange format by allowing file information for multiple file systems in the
+same AppleDouble file.
+
+AppleDouble 2.0 basically replaces the File Info entry (ID = 7) with a File
+Dates entry (ID = 8) and one or more host file system entries, such as a
+Macintosh File Info entry (ID = 10), a ProDOS File Info entry (ID = 11), or an
+MS-DOS File Info entry (ID = 12).  Information on these entries and AppleDouble
+2.0 can be found in the AppleSingle/AppleDouble Formats for Foreign Files
+Developer's Note, available from APDA, AppleLink, and the Developer CD series.
+
+
+Further Reference
+_____________________________________________________________________________
+  o  Inside Macintosh, Volume IV
+  o  ProDOS 8 Technical Reference Manual
+  o  GS/OS Reference
+  o  AppleSingle/AppleDouble Formats for Foreign Files Developer's Note
+
+
+ +
This document is Copyright by Apple Computer, Inc.
+ +
+ + diff --git a/library/FTN.e00005.htm b/library/FTN.e00005.htm new file mode 100644 index 0000000..6ffe007 --- /dev/null +++ b/library/FTN.e00005.htm @@ -0,0 +1,255 @@ + + + + + + +FTN.e00005 + + + +
+ +


+

+
+ +
+

Apple II FTN - DiskCopy disk image

+

Back to nulib.com library

+
+Apple II
+File Type Notes
+_____________________________________________________________________________
+                                                  Developer Technical Support
+
+File Type:       $E0 (224)
+Auxiliary Type:  $0005
+
+Full Name:       DiskCopy disk image
+Short Name:      DiskCopy disk image
+
+Written by:      Matt Deatherage, Dave Lyons & Steve Christensen     May 1992
+
+Files of this type and auxiliary type contain disk images from Apple's
+DiskCopy program on the Macintosh.
+_____________________________________________________________________________
+
+DiskCopy is a program written by Steve Christensen of Apple Computer, Inc.,
+for internal use in duplicating and distributing 3.5" floppy disks.  Because
+of its utility in distributing disk images on the Macintosh, DiskCopy is used
+in several Apple developer products even though DiskCopy is not an official
+Apple product and is not supported as such.
+
+Since the monthly Developer CD Series discs contain many DiskCopy disk images,
+and since the AppleShare and HFS FSTs in System Software 6.0 and later
+automatically translate DiskCopy files (HFS file type dImg and creator dCpy)
+to Apple II file type $E0 and auxiliary type $0005, the format is provided
+here for your utility use only.  Apple does not guarantee that files not
+generated by DiskCopy will work with DiskCopy.
+
+
+DEFINITIONS
+
+DiskCopy uses a simple checksum algorithm to help insure data integrity for
+archived disk images. The algorithm for generating the 32-bit checksum is as
+follows:
+
+     Initialize checksum to zero
+     For each data REVERSE WORD:
+          Add the data REVERSE WORD to the checksum
+          Rotate the 32-bit checksum right one bit (wrapping bit 0 to bit 31)
+
+The following 65816 assembly language routine calculates a DiskCopy checksum.
+It's not a speedy operation--it takes about 12 seconds to calculate the
+checksum on an 800K disk image.  Anyone finding an assembly routine that can
+perform this task in under 5 seconds may apply for their IIgs Certificate of
+Deityship, as documented in the File Type Note for file type $B6.
+
+(Oh, by the way, any entries have to be under 1K in size--the following
+routine is 88 bytes.  So don't think unwinding loops is your ticket to fame
+and fortune.)
+
+****************************************************************************
+*
+* Compute checksum for DiskCopy data
+*
+* v1.2  by David A. Lyons, 18-May-92
+*
+* MPW IIgs assembly format
+*
+* Inputs on stack:
+*    Push pointer to data (long)
+*    Push size of data (long)  (Must be even!)
+*    JSL CalcChecksum
+*    STA TheChecksum+2
+*    STX TheChecksum
+*
+* Output:
+*    Checksum in A and X (bytes +0 and +1 in X, bytes +2 and +3 in A)
+*    (The inputs have been removed from the stack)
+*
+****************************************************************************
+CalcChecksum        PROC
+                    phd                 ;save caller's direct page reg
+                    lda #0
+                    pha
+                    pha                 ;push initial checksum value (zero)
+                    tsc
+                    tcd
+
+checksum            equ 1
+oldD                equ checksum+4
+theRTL              equ oldD+2
+dataSize            equ theRTL+3
+dataPtr             equ dataSize+4
+
+*** Set dataSize to -(dataSize/2)-1 so we can count up by one
+*** (instead of down by two) to see when we're done
+                    lda <dataSize+2
+                    lsr a
+                    eor #$ffff
+                    sta <dataSize+2
+                    lda <dataSize
+                    ror a
+                    eor #$ffff
+                    sta <dataSize
+
+                    ldy #0
+nextWord            inc <dataSize
+                    bne moreData
+                    inc <dataSize+2
+                    beq noMoreData
+moreData
+
+*** Get next 16-bit word from the data buffer
+                    lda [<dataPtr],y
+                    xba                 ;swap to 65816 byte order
+
+*** Add the data word to the checksum
+                    clc
+                    adc <checksum
+                    sta <checksum
+                    bcc noCksumRoll
+                    inc <checksum+2
+noCksumRoll
+*** Rotate the 32-bit checksum right one bit, wrapping bit 0 into bit 31
+                    lda <checksum+2
+                    lsr a
+                    ror <checksum
+                    bcc bit0was0
+                    ora #$8000          ;if we rotated a 1 out of bit 0,
+bit0was0            sta <checksum+2     ;  then set bit 31
+
+*** Advance to the next word and go back for more
+                    iny
+                    iny
+                    bne nextWord        ;go back for more data
+                    inc <dataPtr+2
+                    bra nextWord        ;go back for next bank of data
+
+noMoreData          pla
+                    xba
+                    tay
+                    pla
+                    xba
+                    tax                 ;pull checksum into YX (put in 68000
+order)
+
+                    pld                 ;restore caller's direct page reg
+
+                    lda 2,s
+                    sta 2+8,s
+                    lda 1,s
+                    sta 1+8,s
+                    pla
+                    pla
+                    pla
+                    pla                 ;discard input values
+
+                    tya
+                    rtl
+
+                    EndP
+
+                    END
+
+The following definition is used in this document in addition to those defined
+for all Apple II file types:
+
+Checksum   A 32-byte quantity calculated using the previously-defined
+           algorithm.  When these are contained in the file, they are in
+           REVERSE order.
+
+
+FILE STRUCTURE
+
+All of the information for a DiskCopy disk image is in the data fork.  The
+resource fork usually contains Macintosh resources (in Macintosh resource fork
+format), including vers resources listing the checksums.  This allows
+Macintosh users to use the Macintosh Finder's "Get Info..." function to
+quickly examine the checksums.
+The File Format
+
+Because this is a native Macintosh file format, all the multi-byte constants
+are stored in Reverse order.
+
+diskName       (+000)  64 Bytes  A Pascal String containing the name of the
+                                 disk.  This field takes 64 bytes
+                                 regardless of the length of the String.
+dataSize       (+064) Rev. Long  The number of bytes (not blocks) of user
+                                 data.  User data is the 512 bytes of each
+                                 block that a normal block-reading command
+                                 returns.
+tagSize        (+068) Rev. Long  The number of bytes of tag data.  Tag data
+                                 is the extra 12 bytes of "scavenger"
+                                 information present on 400K and 800K
+                                 Macintosh disks.  Apple II operating
+                                 systems always leave these bytes zeroed,
+                                 and they're not present on 720K or 1440K
+                                 disks.  If there are no tag bytes, this
+                                 field will be zero.
+dataChecksum   (+072)  Checksum  Checksum of all the user data on the disk.
+                                 The checksum algorithm is called for the
+                                 entire disk, not on a block-by-block or
+                                 sector-by-sector basis.  This is in
+                                 Reverse order (most significant byte
+                                 first).
+tagChecksum    (+076)  Checksum  Checksum of all the tag data on the disk.
+                                 If there is no tag data, this should be
+                                 zero.  This is in Reverse order (most
+                                 significant byte first).
+diskFormat     (+080)      Byte  0 = 400K
+                                 1 = 800K
+                                 2 = 720K
+                                 3 = 1440K  (all other values are reserved)
+formatByte     (+081)      Byte  $12 = 400K
+                                 $22 = >400K Macintosh (DiskCopy uses this
+                                       value for all Apple II disks not
+                                       800K in size, and even for some of
+                                       those)
+                                 $24 = 800K Apple II disk
+private        (+082) Rev. Word  Must be $0100.  If this field is not
+                                 $0100, the file may be in a different
+                                 format.
+userData       (+084)            dataSize Bytes
+                                 The data blocks for the disk.  These are
+                                 in order from block zero through the end
+                                 of the disk.
+tagData    (+xxx) tagSize Bytes  The tag data for this disk, starting with
+                                 the tag data for the first block and
+                                 proceeding in order.  This field is not
+                                 present for 720K and 1440K disks, but it
+                                 is present for all other formats even if
+                                 all the data is zeroes.
+Further Reference
+_____________________________________________________________________________
+
+   o   GS/OS Reference
+
+ +
This document is Copyright by Apple Computer, Inc.
+ +
+ + diff --git a/library/FTN.e08000.htm b/library/FTN.e08000.htm new file mode 100644 index 0000000..bc93f6d --- /dev/null +++ b/library/FTN.e08000.htm @@ -0,0 +1,537 @@ + + + + + + +FTN.e08000 + + + +
+ +


+

+
+ +
+

Apple II FTN - Binary II File

+

Back to nulib.com library

+
+Apple II
+File Type Notes
+_____________________________________________________________________________
+                                                  Developer Technical Support
+
+
+File Type:         $E0 (224)
+Auxiliary Type:    $8000
+
+Full Name:     Binary II File
+Short Name:    Binary II File
+
+Written by:    Matt Deatherage                                      July 1989
+
+Files of this type and auxiliary type contain other files with their 
+attributes encoded in Binary II format.
+_____________________________________________________________________________
+
+Binary II is a widely used and accepted standard for keeping file attributes 
+with files as they are transferred, usually by modem or other form of 
+telecommunication.  Files that are known Binary II files should be written to 
+disk with file type $E0 and auxiliary type $8000 as a clear indication to 
+other programs that the file contains files with Binary II specifications.
+
+Binary II was developed by Gary B. Little, author of the Point-To-Point 
+communication's product and author of several Apple II reference books.  He is 
+also Apple's Product Manager for third-party Development Tools and Languages.  
+Gary welcomes your comments and suggestions on the Binary II standard at the 
+following address:
+
+                Gary B. Little
+                3304 Plateau Drive
+                Belmont, CA  94002
+
+                AppleLink:                      LITTLE
+                AppleLink--Personal Edition:    GaryLittle
+                CompuServe:                     70135,1007
+                GEnie:                          GARY.LITTLE
+
+
+Why Binary II?
+
+Transferring Apple II files in binary form to commercial information services 
+and bulletin boards (referred to in this Note as "hosts") can be, to put it 
+mildly, a frustrating exercise.  Although most hosts are able to receive a 
+file's data in binary form (using protocols such as XMODEM), they don't 
+receive the file's all-important attribute bytes.  All the common Apple II 
+file system, notably the ProDOS file system, store the attributes inside the 
+disk directory, not inside the file itself.
+
+The ProDOS attributes are the access code, file type code, auxiliary type 
+code, storage type code, date of creation and last modification, time of 
+creation and last modification, the file size, and the name of the file 
+itself.  Under GS/OS, the same parameters exist for other file systems as well 
+as file system-specific information and two-forked file information.  It is 
+usually not possible to use a ProDOS file's data without knowing the file's 
+attributes (particularly the file type, auxiliary type, and size).  Therefore, 
+ProDOS files uploaded in binary format to a host are useless to those who 
+download them.  The same is true for DOS 3.3 and Pascal files.
+
+Many Apple II communication programs use special protocols for transferring 
+file attributes during a binary file transfer, but none of these protocols 
+have been implemented by hosts.  These programs are only useful for exchanging 
+files with another Apple II running the same program.
+
+Without a standard like Binary II, the only acceptable way to transfer an 
+Apple II file to a host is to convert it into ASCII text before sending it.  
+Such a text file would contain a listing of an AppleSoft program, or a series 
+of Apple II monitor commands (e.g., 300:A4 32).  Someone downloading a file 
+can convert it to binary form using the AppleSoft EXEC command.
+
+The main disadvantage of this technique is that the text version of the file 
+is over three times the size of the original binary file, making it expensive 
+(in terms of time and money) to upload and download.  It is also awkward, and 
+sometimes impossible, to perform the binary-to-text or text-to-binary 
+conversion.
+
+The solution to the problem is to upload an encoded binary file which contains 
+not just the file's data, but the file's attributes as well.  Someone 
+downloading such a file can then use a conversion program to strip the 
+attributes from the file and create a file with the required attributes.
+
+This Note describes such a format:  Binary II.  The description of the format 
+is detailed for the purpose of allowing software developers to implement it in 
+Apple II communication programs.
+
+
+What Binary II is Not
+
+Binary II is not an archival or compression standard.  It is designed to be a 
+simple method to keep the attributes normally in a disk file's directory entry 
+with the file as it is transferred.  Although multiple files may be placed 
+together with Binary II, this is a matter of convenience for telecommunication 
+programs.
+
+A true archival standard must be designed as such, with the capability to 
+manipulate files within the archive as well as linking them together 
+(compressed or uncompressed) for transfer.  NuFX (documented in Apple II File 
+Type Note for File Type $E0, Auxiliary Type $8002) is a good example of a 
+robust, full-featured Apple II archival standard.
+
+Binary II is primarily designed to be added to and subtracted from files "on-
+the-fly" by telecommunication programs.  Binary II files should only be found 
+on disks when they are transferred by a telecommunication program that does 
+not have Binary II capabilities, in which case a separate utility (such as 
+Binary Library Utility by Floyd Zink, Jr.) must be used to extract the files.  
+Telecommunication programs should be able to transfer files without Binary II 
+processing, however, they should support Binary II processing as a default.
+
+
+The Binary II File Format
+
+The Binary II form of a standard file consists of a 128-byte file information 
+header followed by the file's data.  The data portion of the file is padded 
+with nulls ($00 bytes), if necessary, to ensure the data length is an even 
+multiple of 128 bytes.
+
+The file information header contains four ID bytes, the attributes of the file 
+(in ProDOS 8 form), and some control information.
+
+The structure of the header is as follows:
+
++000    ID Bytes   3 Bytes  These three bytes are always $0A $47 $4C 
+                            for identification purposes, so programs 
+                            may recognize Binary II files as they are 
+                            received.
++003    Access Code Byte    ProDOS 8 access byte.
++004    File Type  Byte     ProDOS 8 file type.
++005    Aux Type   Word     ProDOS 8 auxiliary type.
++007    Storage Type Byte   ProDOS 8 storage type value.
++008    File Size   Word    The size of the file in 512-byte blocks.
++010    Mod. Date   2 Bytes Date of modification, in ProDOS 8 
+                            compressed format.
++012    Mod. Time   2 Bytes Time of modification, in ProDOS 8 
+                            compressed format.
++014    Create Date 2 Bytes Date of creation, in ProDOS 8 
+                            compressed format.
++016    Create Time 2 Bytes Time of creation, in ProDOS 8 
+                            compressed format.
++018    ID Byte     Byte    A fourth ID byte.  This must always be 
+                            $02.
++019    Reserved    Byte    Reserved, must be set to zero.
++020    EOF         3 Bytes The end-of-file value for the file (low 
+                            byte first).
++023    File Name   String  Pascal string containing the ASCII 
+                            filename or partial pathname of this file 
+                            in ProDOS 8 format.  The string cannot be 
+                            longer than 64 characters.
+
+If the File Name String is a filename and not a partial pathname, then the 
+following optional parameter may be supplied:
+
++039    Native Name String  Pascal string containing the 
+                            ASCII value of the native filename.  This 
+                            string may not be longer than 48 
+                            characters, and will not be present if the 
+                            length byte of File Name (+023) is larger 
+                            than 15 ($0F).  If this field is 
+                            specified, the File Name field must 
+                            contain a filename, not a partial 
+                            pathname.
+
++088    Reserved   21 Bytes Reserved.  These bytes must be set to zero 
+                            for future compatibility.
+
++109    GAux Type  Word     The high word of the file's GS/OS 
+                            auxiliary type.
++111    GAccess    Byte     The high byte of the file's GS/OS access 
+                            word.
++112    GFile Type Byte     The high byte of the file's GS/OS 
+                            file type.
++113    GStorage   Byte     The high byte of the file's GS/OS storage 
+                            type.
++114    GFile Size Word     The high word of the GS/OS file's 
+                            size in 512-byte blocks.
++116    GEOF       Byte     The high byte of the file's GS/OS EOF 
+                            value.
++117    Disk Space Long     The number of 512-byte disk blocks 
+                            the files inside the BinaryÊII file will 
+                            occupy after they've been removed from the 
+                            BinaryÊII file.  (The format of a Binary 
+                            II file containing multiple files is 
+                            described later in this Note.)  If the 
+                            number is zero, the creator of the Binary 
+                            II file didn't bother to calculate the 
+                            space needed.  This value must be placed 
+                            in the file information header for the 
+                            first file inside the Binary II file; it 
+                            can be set to zero in subsequent headers.  
+                            A downloading program can inspect Disk 
+                            Space and abort the transfer immediately 
+                            if there isn't enough free space on the 
+                            disk.
++121    OS Type    Byte     This value indicates the native operating 
+                            system of the file:
+                            $00    ProDOS or SOS
+                            $01    DOS 3.3
+                            $02    Reserved
+                            $03     DOS 3.2 or DOS 3.1
+                            $04    Apple II Pascal
+                            $05    Macintosh MFS
+                            $06    Macintosh HFS
+                            $07    Lisa Filing System
+                            $08    Apple CP/M
+                            $09    Reserved (returned by the GS/OS 
+                                   Character FST)
+                            $0A    MS-DOS
+                            $0B    High Sierra (CD-ROM)
+                            $0C    ISO 9660 (CD-ROM)
+                            $0D    AppleShare
+                            Note this list is slightly different (in 
+                            the first three entries) from the standard 
+                            GS/OS file system ID list.  A GS/OS 
+                            communication program should not place a 
+                            zero in this field unless the file's 
+                            native file system truly is ProDOS.  The 
+                            file's native file system is returned in 
+                            the file_sys_id parameter from the 
+                            GetDirEntry call.
++122    Native File Type
+                   Word     This has meaning only if OS Type is non-
+                            zero.  If so, it is set to the actual file 
+                            type code assigned to the file by it's 
+                            native operating system.  (Some operating 
+                            systems, such as MS-DOS and CP/M, do not 
+                            use file type codes, however.)  Contrast 
+                            this with the File Type at +004, which is 
+                            the closest equivalent ProDOS file type.  
+                            The Native File Type is needed to 
+                            distinguish files which have the same 
+                            ProDOS file type, but which may have 
+                            different file types in their native 
+                            operating system.  Note that if the file 
+                            type code is only one byte long (the usual 
+                            case), the high-order byte of Native File 
+                            Type is set to zero.
++124    Phantom File Flag
+                   Byte     This byte indicates whether a receiver of 
+                            the Binary II file should save the file 
+                            which follows (flag is zero) or ignore it 
+                            (flag is non-zero).  It is anticipated 
+                            that some communication programs will use 
+                            phantom files to pass non-essential 
+                            explanatory notes or encoded information 
+                            which would be understood only by a 
+                            receiver using the same communication 
+                            program.  Such programs must not rely on 
+                            receiving a phantom file, however, since 
+                            this would mean they couldn't handle 
+                            Binary II files created by other 
+                            communication programs.  Phantom Files may 
+                            also be used to pass extended file 
+                            attributes when available.
+
+                            The first two bytes in a phantom file must 
+                            contain an ID code unique to the 
+                            communication program, or a universal 
+                            identifier concerning the contents of the 
+                            phantom file.  Developers must obtain ID 
+                            codes from Gary Little to ensure 
+                            uniqueness (see the beginning of this Note 
+                            for his address).  Here is a current list 
+                            of approved ID codes for phantom files 
+                            used by Apple II communication programs:
+                            $00 $00    ASCII text terminated with a 
+                                       zero byte.
+                            $00 $01    Point-to-Point
+                            $00 $02    Tele-Master Communications 
+                                       System
+                            $00 $03    ProTERM
+                            $00 $04    Modem MGR
+                            $00 $05    CommWorks
+                            $00 $06    MouseTalk
+                            $01 $00    Option_list data (see later in 
+                                       this Note).
+                            The ID bytes are the first two bytes of 
+                            the phantom file.
++125    Data Flags          Bit 7:    1 = file is compressed
+                  Flag Byte Bit 6:    1 = file is encrypted
+                            Bits 5-1: Reserved
+                            Bit 0:    1 = file is sparse
+                            A Binary II downloading program can 
+                            examine this byte and warn the user that 
+                            the file must be expanded, decrypted or 
+                            unpacked.  The person uploading a Binary 
+                            II file may use any convenient method for 
+                            compressing, encrypting, or packing the 
+                            file but is responsible for providing 
+                            instructions on how to restore the file to 
+                            its original state.
++126    Version    Byte     This release of Binary II has a version 
+                            number of $01.
++127    Number of Files to Follow
+                   Byte     An appealing feature of Binary II is that 
+                            a single Binary II file can hold multiple 
+                            disk files, making it easy to keep a group 
+                            of related files "glued" together when 
+                            they're sent to a host.  This byte 
+                            contains the number of files in this 
+                            Binary II file that are behind it.  If 
+                            this is the first file in a Binary II file 
+                            containing three disk files, this byte 
+                            would be $02.  The second disk file in the 
+                            same Binary II file would have a value of 
+                            $01 in this parameter, and the last would 
+                            have value $00.  This count tells the 
+                            Binary II downloading program how many 
+                            files are remaining.  If any phantom files 
+                            are included, they must be included in 
+                            this count.
+
+
+Filenames and Partial Pathnames
+
+You can put a standard ProDOS filename or a partial pathname in the file 
+information header (but never a complete pathname).  Don't use a partial 
+pathname unless you've included, earlier in the Binary II file, file 
+information headers for each of the directories referred to in the partial 
+pathname.  Such a header must have its "end of file position" bytes set to 
+zero, and no data blocks for the subdirectory file must follow it.
+
+For example, if you want to send a file whose partial pathname is 
+HELP/GS/READ.ME, first send a file information header defining the HELP/ 
+subdirectory, then one defining the HELP/GS/ subdirectory.  If you don't, 
+someone downloading the Binary II file won't be able to convert it because the 
+necessary subdirectories will not exist.
+
+Note:  GS/OS communication programs must use the slash (/) as the 
+       pathname's separator in any partial pathname it puts in the 
+       header.  Since GS/OS's standard separator is the colon (:), a 
+       conversion may be necessary.
+
+
+Filename Convention
+
+Whenever a file is sent to a host, the host asks the sender to provide a name 
+for it.  If it's a file in Binary II form, the name provided should end in 
+.BNY so its special form will be apparent to anyone viewing a list of 
+filenames.  If the file is compacted (using the public-domain Squeeze 
+algorithm) before being converted to Binary II form, use a .BQY suffix 
+instead.  If the file is a NuFX archive, use the suffix .BXY.
+
+
+Identifying Binary II Files
+
+You can determine, while transferring, if a file is in Binary II form by 
+examining the ID bytes at offsets +000, +001, +002 and +018 from the beginning 
+of the file.  They must be $0A, $47, $4C and $02, respectively.
+
+Once Binary II files are identified, you can use the data in the file 
+information header to create and open a ProDOS file with the correct name and 
+attributes, transfer the file data in the Binary II file to the ProDOS file, 
+set the ProDOS file size, then close the ProDOS file.  You would repeat this 
+for each file contained inside the Binary II file.
+
+Note:  The number of 128-byte blocks following the file information 
+       header must be derived from the EOF attribute for the file.  
+       Calculate the number by dividing the EOF by 128 and adding one to 
+       the result if EOF is not 0 or an exact multiple of 128.  However, 
+       if the file information header defines a subdirectory (the file 
+       type is $0F), simple create the subdirectory file.  Do not open it 
+       and do not try to set its size.
+
+Ideally, all this conversion work will be done automatically by a 
+communication program during file transfer.  If not, a separate conversion 
+program (such as the previously mentioned Binary Library Utility, or BLU) must 
+be used to do this for you.
+
+
+Option_List Phantom Files
+
+GS/OS will return, when asked, an option_list for files on many file calls.  
+The option_list consists of a Word buffer length (which must be at least $2E), 
+followed by a Word number of bytes GS/OS put in the buffer, a Word GS/OS file 
+system identification, and the given number of bytes of FST-specific 
+information (minus two; the count GS/OS returns includes the file system 
+identifier).
+
+Option_list values are FST specific and contain values important to the native 
+file system but not important to GS/OS.  For AppleShare, the option_list 
+contains Finder Information, parent directory identification, and access 
+privileges.  This information should be transferred with files.
+
+Binary II uses a phantom file with identifier $01 $00 to indicate an 
+option_list.  When this phantom file is seen, the contents should be used as 
+the option_list for the file that immediately follows this file in the 
+Binary II file.  The other attributes of the phantom file must be set to the 
+same values as those for the file immediately following (the file for which 
+the phantom file contains the option_list).  The EOF for the phantom file must 
+be the size of the option_list + 2, and the file size must be adjusted 
+accordingly to account for the phantom file ID bytes.
+
+When receiving a Binary II file, the contents of this phantom file should be 
+used as option_list input on a GS/OS SetFileInfo call.
+
+If the GS/OS option_list returns a total of two bytes (just the file_sys_ID), 
+there is no FST-specific information, and the option_list phantom file may 
+safely be omitted.
+
+The format of the option_list phantom file is as follows:
+
++000    Phantom ID 2 Bytes  The identifying bytes $01 $00.
++002    List Size  Word     The length of the bytes in the 
+                            option_list, starting with the file system 
+                            ID (the next word).
++004    FileSysID  Word     A GS/OS (not Binary II) file_sys_ID for 
+                            the volume on which the file was stored.
++006    List Bytes Bytes    The bytes of the option list.  
+                            There should be (List Size) of them, 
+                            counting the previous word (FileSysID).
+
+
+Extended File Considerations
+
+Extended files contain two logical segments:  a data fork and a resource fork.  
+These files can be created and manipulated by GS/OS, but not by ProDOS 8 or 
+any other Apple II operating system.
+
+When a GS/OS-based communication program sends an extended file, it must send 
+it in the AppleSingle file format, preceded by a Binary II file information 
+header.  (Such a program could easily convert an extended file to AppleSingle 
+format on the fly.)  The Binary II header must contain the attributes of the 
+AppleSingle file (including a file type of $E0 and an auxiliary type of $0001) 
+and the "storage type code" field must be $01.  (The EOF positions for the 
+data fork and resource fork of the extended file appear in an entry in the 
+AppleSingle file header, not in the Binary II header.)
+
+The AppleSingle format is described in Apple II File Type Note for File Type 
+$E0, Auxiliary Type $0001.
+
+A GS/OS-based communication program that receives an AppleSingle file can 
+easily convert it on the fly to the extended file it defines.  ProDOS 8-based 
+communication programs can only save the AppleSingle file to disk because it's 
+not possible (nor is it encouraged to attempt) to create extended files with 
+ProDOS 8 (without using block-level calls); a GS/OS based utility program is 
+needed to convert the AppleSingle file to its extended form.
+
+
+DOS 3.3 Considerations
+
+With a little extra effort, you can also convert DOS 3.3 files to Binary II 
+form.  This involves translating the DOS 3.3 file attributes to the 
+corresponding ProDOS attributes so that you can build a proper file 
+information header.
+
+  o  Set the name to one that adheres to the stricter ProDOS naming 
+     rules and put its length at +023 and the name itself at +024 to 
+     +038.  Note that the name must be a simple filename and not a 
+     pathname.  The actual DOS 3.3 filename must be placed at +039 
+     (length) and +040 to +087 (name).  (DOS 3.3 actually restricts 
+     filenames to 30 characters.)
+
+  o  Set the ProDOS file type, auxiliary type and access to values 
+     which correspond to the DOS 3.3 file type:
+
+        DOS 3.3      ProDOS       ProDOS            ProDOS
+        File Type    File Type    Auxiliary Type    Access
+        __________________________________________________
+        $00 (T)       $04            $0000           $E3
+        $80 (*T)      $04            $0000           $21
+        $01 (I)       $FA            $0C00           $E3
+        $81 (*I)      $FA            $0C00           $21
+        $02 (A)       $FC            *               $E3
+        $82 (*A)      $FC            *               $21
+        $04 (B)       $06            **              $E3
+        $84 (*B)      $06            **              $21
+        $08 (S)       $06            $0000           $E3
+        $88 (*S)      $06            $0000           $21
+        $10 (R)       $FE            $0000           $E3
+        $90 (*R)      $FE            $0000           $21
+        $20 (A)       $06            $0000           $E3
+        $A0 (*A)      $06            $0000           $E3
+        $40 (B)       $06            $0000           $E3
+        $C0 (*B)      $06            $0000           $21
+        __________________________________________________
+
+        *  Set the auxiliary type for an A file to the
+           memory address from which the program was saved.
+           This is usually $0801.
+        ** Set the auxiliary type for a B file to the
+           value stored in the first two bytes of the the
+           file (this is the default load address).
+
+  o  Set the storage type code to $01.
+  o  Set the size of file in blocks, date of creation, date of 
+     modification, time of creation and time of modification all to 
+     $0000.
+  o  Set the end-of-file position to the length of the DOS 3.3 file, in 
+     bytes.  For a B file (code $04 or $84), this number is stored in 
+     the third and fourth bytes of the file.  For an I file (code $01 
+     or $81) or an A file (code $02 or $82), this number is stored in 
+     the first and second bytes of the file.
+  o  Set the operating system type to $01.
+  o  Set the native file type code to the value of the DOS 3.3 file 
+     type code.
+
+Attribute bytes inside a DOS 3.3 file (if any) must not be included in the 
+data portion of the Binary II file.  This includes the first four bytes of a B 
+(Binary) file, and the first two bytes of an A (AppleSoft) or I (Integer 
+BASIC) file.
+
+
+Further Reference
+_____________________________________________________________________________
+    o    GS/OS Reference
+    o    ProDOS 8 Technical Reference Manual
+    o    Apple II File Type Note, File Type $E0, Auxiliary Type $0001
+    o    Apple II File Type Note, File Type $E0, Auxiliary Type $8002
+    o    Apple II Miscellaneous Technical Note #14, Guidelines for 
+         Telecommunication Programs
+
+ +
This document is Copyright by Apple Computer, Inc.
+ +
+ + diff --git a/library/FTN.e08002.htm b/library/FTN.e08002.htm new file mode 100644 index 0000000..95406bf --- /dev/null +++ b/library/FTN.e08002.htm @@ -0,0 +1,966 @@ + + + + + + +FTN.e08002 + + + +
+ +


+

+
+ +
+

Apple II FTN - ShrinkIt (NuFX) document

+

Back to nulib.com library

+

NOTE to CRC-16 seekers:

+

Looks like a lot of people are hitting this page while looking for a CRC-16 +algorithm. That's not what this page is about. If you want a CRC-16 +implementation in C, try this one. +If you want one in 6502 assembly, skip down to the end of this document. +

+

+Apple II
+File Type Notes
+_____________________________________________________________________________
+                                                  Developer Technical Support
+
+File Type:         $E0 (224)
+Auxiliary Type:    $8002
+
+Full Name:     NuFile Exchange Archival Library
+Short Name:    ShrinkIt (NuFX) document
+
+Revised by:    Andy Nicholas and Matt Deatherage                    July 1990
+Written by:    Matt Deatherage                                      July 1989
+
+Files of this type and auxiliary type contain NuFX Archival Libraries.
+Changes since July 1989:  Rewrote major portions to reflect Master Version 
+$0002 of the NuFX standard.
+_____________________________________________________________________________
+
+Introduction
+
+NuFX is a robust, full-featured archival standard for the Apple II family.  
+The standard, as presented in this Note, allows for full archival of ProDOS 
+and GS/OS files while keeping all file attributes with each file, as well as 
+providing necessary archival functions such as multiple compression schemes 
+and multiple archival implementations of the same standard.  NuFX is 
+implemented in the application ShrinkIt, a free archival utility program for 
+enhanced IIe, IIc and IIgs computers.  (Versions for earlier Apple II models 
+are also available.)
+
+The NuFX standard was developed by Andrew Nicholas for Paper Bag Productions.  
+Comments or suggestions on the NuFX standard, or comments and suggestions on 
+ShrinkIt are welcome at:
+
+                    Paper Bag Productions
+                    8415 Thornberry Drive East
+                    Upper Marlboro, MD  20772
+                    Attn:  NuFX Technical Support
+                    America Online:    ShrinkIt
+                    GEnie:    ShrinkIt
+                    CompuServe:    70771,2615
+
+
+History
+
+The Apple II community has always lacked a well-defined method for archiving 
+files.  NuFX is an attempt to rectify the situation by providing a flexible, 
+consistent standard for archiving files, disks, and other computer media.  
+Although many files are archived using the Binary II standard (see Apple II 
+File Type Note, File Type $E0, Auxiliary Type $8000), it was not designed as 
+an archival standard and its continued use as such creates problems.  More 
+people are using Binary II as an archival standard than as a way to keep 
+attributes with a file when transferred, and this use is causing the original 
+intent of Binary II to become lost and unused.
+
+NuFX, developed as an archival standard for the days of GS/OS, allows:
+
+  o  Filenames longer than 64 characters (GS/OS can create 8,000-
+     character filenames).
+  o  A convenient way to add to, remove from, and work on an archive.
+  o  Including GS/OS files which contain resource forks.
+  o  Including entire disk images.
+  o  Including comments with a file.
+  o  A convenient way to represent a file compressed or encrypted by a 
+     specific application.
+  o  A true archive standard.  Binary IIs original intent was to make 
+     transfer of Apple II files from local machines to large 
+     information services possible; otherwise, a file's attribute 
+     information would be lost.  Use of Binary II to archive files 
+     rather than simply maintain their attributes stretches it beyond 
+     its original intent.
+
+Adding all of these features to the existing Binary II standard would be 
+nearly impossible without violating the existing standard and causing a great 
+deal of confusion.  Although Binary II is flexible, it is simply unable to 
+address all of these concerns without alienating existing Binary II extraction 
+programs.
+
+To provide some differentiation between standards and provide a better 
+functioning format, this Note presents a new standard called NuFX (NuFile 
+eXchange for the Apple II; pronounced new-F-X).  NuFX fixes the problems that 
+Apple IIgs users would soon be experiencing as other filing systems become 
+available for GS/OS.  NuFX attempts to stem a set of problems before they have 
+a chance to develop.  NuFX provides all of the features of Binary II, but goes 
+further to allow the user the ultimate in flexibility, usefulness and 
+performance.
+
+
+Additional Date/Time Data type:
+
+Date/Time (8 Bytes):
+
++000    second    Byte    The second, 0 through 59.
++001    minute    Byte    The minute, 0 through 59.
++002    hour      Byte    The hour, 0 through 23.
++003    year      Byte    The current year minus 1900.
++004    day       Byte    The day, 0 through 30.
++005    month     Byte    The month, 0 through 11 (0 = January).
++006    filler    Byte    Reserved, must be zero.
++007    weekDay   Byte    The day of the week, 1 through 7 (1 = Sunday).
+
+The format of the Date/Time field is identical to that described for the 
+ReadTimeHex call in the Apple IIgs Toolbox Reference Manual.
+
+
+Implementation
+
+Figure 1 illustrates the basic structure of a NuFX archive.
+
+                    |     First Record      |      Next Record      |
+     _______________|_______________________|_______________________|
+    | Master Header | Header |     Data     | Header |     Data     |
+    |_______________|________|______________|________|______________|
+
+                    Figure 1-NuFX Archive Structure
+
+A single master header block contains values which describe the entire archive 
+(those with knowledge of structured programming may consider them archive 
+globals).  Each of the succeeding header blocks contains only information 
+about the record it precedes (consider each an archive local).
+
+Each header block is followed by a list of threads, which is followed by the 
+actual threads.  The data for each thread may be a data fork, resource fork, 
+message, control sequence for a NuFX utility program, or almost any kind of 
+sequential data.
+
+Possible Block Combinations:
+
+The blocks must occur in the following fashion:
+
+    Master Header Block containing N entries
+
+    Header Block
+    Threads list:
+        filename_thread (16 bytes)
+        message_thread (16 bytes)
+        data thread (16 bytes)
+        .
+        .
+        .
+    filename_thread's data (filename_thread's comp_thread_eof # of bytes)
+    message_thread's data (message_thread's comp_thread_eof # of bytes)
+    data_thread's data (data_thread's comp_thread_eof # of bytes)
+    .
+    .
+    .
+    Next Header Block (notice no second Master Header block)
+    Threads list (message, control, data or resource)
+    .
+    .
+    .
+    Nth Header Block
+    Threads list (message, control, data or resource)
+
+Master Header Block Contents
+
++000    nufile_id      6 Bytes    These six bytes spell the word "NuFile" in 
+                                  alternating ASCII (low, then high) for 
+                                  uniqueness.  The six bytes are $4E $F5 $46 
+                                  $E9 $6C $E5.
++006    master_crc     Word       A 16-bit cyclic redundancy check 
+                                  (CRC) of the remaining fields in this 
+                                  block (bytes +008 through +047).  Any 
+                                  programs which modify the master header 
+                                  block must recalculate the CRC for the 
+                                  master header.  (see the section "A Sample 
+                                  CRC Algorithm")  The initial value of this 
+                                  CRC is $0000.
++008    total_records  Long       The total number of records in this 
+                                  archive file.  It is possible to chain 
+                                  multiple records (files or disks) 
+                                  together, as it is possible to chain 
+                                  different types of records together (mixed 
+                                  files and disks).
++012    archive_create_when
+                       Date/Time  The date and time on which this archive 
+                                  was initially created.  This field should 
+                                  never be changed once initially written.  
+                                  If the date is not known, or is unable to 
+                                  be calculated, this field should be set to 
+                                  zero.  If the weekday is not known, or is 
+                                  unable to be calculated, this field should 
+                                  be set to null.
++020    archive_mod_when
+                       Date/Time  The date of the last modification to this 
+                                  archive.  This field should be changed 
+                                  every time a change is made to any of the 
+                                  records in the archive.  If the date is 
+                                  not known, or is unable to be calculated, 
+                                  this field should be set to zero.  If the 
+                                  weekday is not known, or is unable to be 
+                                  calculated, this field should be set to 
+                                  null.
++028    master_version
+                       Word       The master version number of the NuFX 
+                                  archive.  This Note describes 
+                                  master_version $0002, for which the next 
+                                  eight bytes are zeroed.
++030    reserved       8 Bytes    Must be null ($00000000).
++038    master_eof     Long       The length of the NuFX archive, in 
+                                  bytes.  Any programs which modify the 
+                                  length of an archive, either increasing it 
+                                  or decreasing it in size, must change this 
+                                  field in the master header to reflect the 
+                                  new size.
+
+Header Block Contents:
+
+Following the Master Header block is a regular Header Block, which precedes 
+each record within the NuFX archive.  A cyclic redundancy check (CRC) has been 
+provided to detect archives which have possibly been corrupted.  The only time 
+the CRC should be included in a block is for the Master Header and for each of 
+the regular Header Blocks.  The CRC ensures reliability and data integrity.
+
++000    nufx_id        4 Bytes    These four bytes spell the word "NuFX" in 
+                                  alternating ASCII (low, then high) for 
+                                  uniqueness.  The four bytes are $4E $F5 
+                                  $46 $D8.
++004    header_crc     Word       The 16-bit CRC of the remaining 
+                                  fields of this block (bytes +006 through 
+                                  the end of the header block and any 
+                                  threads following it).  This field is used 
+                                  to verify the integrity of the rest of the 
+                                  block.  Programs which create NuFX 
+                                  archives must include this in every 
+                                  header.  It is up to the discretion of the 
+                                  extracting program to check the validity 
+                                  of this CRC.  Any programs which might 
+                                  modify the header of a particular record 
+                                  must recalculate the CRC for the header 
+                                  block.  The initial value for this CRC is 
+                                  zero ($0000).
++006    attrib_count   Word       This field describes the length of 
+                                  the attribute section of each record in 
+                                  bytes.  This count measures the distance 
+                                  in bytes from the first field (offset 
+                                  +000) up to and including the 
+                                  filename_length field.  By convention, the 
+                                  filename_length field will always be the 
+                                  last 2 bytes of the attribute section 
+                                  regardless of what has preceded it.
++008    version_number
+                       Word       Version of this record.  If version_number 
+                                  is $0000, no option_list fields are 
+                                  present.  If the version_number is $0001 
+                                  option_list fields may be present.  If the 
+                                  version_number is $0002 then option_list 
+                                  fields may be present and a valid CRC-16 
+                                  exists for the compressed data in the data 
+                                  threads of this record.  If the 
+                                  version_number is $0003 then option_list 
+                                  fields may be present and a valid CRC-16 
+                                  exists for the uncompressed data in the 
+                                  data threads of this record.  The current 
+                                  version number is $0003 and should always 
+                                  be used when making archives.
++010    total_threads  Long       The number of thread subrecords 
+                                  which should be expected immediately 
+                                  following the filename or pathname at the 
+                                  end of this header block.  This field is 
+                                  extremely important as it contains the 
+                                  information about the length of the last 
+                                  third of the header.
++014    file_sys_id    Word       The native file system identifier:
+                                      $0000    reserved
+                                      $0001    ProDOS/SOS
+                                      $0002    DOS 3.3
+                                      $0003    DOS 3.2
+                                      $0004    Apple II Pascal
+                                      $0005    Macintosh HFS
+                                      $0006    Macintosh MFS
+                                      $0007    Lisa File System
+                                      $0008    Apple CP/M
+                                      $0009    reserved, do not use (The 
+                                               GS/OS Character FST returns 
+                                               this value)
+                                      $000A    MS-DOS
+                                      $000B    High Sierra
+                                      $000C    ISO 9660
+                                      $000D    AppleShare
+                                      $000E-$FFFF    Reserved, do not use
+                                  If the file system of a disk being 
+                                  archived is not known, it should be set to 
+                                  zero.
++016    file_sys_info  Word       Information about the current filing 
+                                  system.  The low byte of this word (offset 
+                                  +016) is the native file system separator.  
+                                  For ProDOS, this is the slash (/ or $2F).  
+                                  For HFS and GS/OS, the colon (: or $3F) is 
+                                  used, and for MS-DOS, the separator is the 
+                                  backslash (\ or $5C).  This separator is 
+                                  provided so archival utilities may know 
+                                  how to parse a valid file or pathname from 
+                                  the filename field for the receiving file.  
+                                  GS/OS archival utilities should not 
+                                  attempt to parse pathnames, as it is not 
+                                  possible to build in syntax rules for file 
+                                  systems not currently defined.  Instead, 
+                                  pass the pathname directory to GS/OS and 
+                                  attempt translation (asking the user for 
+                                  suggestions) only if GS/OS returns an 
+                                  "Invalid Path Name Syntax" error.  The 
+                                  high byte of this word is reserved and 
+                                  should remain zero.
++018    access         Flag Long      Bits 31-8    reserved, must be zero
+                                      Bit 7 (D)    1 = destroy enabled
+                                      Bit 6 (R)    1 = rename enabled
+                                      Bit 5 (B)    1 = file needs to be 
+                                                   backed up
+                                      Bits 4-3    reserved, must be zero
+                                      Bit 2 (I)    1 = file is invisible
+                                      Bit 1 (W)    1 = write enabled
+                                      Bit 0 (R)    1 = read enabled
++022    file_type      Long       The file type of the file being archived.  
+                                  For ProDOS 8 or GS/OS, this field should 
+                                  always be what the operating system 
+                                  returns when asked.  For disks being 
+                                  archived, this field should be zero.
++026    extra_type     Long       The auxiliary type of the file being 
+                                  archived.  For ProDOS 8 or GS/OS, this 
+                                  field should always be what the operating 
+                                  system returns when asked.  For disks 
+                                  being archived, this field should be the 
+                                  total number of blocks on the disk.
++030    storage_type   Word       For Files:  The storage type of the 
+                                  file.  Types $1 through $3 are standard 
+                                  (one-forked) files, type $5 is an extended 
+                                  (two-forked) file, and type $D is a 
+                                  subdirectory.
+        file_sys_block_size
+                       Word       For Disks:  The block size used by the 
+                                  device should be placed in this field.  
+                                  For example, under ProDOS, this field will 
+                                  be 512, while for HFS it might be 524.  
+                                  The GS/OS Volume call will return this 
+                                  information if asked.
++032    create_when    Date/Time  The date and time on which 
+                                  this record was initially created.  If the 
+                                  creation date and time are available from 
+                                  a disk device, this information should be 
+                                  included.  If the date is not known, or is 
+                                  unable to be calculated, this field should 
+                                  be set to zero.  If the weekday is not 
+                                  known, or is unable to be calculated, this 
+                                  field should be set to zero.
++040    mod_when       Date/Time  The date and time on which this record was 
+                                  last modified.  If the modification date 
+                                  and time are available from a disk device, 
+                                  this information should be included.  If 
+                                  the date is not known, or is unable to be 
+                                  calculated, this field should be set to 
+                                  zero.  If the weekday is not known, or is 
+                                  unable to be calculated, this field should 
+                                  be set to zero.
++048    archive_when   Date/Time  The date and time on which 
+                                  this record was placed in this archive.  
+                                  If the date is not known, or is unable to 
+                                  be calculated, this field should be set to 
+                                  zero.  If the weekday is not known, or is 
+                                  unable to be calculated, this field should 
+                                  be set to zero.
+
+The following option_list information is only present if the NuFX version 
+number for this record is $0001 or greater.
+
++056    option_size    Word       The length of the FST-specific 
+                                  portion of a GS/OS option_list returned by 
+                                  GS/OS.  This field may be $0000, 
+                                  indicating the absence of a valid 
+                                  option_list.
+
+A GS/OS option_list is formatted as follows:
+
+        +000           buffer_size
+                       Word       Size of the buffer for GS/OS to 
+                                  place the option_list in, including 
+                                  this count word.  This must be at 
+                                  least $2E.
+        +002           list_size
+                       Word       The number of bytes of information 
+                                  returned by GS/OS.
+        +004           file_sys_ID
+                       Word       A file system ID word (see list 
+                                  above) identifying the FST owning 
+                                  the file in question.
+        +006           option_bytes
+                       Bytes      The bytes returned by the FST.  
+                                  There are (buffer_size - 6) of them.
+
+The option_list contains information specific to native file systems that 
+GS/OS doesn't normally use (such as true creator_type, file_type, and access 
+privileges for AppleShare).  Other FSTs released in the future will follow 
+similar conventions to return native file system specific parameters in the 
+option_list.  Information in the option_list should always be copied from file 
+to file.
+
+The value option_size in the NuFX header is the value of list_size minus two.  
+Immediately following the option_size count word are (list_size - 2) bytes.  
+To pass these values back to the destination file system, construct an 
+option_list with a suitably large buffer_size, a list_size of the NuFX 
+option_size + 2, the file_sys_id of the source file, and the FST-returned 
+option_bytes.
+
++058    list_bytes     Bytes      FST-specific bytes returned in an 
+                                  option_list.  These are the bytes in the 
+                                  GS/OS option_list not including the FST ID 
+                                  word.  There are option_size of them.  If 
+                                  option_size is an odd number, one zero 
+                                  byte of padding is added to keep the block 
+                                  size an even number.
+
+Because the attributes section does not have a fixed size, the next field must 
+be found by looking two bytes before the offset indicated by attrib_count 
+(+006).
+
++attrib_count - 2
+        filename_length
+                       Word       Obsolete, should be set to zero.  In 
+                                  previous versions of NuFX, this field was 
+                                  the length of a file name or pathname 
+                                  immediately following this field.
+
+                                  To allow the inclusion of future 
+                                  additional parameters in the attributes 
+                                  section, NuFX utility programs should rely 
+                                  on the attribs_count field to find the 
+                                  filename_length field.
+
+                                  Current convention is to zero this field 
+                                  when building an archive and put the file 
+                                  or pathname into a filename thread so the 
+                                  record can be renamed in the archive.  
+                                  Archival programs should recognize both 
+                                  methods to find a valid file name or 
+                                  pathname.
++attrib_count
+        filename       Bytes      Filename or partial pathname if 
+                                  applicable.  If this is a disk being 
+                                  archived, then the volume_name should be 
+                                  included in this field.  If a volume name 
+                                  is included in this field, a separator 
+                                  should not be included in, or precede the 
+                                  name.  If a volume name is not available, 
+                                  then this field should be zeros.
+
+                                  If a partial pathname is specified, the 
+                                  directories to which the current pathname 
+                                  refers need not have preceded this 
+                                  particular record.  The extraction program 
+                                  must test each referenced directory 
+                                  individually.  If the directory in 
+                                  question does not exist, the extracting 
+                                  program should create it.
+
+                                  Any utility which extracts file from a 
+                                  NuFX archive must not assume that this 
+                                  field will be in a format it is able to 
+                                  handle.  In particular, extraction 
+                                  programs should check for syntax 
+                                  unacceptable to the operating system under 
+                                  which they run and perform whatever 
+                                  conversions are necessary to parse a legal 
+                                  filename or pathname.  In general, assume 
+                                  nothing.  (GS/OS programs should pass the 
+                                  filename or pathname directly to GS/OS, 
+                                  and only attempt to convert the name if 
+                                  GS/OS returns an "invalid pathname syntax" 
+                                  error.)
+
+                                  Both high and low ASCII values are valid 
+                                  but may not mean the same to each file 
+                                  system (for example, all eight bits are 
+                                  significant in AppleShare pathnames while 
+                                  only seven are significant in ProDOS 
+                                  pathnames).
+
+
+Threads
+
+Thread Records are 16-byte records which immediately follow the Header Block 
+(composed of the attributes and file name of the current record) and describe 
+the types of data structures which are included with a given record.  The 
+number of Thread Records is described in the attribute section by a Word, 
+total_threads.
+
+Each Thread Record should be checked for the type of information that a given 
+utility program can extract.  If a utility is incapable of extracting a 
+particular thread, that thread should be skipped (with the exception of 
+extended files under ProDOS 8, which should be dearchived into AppleSingle 
+format, or both threads should be skipped).  If a utility finds a redundancy 
+in a Thread Record, it must decide whether to skip the record or to do 
+something with that particular thread (i.e., if a utility finds two 
+message_thread threads it can either ignore the second one or display it.  
+Likewise, if a utility finds two data_thread threads for the same file, it 
+should inspect the thread_kind of each.  If they match, it can either 
+overwrite the first thread extracted, or warn the user and skip the second 
+thread).
+
+Thread records can be represented as follows:
+
++000    thread_class   Word       The classification of the thread:
+                                      $0000    message_thread
+                                      $0001    control_thread
+                                      $0002    data_thread
+                                      $0003    filename_thread
++002    thread_format  Word       The format of the data within the thread:
+                                      $0000    Uncompressed
+                                      $0001    Huffman Squeeze
+                                      $0002    Dynamic LZW/1 (ShrinkIt 
+                                               specific)
+                                      $0003    Dynamic LZW/2 (ShrinkIt 
+                                               specific)
+                                      $0004    Unix 12-bit Compress
+                                      $0005    Unix 16-bit Compress
++004    thread_kind    Word       Describes the kind of data within 
+                                  the thread.
+
+thread_kind must be interpreted on the basis of thread_class.  See the table 
+below for the currently defined thread_kind interpretations:
+
+            class $0000  class $0001       class $0002            class $0003
+            -----------  ----------------  ---------------------  -----------
+kind $0000  ASCII text   create directory  data fork of file      filename
+kind $0001  see below    undefined         disk image             undefined
+kind $0002  see below    undefined         resource fork of file  undefined
+
++006    thread_crc     Word       For version_number $0003, this field 
+                                  is the CRC of the original data before it 
+                                  was compressed or otherwise transformed.  
+                                  The CRC-16's initial value is set to $FFFF.
++008    thread_eof     Long       The length of the thread when uncompressed.
++012    comp_thread_eof
+                       Long       The length of the thread when compressed.
+
+Class $0000 with kind $0000 is obsolete and should not be used.
+
+Class $0000 with kind $0001 has a predefined comp_thread_eof and a thread_eof 
+whose length may change.  This way, a certain amount of space may be allocated 
+when a record is created and edited later.
+
+Class $0000 with kind $0002 is a standard Apple IIgs icon.  comp_thread_eof is 
+the length of the icon image; thread_eof is ignored.
+
+Class $0003 with kind $0000 has a predefined comp_thread_eof and a thread_eof 
+whose length may change.  After this record is placed into the archive, the 
+thread_eof can be changed if the name is changed, but the length of the name 
+may not extend beyond the space allocated for it, comp_thread_eof.
+
+A thread_format of $0001 indicates Huffman Squeeze.  NuFX's Huffman is the 
+same Huffman used  by ARC v5.x, SQ and USQ, the source of which is publicly 
+available and was originally written by Richard Greenlaw.  The first word of 
+the thread data is the number of nodes followed by the Huffman tree and the 
+actual data.  This is also the same algorithm decoded by the Apple II version 
+of USQ written by Don Elton.  The C source to this is widely available.
+
+A thread_format of $0002 indicates a special variant of LZW (LZW/1) used by 
+ShrinkIt.  The first two bytes of this thread are a CRC-16 of the uncompressed 
+data within the thread.  This CRC-16 is initialized to zero ($0000). The third 
+byte is the low-level volume number used by the eight-bit version of ShrinkIt 
+to format 5.25" disks.  The fourth byte is the run-length character used to 
+decode the rest of the thread.  The data which comprises the compressed file 
+or disk immediately follows the RLE character.
+
+When ShrinkIt compresses a file, it reads 4096-byte chunks of the file until 
+it reaches the file's EOF.  The last 4096-byte chunk is padded with zeroes if 
+the file's length is not an exact multiple of 4096.  Compressing a disk is 
+also done by reading sequential blocks of 4096-bytes.
+
+Each 4K chunk is first compressed with RLE compression.  The RLE character is 
+determined by reading the fourth byte of the thread.  The RLE character which 
+is used by most current versions of ShrinkIt is $DB.  A run of characters is 
+represented by three bytes, consisting of the run character, the number of 
+characters in the run and the character in the run.  If the 4K chunk expands 
+after being compressed with RLE then the uncompressed 4K chunk is passed to 
+the LZW compressor. If the 4K chunk shrinks after being compressed with RLE 
+then the RLE-compressed image of the 4K chunk is passed to the LZW compressor.
+
+ShrinkIt's LZW compressor individually compresses each 4K chunk passed to it 
+by using variable length (9 to 12 bits) codes.  The way that ShrinkIt's LZW 
+compressor functions is almost identical to the algorithm used in the public 
+domain utility Compress.  The first code is $0101.  The LZW string table is 
+cleared before compressing each 4K chunk.  If the compressed chunk increases 
+in size, then the previous 4K chunk (which may be run-length-encoded or just 
+uncompressed data) is written to the file.
+
+The first word of every 4K chunk is aligned to a byte boundary within the file 
+and is the length which resulted from the attempt at compressing the chunk 
+with RLE.  If the value of this word is 4096, then RLE was not successful at 
+compressing the chunk.  A single byte follows the word and indicates whether 
+or not LZW was performed on this chunk.  A value of zero indicates that LZW 
+was not used, while a value of one indicates that LZW was used and that the 
+chunk must first be decompressed with LZW before doing any further processing.
+
+To decompress a file, each 4K chunk must first be expanded if it was 
+compressed by LZW.  If the 4K chunk wasn't compressed with LZW, then the word 
+which appears at the beginning of each chunk must be used to determine if the 
+data for the current chunk needs to be processed by the run-length decoder.  
+If the value of the word is 4096, then run-length decoding does not need to 
+occur because the data is uncompressed.
+
+If the word indicates that the length of the chunk after being decompressed by 
+LZW is 4096-bytes long, then no run-length decoding needs to take place.  If 
+value of the word is less than 4096 then the chunk must be run-length decoded 
+to 4096 bytes.
+
+There are four varying degrees of compression which can occur with a chunk: it 
+can be uncompressed data.  It can be run-length-encoded data without LZW 
+compression.  It can also be uncompressed data on which RLE was attempted (but 
+failed) and then was subsequently compressed with LZW.  Or, finally, the chunk 
+can be compressed with RLE and then also compressed with LZW.
+
+A thread_format of $0003 indicates a special variant of LZW (LZW/2) used by 
+ShrinkIt.  The first byte is the low-level volume number used by the eight-bit 
+version of ShrinkIt to format 5.25" disks.  The second byte is the run-length 
+character used to decode the rest of the thread.  The data which comprises the 
+compressed file or disk immediately follows the second byte of the thread.
+
+The format of LZW/2 is almost the same as LZW/1 with a few exceptions.  Unlike 
+LZW/1, where the LZW string table is automatically cleared before each 4K 
+chunk is processed, the LZW string table used by LZW/2 is only cleared when 
+the table becomes full, indicating a change in the redundancy of the source 
+text.  Not clearing the string table almost always yields improved compression 
+ratios because the compressor's dictionary is not being depleted every 4K and 
+larger strings are allowed to accumulate. The clear code used by ShrinkIt is 
+$100.  Whenever the decompressor sees a $100 code, it must clear the string 
+table.
+
+The string table is also cleared when the compressor has to "back track" 
+because a 4K chunk became larger.  Whenever a chunk that is not compressed by 
+LZW is seen by the decompressor, the LZW string table must be cleared.  Bits 
+0-12 of the first word of each chunk in a LZW/2 thread indicate the size of 
+the chunk after being compressed with RLE.  The high bit (bit 15) indicates 
+whether or not LZW was used on the chunk.  If LZW was not used (bit 15 = 0), 
+the data for the chunk immediately follows the first word.  If LZW was used 
+(bit 15 = 1), a second word which is a count of the total number of bytes used 
+by the current chunk follows the first word.  The mark of the next chunk can 
+be found by taking the mark at the beginning of the current chunk and adding 
+the second word to it, using that as an offset for a ProDOS 8 or GS/OS SetMark 
+call.  This is not normally necessary because the next chunk is processed 
+immediately after the current chunk.
+
+This second word is an improvement over LZW/1 because if a chunk becomes 
+corrupted, but the second word is valid, the next chunk can be found and most 
+of the file recovered.  The second word is not needed (and not present) when 
+LZW is not used on the chunk because the first word is also a count of the 
+number of bytes which follow that word.
+
+A thread_format of $0004 indicates that a maximum of 12 bits per LZW code by 
+Compress was used to build this thread.  The actual thread data contains 
+Compress's usual three-byte signature, the third byte of which contains the 
+actual number of bits per LZW code that was actually used.  The number of bits 
+may be less than or equal to 12.  Optimally, this requires (at 12 bits) a 16K 
+hash table to decode and should be used only for transferring to machines with 
+limited amounts of memory.  The C source to Compress is in the public domain 
+and is widely available.
+
+A thread_format of $0005 indicates that a maximum of 16 bits per LZW code by 
+Compress was used to build this thread.  The actual thread data contains 
+Compress's usual three-byte signature, the third byte of which contains the 
+actual number of bits per LZW code that was actually used.  The number of bits 
+may be less than or equal to 16.  Optimally, this requires (at 16 bits) a 256K 
+hash table to decode.  The C source to Compress is in the public domain and is 
+widely available.
+
+If a control_thread indicates that a directory should be created on the 
+destination device, the path to be created must take the form of a ProDOS 
+partial pathname.  That is, the path must not be preceded with a volume name.  
+For example, /Stuff/SubDir is an invalid path for this control_thread, while 
+SubDir/AnotherSubDir is valid.
+
+If a control_thread indicates that a path is to be created, all subdirectories 
+that are contained in the pathname must be created.
+
+control_thread threads will eventually be used to control the execution of 
+utility programs by allowing them to create, rename, and delete directories 
+and files and to move and modify files.  A form of scripting language will 
+eventually be able to allow utility programs to perform these actions 
+automatically.  control_thread threads will allow extraction programs to 
+perform operations similar to those of the Apple IIgs Installer, allowing 
+updates to program sets dependent on such things as creation or modification 
+dates and version numbers.
+
+
+Extra Information
+
+If the file system of a particular disk is not known, the file_sys_id field 
+should be set to zero, the volume name should also be zeroed, and all the 
+other fields pertaining only to files should be set to zero.
+
+If the file system of a particular disk is known, as many of the fields as 
+possible should be filled with the correct information.  Fields which do not 
+pertain to an archived disk should remain set to zero.
+
+If an entire disk is added to the archive without some form of compression 
+(i.e., record_format = uncompressed), then the blocks which comprise the disk 
+image must be added sequentially from the first through the last block.  Since 
+there will be no character included in the data stream to mark the end or 
+beginning of a block, extraction programs should rely on the 
+file_sys_block_size field to determine how many bytes to read from the record 
+to properly fill a block.
+
+Some Useful Thread Algorithms:
+
+The beginning of the thread records can be found with the following algorithm:
+
+    Threads := (mark at beginning of header) + (attrib_count) + 
+               (filename_length)
+
+The end of the thread records can be found with the following algorithm:
+
+    endOfThreads := Threads + (16 * total_threads)
+
+The beginning of a data_thread can be found with the following formula:
+
+    Data Mark := endOfThreads + (comp_thread_eof of all threads in the thread 
+                 list which are not data prior to finding a data_thread)
+
+The beginning of a resource_thread may be found with the following algorithm:
+
+    Resource Mark := endOfThreads + (comp_thread_eof of all threads in the 
+                     thread list which are not data prior to finding a 
+                     resource_thread)
+
+The next record can be found using the following algorithm:
+
+    Next Mark := endOfThreads + (comp_thread_eof of each thread)
+
+The file name and its length can be found with the following algorithm:
+
+    if (filename_length > 0)
+        then
+            length of filename is filename_length;
+            filename is found at attrib_count;
+        else
+            look through list of threads for a filename_thread;
+            if you find one, then length of filename is thread_eof;
+            if you don't find one, then you don't have a filename.
+
+
+Directories
+
+Directories are handled almost the same way that normal files are handled with 
+the exception that there will be no data in the thread which follows the 
+entry.  A Thread Record must exist to inform a utility that a directory is to 
+be created through the use of the proper control_thread value.
+
+Directories do not necessarily have to precede a record which references a 
+directory.  For example, if a record contains Stuff/MyStuff, the directory 
+Stuff need not exist for the extracting program to properly extract the 
+record.  The extracting program must check to see if each of the directories 
+referenced exist, and if one does not exist, create it.  While this method 
+places a great burden on the abilities of the extraction program, it avoids 
+the anomalies associated with the deletion of directories within an archive.
+
+
+A Sample CRC Algorithm
+
+Paper Bag Productions provides the source code to a very fast routine which 
+does the CRC calculation as needed for NuFX archives.  The routine makeLookup 
+needs to be called only once.  After the first call, the routine doByte should 
+be called repeatedly with each new byte in succession to generate the 
+cumulative CRC for the block.  The CRC word should be reset to null ($0000) 
+before beginning each new CRC.
+
+This is the same CRC calculation which is done for CRC/Xmodem and Ymodem.  The 
+code is easily portable to a 16-bit environment like the Apple IIgs.  The only 
+detrimental factor with this routine is that it requires 512 bytes of main 
+memory to operate.  If you can spare the space, this is one of the fastest 
+routines Paper Bag Productions knows to generate a CRC-16 on a 6502-type 
+machine.
+
+The CRC word should be reset to $0000 for normal CRC-16 and to $FFFF before 
+generating the CRC on the unpacked data for each data thread.
+
+
+*-------------------------------
+* fast crc routine based on table lookups by
+* Andy Nicholas - 03/30/88 - 65C02 - easily portable to nmos 6502 also.
+* easily portable into orca/m format, just snip and save.
+* Modified for generic EDAsm type assemblers - MD 6/19/89
+
+         X6502                          turn 65c02 opcodes on
+
+*-------------------------------
+* routine to make the lookup tables
+*-------------------------------
+
+makeLookup
+         LDX   #0                       zero first page
+zeroLoop STZ   crclo,x                  zero crc lo bytes
+         STZ   crchi,x                  zero crc hi bytes
+         INX
+         BNE   zeroLoop
+
+*-------------------------------
+* the following is the normal bitwise computation
+* tweeked a little to work in the table-maker
+
+docrc
+         LDX   #0                       number to do crc for
+
+fetch    TXA
+         EOR   crchi,x                  add byte into high
+         STA   crchi,x                  of crc
+
+         LDY   #8                       do 8 bits
+loop     ASL   crclo,x                  shift current crc-16 left
+         ROL   crchi,x
+         BCC   loop1
+
+* if previous high bit wasn't set, then don't add crc
+* polynomial ($1021) into the cumulative crc.  else add it.
+
+         LDA   crchi,x                  add hi part of crc poly into
+         EOR   #$10                     cumulative crc hi
+         STA   crchi,x
+
+         LDA   crclo,x                  add lo part of crc poly into
+         EOR   #$21                     cumulative crc lo
+         STA   crclo,x
+loop1    DEY                            do next bit
+         BNE   loop                     done? nope, loop
+
+         INX                            do next number in series (0-255)
+         BNE   fetch                    didn't roll over, so fetch more
+         RTS                            done
+
+crclo    ds    256                      space for low byte of crc table
+crchi    ds    256                      space for high bytes of crc table
+
+
+*-------------------------------
+* do a crc on 1 byte/fast
+* on initial entry, CRC should be initialized to 0000
+* on entry, A = byte to be included in CRC
+* on exit, CRC = new CRC
+*-------------------------------
+
+doByte
+         EOR   crc+1                    add byte into crc hi byte
+         TAX                            to make offset into tables
+
+         LDA   crc                      get previous lo byte back
+         EOR   crchi,x                  add it to the proper table entry
+         STA   crc+1                    save it
+
+         LDA   crclo,x                  get new lo byte
+         STA   crc                      save it back
+
+         RTS                            all done
+
+crc      dw    0000                     cumulative crc for all data
+
+The following CRC check is written in APW assembler format for an Apple IIgs 
+with 16-bit memory and registers on entry.
+
+crcByte  start
+
+crc      equ         $0
+crca     equ         $2
+crcx     equ         $4
+crctemp  equ         $6
+
+         sta         crca                                                 4
+         stx         crcx                                                 4
+
+         eor         crc+1              on entry, number to add to CRC    4
+         and         #$00ff             is in (A)                         3
+         asl         a                                                    2
+         tax                                                              2
+         lda         crc16Table,x                                         5
+         and         #$00ff                                               3
+         sta         crcTemp                                              4
+
+         lda         crc-1                                                4
+         eor         crc16Table,x                                         5
+         and         #$ff00                                               3
+         ora         crcTemp                                              4
+         sta         crc                                                  4
+
+         lda         crca                                                 4
+         ldx         crcx                                                 4
+         rts                                                    cycles = 59
+
+
+;
+; CRC-16 Polynomial = $1021
+;
+crc16table anop
+         dc    i'$0000, $1021, $2042, $3063, $4084, $50a5, $60c6, $70e7'
+         dc    i'$8108, $9129, $a14a, $b16b, $c18c, $d1ad, $e1ce, $f1ef'
+         dc    i'$1231, $0210, $3273, $2252, $52b5, $4294, $72f7, $62d6'
+         dc    i'$9339, $8318, $b37b, $a35a, $d3bd, $c39c, $f3ff, $e3de'
+         dc    i'$2462, $3443, $0420, $1401, $64e6, $74c7, $44a4, $5485'
+         dc    i'$a56a, $b54b, $8528, $9509, $e5ee, $f5cf, $c5ac, $d58d'
+         dc    i'$3653, $2672, $1611, $0630, $76d7, $66f6, $5695, $46b4'
+         dc    i'$b75b, $a77a, $9719, $8738, $f7df, $e7fe, $d79d, $c7bc'
+         dc    i'$48c4, $58e5, $6886, $78a7, $0840, $1861, $2802, $3823'
+         dc    i'$c9cc, $d9ed, $e98e, $f9af, $8948, $9969, $a90a, $b92b'
+         dc    i'$5af5, $4ad4, $7ab7, $6a96, $1a71, $0a50, $3a33, $2a12'
+         dc    i'$dbfd, $cbdc, $fbbf, $eb9e, $9b79, $8b58, $bb3b, $ab1a'
+         dc    i'$6ca6, $7c87, $4ce4, $5cc5, $2c22, $3c03, $0c60, $1c41'
+         dc    i'$edae, $fd8f, $cdec, $ddcd, $ad2a, $bd0b, $8d68, $9d49'
+         dc    i'$7e97, $6eb6, $5ed5, $4ef4, $3e13, $2e32, $1e51, $0e70'
+         dc    i'$ff9f, $efbe, $dfdd, $cffc, $bf1b, $af3a, $9f59, $8f78'
+         dc    i'$9188, $81a9, $b1ca, $a1eb, $d10c, $c12d, $f14e, $e16f'
+         dc    i'$1080, $00a1, $30c2, $20e3, $5004, $4025, $7046, $6067'
+         dc    i'$83b9, $9398, $a3fb, $b3da, $c33d, $d31c, $e37f, $f35e'
+         dc    i'$02b1, $1290, $22f3, $32d2, $4235, $5214, $6277, $7256'
+         dc    i'$b5ea, $a5cb, $95a8, $8589, $f56e, $e54f, $d52c, $c50d'
+         dc    i'$34e2, $24c3, $14a0, $0481, $7466, $6447, $5424, $4405'
+         dc    i'$a7db, $b7fa, $8799, $97b8, $e75f, $f77e, $c71d, $d73c'
+         dc    i'$26d3, $36f2, $0691, $16b0, $6657, $7676, $4615, $5634'
+         dc    i'$d94c, $c96d, $f90e, $e92f, $99c8, $89e9, $b98a, $a9ab'
+         dc    i'$5844, $4865, $7806, $6827, $18c0, $08e1, $3882, $28a3'
+         dc    i'$cb7d, $db5c, $eb3f, $fb1e, $8bf9, $9bd8, $abbb, $bb9a'
+         dc    i'$4a75, $5a54, $6a37, $7a16, $0af1, $1ad0, $2ab3, $3a92'
+         dc    i'$fd2e, $ed0f, $dd6c, $cd4d, $bdaa, $ad8b, $9de8, $8dc9'
+         dc    i'$7c26, $6c07, $5c64, $4c45, $3ca2, $2c83, $1ce0, $0cc1'
+         dc    i'$ef1f, $ff3e, $cf5d, $df7c, $af9b, $bfba, $8fd9, $9ff8'
+         dc    i'$6e17, $7e36, $4e55, $5e74, $2e93, $3eb2, $0ed1, $1ef0'
+         end
+
+Further Reference
+_____________________________________________________________________________
+  o  ProDOS 8 Technical Reference Manual
+  o  GS/OS Reference
+  o  Apple IIgs Toolbox Reference Manual
+  o  Apple II File Type Note, File Type $E0, Auxiliary Type $8000
+  o  Apple II Miscellaneous Technical Note #14, Guidelines for 
+     Telecommunication Programs
+  o  "A Technique for High-Performance Data Compression," T. Welch, 
+     IEEE Computer, Vol. 17, No.6, June 1984, pp. 8-19.
+
+ +
This document is Copyright by Apple Computer, Inc.
+ +
+ + diff --git a/library/Lesson9.htm b/library/Lesson9.htm new file mode 100644 index 0000000..c91b089 --- /dev/null +++ b/library/Lesson9.htm @@ -0,0 +1,353 @@ + + + + + + +Hacking Data Compression - Lesson 9 + + + +
+ +


+

+
+ +
+

Hacking Data Compression - Lesson 9

+

Back to nulib.com library

+
+Hacking Data Compression  Lesson 9  NuFX and ShrinkIt
+By Andy McFadden
+GEnie A2Pro Apple II University - Copyright (C) 1992 GEnie
+
+
+This week's lesson is about the de facto archiving standard for the
+Apple II world: NuFX, as implemented by ShrinkIt and GS/ShrinkIt.
+There are a variety of other programs, like ][+ ShrinkIt,
+Auto-Unshrink, NuLib, NuPak, and YankIt, but they all handle the same
+archive format.
+
+The information covered here will be of interest to people who want to
+write software that works with NuFX archives.  If you're only
+interested in general data compression, you can probably skip this
+one.
+
+-=O=-     ShrinkIt
+
+Back in the Olden Days, the format for storage and transmission of
+data was Binary II (abbreviated "BNY", and occasionally referred to as
+"bunny" format).  It arranged things on 128-byte boundaries for the
+convenience of terminal programs, and is still used today for that
+purpose.  Actually, it was primarily meant as a way to retain the
+filetype, auxtype, and other goodies associated with a file, much like
+MacBinary for the Macintosh, but it also worked okay as an archive
+format.
+
+What made it Really Cool was the Huffman + RLE compressor which was,
+well, almost built in to the Binary II Library Utility (BLU).  It did
+a reasonable job of making things smaller, and in a modest amount of
+time.  It was called SQueezed compression, which somehow got
+abbreviated ".QQ".  Archives with squeezed files in them were ".BQY".
+
+However, it was really depressing to log onto a UNIX system and watch
+UNIX compress crunch files into nothing.  So, one fine day, a skinny
+little college student named Andy Nicholas, and a taller but still
+rather slim guy named Kent Dickey decided it would be a really neat
+thing to do LZW on the Apple II.
+
+Since the IIgs and System 4.0 had been released, it was apparent that
+the old BNY format was missing a few pieces.  Specifically, it had no
+clear provision for forked files, file comments, other file systems,
+and the expanded size of the stuff that GS/OS was returning in
+GetFileInfo calls.  So, with some suggestions from a few dozen
+Apple II personalities, the NuFX format came into being.
+
+(Sounds pretty dramatic, huh?  Hmm, maybe I should write fiction...)
+
+And so it came to pass that ShrinkIt was born.  It took a little while
+to work all the kinks out, but before long it was recognized that the
+program worked faster and packed better than the old SQueezed format.
+When Binary II support was added, the fate of BLU and its ilk was
+sealed.
+
+One interesting bit of history is that the LZW routines were initially
+meant for a program like DDD (Dalton's Disk Disintegrator), i.e. a
+*disk* packer which happened to pack files as well.  This caused some
+interesting twists in the design.
+
+-=O=-     Details, Details
+
+The first implementation of LZW was designed around packing a 5.25"
+disk using as little memory as possible.  The strategy was to grab 4K
+chunks (the size of a single track on a 5.25" floppy), put it through
+an RLE pass (since tracks were often filled with zeros, this was very
+effective), and then an LZW pass.  If the RLE or LZW pass failed, it
+would be omitted.  If both failed, the block was stored without
+compression.  A few bits at the start of each chunk in the compressed
+output held the information.
+
+It turns out that putting a typical file through an RLE pass first
+doesn't really do you much good, though it isn't likely to do much
+harm.  Mostly it just slows things down, unless the file is filled
+with a single character.  In some cases it can make things slightly
+worse, because LZW is encoding the three-byte RLE codes instead of
+encoding long strings of zeros with a single 9 to 12-bit symbol.  If
+more or all of a *disk* track is empty, RLE is a win, but it just
+doesn't happen all that often in text files.
+
+The LZW encoder was reset after every 4K of *input*, which meant that
+it never had a chance to get completely full, and some of the 12-bit
+code space was wasted.  However, this limited the maximum height of
+the tree considerably.  For example, 4K of zeros can be compressed by
+a degenerate tree with 91 nodes in it.  Since RLE would compress such
+a chunk, to get the worst case you have to alternate characters, so
+the worst you can get is around 64.  This meant that the LZW stack
+required well under 100 bytes instead of 4096, and the loss in
+compression wasn't too unpleasant.
+
+The other nice feature is that, since the LZW table can't get
+completely full, there's no need for explicit (or implicit) table
+clears.  The whole thing just gets wiped on 4K boundaries.
+
+To find the end of the data, the size of the output was stored.
+(Recall there are three ways of knowing when to stop: an explicit stop
+character, knowing the size of the input, and knowing the size of the
+output.)  The LZW code given in lesson 8 used an explicit "end of
+file" code because I thought it was more interesting to do it that
+way, and because it works on a stream of data (you can't seek back and
+write the length bytes on a continuous stream... but since ShrinkIt
+knows the data will be < 4K, it can just stuff the bytes in before it
+writes to disk).
+
+This scheme was eventually referred to as "LZW-I", and is still used
+by the ][+ and //e versions of ShrinkIt.
+
+-=*=-
+
+When writing GS/ShrinkIt, Andy Nicholas decided to hunker down and do
+LZW with table clears and all the fancy nonsense.  However, he kept
+the 4K boundaries, which made things a bit confusing (and led to a bug
+which hung around in GSHK, NuLib, and YankIt until rather recently).
+
+Basically, the input was still read in 4K chunks, and the RLE pass was
+still done.  As before, the LZW pass was done on either the original
+4K or the RLE output, depending on whether or not RLE helped.  The
+difference was that the table clears were no longer done after the end
+of the input.  Instead, an explicit table clear code ($100) was sent
+whenever the table filled.  To avoid having to make a copy of the
+table on every chunk, the table was also cleared whenever LZW failed
+to compress a piece of the file.
+
+Of course, now the decoder stack had to be 4K, and all the fun with
+the table clears had to be added in.  The long-lasting bug mentioned
+earlier occurred whenever a chunk ended after a table clear plus one
+character - if you look at the LZW decoder, you'll see that it grabs
+the first character, sets the prefix to that, and then plows onward.
+So the decoder was grabbing the first character, reaching the end of
+input, and then restarting with the next chunk by grabbing the first
+character again.  This isn't an issue if you don't chop things into 4K
+pieces, which is why you don't see it mentioned in the class stuff.
+
+So, LZW-II has all the advantages of an encoder which doesn't break
+the input into pieces, and retains its heritage as a disk compressor.
+One nice feature of the "chunking" is that it enables files with tough
+spots in them to be compressed better.  Suppose you have a file with
+three pieces, call them A, B, and C.  Let A and C be squishy text-like
+substance, while B is essentially random digitized sound.  Even if we
+get 50% compression in A and C, B could expand by some amount (say,
+25%), reducing or even nullifying the compression of A and C.  By
+storing the tough parts without compression, the only increase in B's
+size is the few bits of overhead needed to identify it as
+uncompressed.
+
+This slightly anachronistic "chunking" behavior can help significantly
+for some kinds of files.  By reducing the overhead to only a few bytes
+per 4K chunk, ShrinkIt will almost always do better than the simple
+LZW encoder given in class.  The price is greater complexity in the
+compressor and decompressor, and a few bytes of overhead in the file.
+
+If you want to see what effect the improvements had, compress some
+files with ShrinkIt and then compress them with GS/ShrinkIt.  The
+difference tends to be small, which is counter to intuition, since LZW
+does poorly at the start.
+
+-=*=-
+
+This section explains the format of the LZW-I and LZW-II compressed
+threads. To find out what "threads" are, you'll need to consult the
+NuFX file type note.
+
+LZW-I looks like this:
+
++0   (word) 16-bit CRC for the uncompressed data
++2   (byte) disk volume # (remember, it started as a 5.25 disk packer)
++3   (byte) RLE escape char, usually 0xdb
+
++4   a series compressed chunks, each built from 4K of input.  Each
+     chunk starts with the following:
+
+++0  (word) size of data after RLE but before LZW (will be 4096
+     if RLE wasn't used)
+++2  (byte) LZW flag - zero if LZW not used
+++3  compressed data
+
+So something packed with only RLE would have a size != 4096 and an LZW
+flag of zero.  LZW only would have a size == 4096 and LZW flag not
+equal to zero.  RLE+LZW and storing without compression can be
+expressed by juggling things.  The value at ++0 is used to decide when
+the LZW uncompressor should stop.
+
+The CRC is computed on the uncompressed data.  If the last piece of
+the file doesn't fill an entire 4K chunk, then the rest of the 4K
+chunk is filled with zeros.  The ENTIRE chunk is compressed, zeros and
+all, and the CRC includes the extra zeros.
+
+The actual LZW compression works like it did in class, with the
+modifications described earlier.  Even though no explicit table clears
+are used, the codes begin at $101.
+
+
+LZW-II looks like this:
+
++0   (byte) disk volume #
++1   (byte) RLE escape char, usually 0xdb
+
++2   a series compressed chunks, each built from 4K of input.  Each
+     chunk starts with the following:
+
+++0  (word) size of data after RLE but before LZW (will be 4096
+     if RLE wasn't used).  The hi bit is used as the LZW flag,
+     i.e. $8000 is ORed in if LZW is used.
+++2  (word) if and only if LZW is used, an extra length word is added
+     here, which gives the total length of the compressed data after
+     BOTH RLE and LZW.
+++2  or
+++4  compressed data
+
+The reason for the extra length word is to provide a way to recover
+pieces of a file even if one of the chunks has been corrupted.  With
+LZW-I, there's no way to identify the start of a chunk; with LZW-II,
+you can seek from the current chunk header to the next.  The value
+isn't actually needed by the decompressor.  If only RLE is used, then
+the size at ++0 will be accurate, so it's omitted from the output.
+
+Notice that there's no CRC included in the header.  This is because
+the CRC was moved to the thread header.  It's computed with an initial
+seed of 0xffff (for historical reasons), and is computed on the
+original file.  It does NOT include the zeros used to pad things to 4K
+boundaries.
+
+-=O=-     Knowne Implementations of NuFX Programmes
+
+- ShrinkIt (a/k/a "P8 ShrinkIt").  Runs on enhanced //e, //c, IIgs.
+Requires 65c02.  Packs with LZW-I, unpacks LZW-I and LZW-II.
+
+- GS/ShrinkIt.  Runs on IIgs only.  Packs with LZW-II, unpacks LZW-I
+and II.  Also handles several other file formats, including .Z (UNIX
+compress) and .ARC.
+
+- ShrinkIt ][+ and UnShrinkIt ][+.  Run on any Apple II, including
+Apple ][+.  One packs, the other unpacks.
+
+- Auto-Unshrink.  Runs on any Apple II (?).  Sort of a strange
+concept, but what the heck.  Notable feature is a certain amount of
+error recovery for damaged archives (it knows about and will use the
+extra length word included for LZW-II).
+
+- NuLib.  Shell-based, runs on darn near anything.  Packs with LZW-I,
+unpacks LZW-I, LZW-II, and UNIX compress (yes Virginia, you can have
+files packed with UNIX compress in a ShrinkIt archive... you just
+can't unpack them with anything else).
+
+- NuPak.  Started out like NuLib and GS/ShrinkIt slammed together at
+high speed, but never went anywhere.  The $15 shareware fee and the
+release of GSHK probably did it in.
+
+- YankIt.  Shell-based, IIgs only.  Unpacks LZW-I and II, does not
+pack at all.  Faster (and smaller) than the others.
+
+-=O=-     PROGRAM
+
+Being the author of both NuLib and YankIt, you'd figure I'd have some
+decent sample source code to give away.  However, it shapes up like
+this:
+
+- NuLib wasn't so much written as it was "evolved."  It started out as
+an archive viewer, and expanded into a rather grotesque archiver.  It
+looks okay on the outside, but the innards are a mess.  Fortunately,
+it was written in C.
+
+- YankIt is a cute little bare-bones archive viewer and extractor.  It
+fit into about 6000 lines of assembly, and took under a month to
+design, implement, and test.  My original intent was to release the
+complete source code with this lesson, but some bugs have been found
+and I haven't had the time to fix them, test it, etc, etc.
+
+So, this means I'm going to flake and just point you at some useful
+stuff.  I'll get the YankIt sources out eventually.
+
+-=*=-
+
+First of all, the definitive reference on the NuFX format is the
+filetype note for $e0/8002, which can be found in GEnie's A2Pro
+library.  The not-quite-so-definitive-anymore reference is in the
+article AndyN wrote for the Winter 1990 Call-A.P.P.L.E.  The article
+also contains a bit of history about NuFX and ShrinkIt.
+
+The NuLib source code, which contains a bug in the stack declaration
+(only 100 bytes - correct for LZW-I, but not LZW-II, which should be
+4096), can be found in the A2Pro library on GEnie, and on the
+cco.caltech.edu anonymous FTP site.
+
+NuLib executables for the IIgs and MS-DOS can also be found on GEnie,
+in the appropriate sections.  Documentation for the MS-DOS version got
+renamed to "NULIB.SHK" or something equally meaningless.  The file
+comment is also wrong; use "nulib xvt0 nulib.shk" to convert the line
+terminators from CR to CRLF.  (It says to use "xvt2", which would
+convert CRLF to CRLF - a slightly silly thing to do.)  NuLib has a
+whole pile of options, and would be a great program if the source code
+weren't so ugly.
+
+(Hey, I started it a year after I started learning C, and well before
+I figured out how LZW worked... be charitable.)
+
+The YankIt executable can be found in GEnie's A2 section.  This
+contains the same stack bug found in NuLib, but it's smart enough to
+complain about the archive being corrupted and bail without scrubbing
+itself.  It's got a nice "integrity check" feature which runs through
+and uncompresses every file, checking CRCs, but doesn't actually store
+them anywhere.  NuLib acts like it has a similar feature, but it lies.
+GSHK should have a similar feature, but I caught AndyN too close to
+the shipping date of v1.1.
+
+Since NuLib and YankIt are shell-based, and I didn't want to add the
+code to correctly handle disk archives, they extract disks as a single
+file.  This can be useful for UNIX and MS-DOS Apple II emulators which
+require disk-files.  Note that ShrinkIt uses ProDOS sector mapping, so
+the disks will work fine for the known UNIX emulators, but not for the
+MS-DOS emulator (which expects DOS 3.3 sector ordering).  There are
+programs available which change the order around.
+
+-=O=-     DEEP THOUGHTS
+
+- What are the advantages of writing an archiver which processes
+archive entries individually instead of reading them all at once?
+(hint: "yankit xv - < file.shk")
+
+- Is 4096 the optimal size for stopping and checking compression
+progress?
+
+- How does ShrinkIt's "checkpoint every 4K" compare to v.42bis?  (see
+lesson 8 for details on v.42bis)
+
+- Why does ShrinkIt clear the table after a chunk fails to compress
+with LZW?  What would you have to do to avoid doing so?
+
+-=!=-
+
+
This document is Copyright by Genie and Andy McFadden
+
+ + diff --git a/library/_vti_cnf/Crc16.c.txt b/library/_vti_cnf/Crc16.c.txt new file mode 100644 index 0000000..877b299 --- /dev/null +++ b/library/_vti_cnf/Crc16.c.txt @@ -0,0 +1,4 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|23 Aug 2000 17:09:22 -0000 +vti_extenderversion:SR|4.0.2.2717 +vti_backlinkinfo:VX|library/ftn.e08002.htm diff --git a/library/_vti_cnf/FTN.e00001.htm b/library/_vti_cnf/FTN.e00001.htm new file mode 100644 index 0000000..a995a7b --- /dev/null +++ b/library/_vti_cnf/FTN.e00001.htm @@ -0,0 +1,24 @@ +vti_encoding:SR|utf8-nl +vti_author:SR|fadden +vti_modifiedby:SR|fadden +vti_timecreated:TR|31 Mar 2001 23:27:59 -0000 +vti_timelastmodified:TR|31 Mar 2001 23:50:31 -0000 +vti_shadowfiles:VX| +vti_filesize:IR|18223 +vti_title:SR|FTN.e00001 +vti_metatags:VR|HTTP-EQUIV=Content-Type text/html;\\ charset=windows-1252 GENERATOR Microsoft\\ FrontPage\\ 4.0 ProgId FrontPage.Editor.Document +vti_progid:SR|FrontPage.Editor.Document +vti_generator:SR|Microsoft FrontPage 4.0 +vti_extenderversion:SR|4.0.2.2717 +vti_backlinkinfo:VX|library/index.htm +vti_nexttolasttimemodified:TR|31 Mar 2001 23:29:42 -0000 +vti_cacheddtm:TX|31 Mar 2001 22:50:30 -0000 +vti_cachedlinkinfo:VX|H|index.htm +vti_cachedsvcrellinks:VX|FHUS|library/index.htm +vti_cachedtitle:SR|FTN.e00001 +vti_cachedbodystyle:SR| +vti_cachedhasbots:BR|true +vti_cachedhastheme:BR|false +vti_cachedhasborder:BR|true +vti_botnavbits:SW|SHUB +vti_borderaggregate:SR|default diff --git a/library/_vti_cnf/FTN.e000023.htm b/library/_vti_cnf/FTN.e000023.htm new file mode 100644 index 0000000..004fe3e --- /dev/null +++ b/library/_vti_cnf/FTN.e000023.htm @@ -0,0 +1,24 @@ +vti_encoding:SR|utf8-nl +vti_author:SR|fadden +vti_modifiedby:SR|fadden +vti_timecreated:TR|31 Mar 2001 23:27:59 -0000 +vti_timelastmodified:TR|31 Mar 2001 23:50:40 -0000 +vti_filesize:IR|23942 +vti_title:SR|FTN.e000023 +vti_metatags:VR|HTTP-EQUIV=Content-Type text/html;\\ charset=windows-1252 GENERATOR Microsoft\\ FrontPage\\ 4.0 ProgId FrontPage.Editor.Document +vti_progid:SR|FrontPage.Editor.Document +vti_generator:SR|Microsoft FrontPage 4.0 +vti_extenderversion:SR|4.0.2.2717 +vti_backlinkinfo:VX|library/index.htm +vti_nexttolasttimemodified:TR|31 Mar 2001 23:32:29 -0000 +vti_shadowfiles:VX| +vti_cacheddtm:TX|31 Mar 2001 22:50:40 -0000 +vti_cachedlinkinfo:VX|H|index.htm +vti_cachedsvcrellinks:VX|FHUS|library/index.htm +vti_cachedtitle:SR|FTN.e000023 +vti_cachedbodystyle:SR| +vti_cachedhasbots:BR|true +vti_cachedhastheme:BR|false +vti_cachedhasborder:BR|true +vti_botnavbits:SW|SHUB +vti_borderaggregate:SR|default diff --git a/library/_vti_cnf/FTN.e00005.htm b/library/_vti_cnf/FTN.e00005.htm new file mode 100644 index 0000000..57f3908 --- /dev/null +++ b/library/_vti_cnf/FTN.e00005.htm @@ -0,0 +1,23 @@ +vti_encoding:SR|utf8-nl +vti_author:SR|fadden +vti_timecreated:TR|31 Mar 2001 23:27:59 -0000 +vti_timelastmodified:TR|08 Oct 2002 02:13:58 -0000 +vti_filesize:IR|10628 +vti_title:SR|FTN.e00005 +vti_metatags:VR|HTTP-EQUIV=Content-Type text/html;\\ charset=windows-1252 GENERATOR Microsoft\\ FrontPage\\ 4.0 ProgId FrontPage.Editor.Document +vti_progid:SR|FrontPage.Editor.Document +vti_generator:SR|Microsoft FrontPage 4.0 +vti_extenderversion:SR|4.0.2.5526 +vti_backlinkinfo:VX|library/index.htm +vti_nexttolasttimemodified:TR|31 Mar 2001 23:33:44 -0000 +vti_shadowfiles:VX| +vti_cacheddtm:TX|08 Oct 2002 02:13:58 -0000 +vti_cachedlinkinfo:VX|H|index.htm +vti_cachedsvcrellinks:VX|FHUS|library/index.htm +vti_cachedtitle:SR|FTN.e00005 +vti_cachedbodystyle:SR| +vti_cachedhasbots:BR|true +vti_cachedhastheme:BR|false +vti_cachedhasborder:BR|true +vti_botnavbits:SW|SHUB +vti_borderaggregate:SR|default diff --git a/library/_vti_cnf/FTN.e08000.htm b/library/_vti_cnf/FTN.e08000.htm new file mode 100644 index 0000000..bcd831c --- /dev/null +++ b/library/_vti_cnf/FTN.e08000.htm @@ -0,0 +1,24 @@ +vti_encoding:SR|utf8-nl +vti_author:SR|fadden +vti_modifiedby:SR|fadden +vti_timecreated:TR|31 Mar 2001 23:27:59 -0000 +vti_timelastmodified:TR|31 Mar 2001 23:50:58 -0000 +vti_filesize:IR|28685 +vti_title:SR|FTN.e08000 +vti_metatags:VR|HTTP-EQUIV=Content-Type text/html;\\ charset=windows-1252 GENERATOR Microsoft\\ FrontPage\\ 4.0 ProgId FrontPage.Editor.Document +vti_progid:SR|FrontPage.Editor.Document +vti_generator:SR|Microsoft FrontPage 4.0 +vti_extenderversion:SR|4.0.2.2717 +vti_backlinkinfo:VX|library/old-binary2-spec.htm library/index.htm +vti_nexttolasttimemodified:TR|31 Mar 2001 23:35:11 -0000 +vti_shadowfiles:VX| +vti_cacheddtm:TX|31 Mar 2001 22:50:58 -0000 +vti_cachedlinkinfo:VX|H|index.htm +vti_cachedsvcrellinks:VX|FHUS|library/index.htm +vti_cachedtitle:SR|FTN.e08000 +vti_cachedbodystyle:SR| +vti_cachedhasbots:BR|true +vti_cachedhastheme:BR|false +vti_cachedhasborder:BR|true +vti_botnavbits:SW|SHUB +vti_borderaggregate:SR|default diff --git a/library/_vti_cnf/FTN.e08002.htm b/library/_vti_cnf/FTN.e08002.htm new file mode 100644 index 0000000..5096ff6 --- /dev/null +++ b/library/_vti_cnf/FTN.e08002.htm @@ -0,0 +1,23 @@ +vti_encoding:SR|utf8-nl +vti_author:SR|fadden +vti_timecreated:TR|31 Mar 2001 23:27:59 -0000 +vti_timelastmodified:TR|05 May 2002 21:01:30 -0000 +vti_filesize:IR|53231 +vti_title:SR|FTN.e08002 +vti_metatags:VR|HTTP-EQUIV=Content-Type text/html;\\ charset=windows-1252 GENERATOR Microsoft\\ FrontPage\\ 4.0 ProgId FrontPage.Editor.Document +vti_progid:SR|FrontPage.Editor.Document +vti_generator:SR|Microsoft FrontPage 4.0 +vti_extenderversion:SR|4.0.2.7802 +vti_backlinkinfo:VX|library/old-nufx-spec.htm nufxlibapi.htm library/index.htm library/nufx-addendum.htm +vti_nexttolasttimemodified:TR|31 Mar 2001 23:51:11 -0000 +vti_shadowfiles:VX| +vti_cacheddtm:TX|05 May 2002 21:01:30 -0000 +vti_cachedlinkinfo:VX|H|index.htm H|Crc16.c.txt +vti_cachedsvcrellinks:VX|FHUS|library/index.htm FHUS|library/Crc16.c.txt +vti_cachedtitle:SR|FTN.e08002 +vti_cachedbodystyle:SR| +vti_cachedhasbots:BR|true +vti_cachedhastheme:BR|false +vti_cachedhasborder:BR|true +vti_botnavbits:SW|SHUB +vti_borderaggregate:SR|default diff --git a/library/_vti_cnf/Lesson9.htm b/library/_vti_cnf/Lesson9.htm new file mode 100644 index 0000000..ab05909 --- /dev/null +++ b/library/_vti_cnf/Lesson9.htm @@ -0,0 +1,24 @@ +vti_encoding:SR|utf8-nl +vti_author:SR|fadden +vti_modifiedby:SR|fadden +vti_timecreated:TR|31 Mar 2001 23:27:59 -0000 +vti_timelastmodified:TR|31 Mar 2001 23:42:04 -0000 +vti_filesize:IR|16806 +vti_title:SR|Hacking Data Compression - Lesson 9 +vti_metatags:VR|HTTP-EQUIV=Content-Type text/html;\\ charset=windows-1252 GENERATOR Microsoft\\ FrontPage\\ 4.0 ProgId FrontPage.Editor.Document +vti_progid:SR|FrontPage.Editor.Document +vti_generator:SR|Microsoft FrontPage 4.0 +vti_extenderversion:SR|4.0.2.7802 +vti_backlinkinfo:VX|library/index.htm +vti_nexttolasttimemodified:TR|31 Mar 2001 23:40:39 -0000 +vti_shadowfiles:VX| +vti_cacheddtm:TX|31 Mar 2001 23:42:04 -0000 +vti_cachedlinkinfo:VX|H|index.htm +vti_cachedsvcrellinks:VX|FHUS|library/index.htm +vti_cachedtitle:SR|Hacking Data Compression - Lesson 9 +vti_cachedbodystyle:SR| +vti_cachedhasbots:BR|true +vti_cachedhastheme:BR|false +vti_cachedhasborder:BR|true +vti_botnavbits:SW|SHUB +vti_borderaggregate:SR|default diff --git a/library/_vti_cnf/gshk11.sea b/library/_vti_cnf/gshk11.sea new file mode 100644 index 0000000..2648486 --- /dev/null +++ b/library/_vti_cnf/gshk11.sea @@ -0,0 +1,4 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|31 Jan 2000 05:35:38 -0000 +vti_extenderversion:SR|4.0.2.2717 +vti_backlinkinfo:VX|library/index.htm diff --git a/library/_vti_cnf/index.htm b/library/_vti_cnf/index.htm new file mode 100644 index 0000000..28a11e9 --- /dev/null +++ b/library/_vti_cnf/index.htm @@ -0,0 +1,24 @@ +vti_encoding:SR|utf8-nl +vti_author:SR|fadden +vti_modifiedby:SR|Administrator +vti_timecreated:TR|11 Jan 2000 07:01:48 -0000 +vti_timelastmodified:TR|08 Oct 2002 02:36:28 -0000 +vti_shadowfiles:VX| +vti_filesize:IR|3953 +vti_title:SR|NuLib Library +vti_metatags:VR|HTTP-EQUIV=Content-Language en-us HTTP-EQUIV=Content-Type text/html;\\ charset=windows-1252 GENERATOR Microsoft\\ FrontPage\\ 4.0 ProgId FrontPage.Editor.Document +vti_progid:SR|FrontPage.Editor.Document +vti_generator:SR|Microsoft FrontPage 4.0 +vti_extenderversion:SR|4.0.2.7802 +vti_backlinkinfo:VX|library/ftn.e00005.htm library/old-nufx-spec.htm downloads/index.htm index.htm library/ftn.e00001.htm library/ftn.e08000.htm library/ftn.e08002.htm library/old-binary2-spec.htm library/ftn.e000023.htm library/lesson9.htm +vti_nexttolasttimemodified:TR|31 Mar 2001 23:44:18 -0000 +vti_cacheddtm:TX|08 Oct 2002 03:36:30 -0000 +vti_cachedlinkinfo:VX|H|nufx-addendum.htm H|nulib2-preserve.htm H|FTN.e00001.htm H|FTN.e000023.htm H|FTN.e00005.htm H|FTN.e08000.htm H|FTN.e08002.htm H|Lesson9.htm K|http://www.fadden.com/dl-apple2/ H|old-nufx-spec.htm H|old-binary2-spec.htm H|../downloads/nulib325.tar.gz H|nulib324.tar.gz H|nulib324doc.txt H|nulib303.tar.gz H|nulib22.tar.gz H|nulib21.tar.gz H|nuview.tar.gz H|shrinkit.sdk H|gshk11.sea H|yanksrc.shk +vti_cachedsvcrellinks:VX|FHUS|library/nufx-addendum.htm FHUS|library/nulib2-preserve.htm FHUS|library/FTN.e00001.htm FHUS|library/FTN.e000023.htm FHUS|library/FTN.e00005.htm FHUS|library/FTN.e08000.htm FHUS|library/FTN.e08002.htm FHUS|library/Lesson9.htm NHHS|http://www.fadden.com/dl-apple2/ FHUS|library/old-nufx-spec.htm FHUS|library/old-binary2-spec.htm FHUS|downloads/nulib325.tar.gz FHUS|library/nulib324.tar.gz FHUS|library/nulib324doc.txt FHUS|library/nulib303.tar.gz FHUS|library/nulib22.tar.gz FHUS|library/nulib21.tar.gz FHUS|library/nuview.tar.gz FHUS|library/shrinkit.sdk FHUS|library/gshk11.sea FHUS|library/yanksrc.shk +vti_cachedtitle:SR|NuLib Library +vti_cachedbodystyle:SR| +vti_cachedhasbots:BR|true +vti_cachedhastheme:BR|false +vti_cachedhasborder:BR|true +vti_botnavbits:SW|SHUB +vti_borderaggregate:SR|default diff --git a/library/_vti_cnf/nufx-addendum.htm b/library/_vti_cnf/nufx-addendum.htm new file mode 100644 index 0000000..5ee7d1b --- /dev/null +++ b/library/_vti_cnf/nufx-addendum.htm @@ -0,0 +1,25 @@ +vti_encoding:SR|utf8-nl +vti_author:SR|fadden +vti_modifiedby:SR|fadden +vti_timecreated:TR|16 Jan 2000 01:49:15 -0000 +vti_timelastmodified:TR|26 Sep 2004 18:51:25 -0000 +vti_filesize:IR|34265 +vti_title:SR|NuFX Addendum +vti_metatags:VR|HTTP-EQUIV=Content-Language en-us HTTP-EQUIV=Content-Type text/html;\\ charset=windows-1252 GENERATOR Microsoft\\ FrontPage\\ 4.0 ProgId FrontPage.Editor.Document +vti_progid:SR|FrontPage.Editor.Document +vti_generator:SR|Microsoft FrontPage 4.0 +vti_extenderversion:SR|4.0.2.7802 +vti_backlinkinfo:VX|nufxlibapi.htm library/index.htm +vti_nexttolasttimemodified:TR|26 Sep 2004 18:47:01 -0000 +vti_structuredtm:TR|16 Jan 2000 19:37:14 -0000 +vti_shadowfiles:VX| +vti_cacheddtm:TX|26 Sep 2004 18:51:26 -0000 +vti_cachedlinkinfo:VX|H|FTN.e08002.htm H|http://www.zlib.org/ H|http://sources.redhat.com/bzip2/ H|http://www.fadden.com/ H|http://www.nulib.com/ +vti_cachedsvcrellinks:VX|FHUS|library/FTN.e08002.htm NHHS|http://www.zlib.org/ NHHS|http://sources.redhat.com/bzip2/ NHHS|http://www.fadden.com/ NHHS|http://www.nulib.com/ +vti_cachedtitle:SR|NuFX Addendum +vti_cachedbodystyle:SR| +vti_cachedhasbots:BR|true +vti_cachedhastheme:BR|false +vti_cachedhasborder:BR|true +vti_botnavbits:SW|SHUB +vti_borderaggregate:SR|default diff --git a/library/_vti_cnf/nulib2-preserve.htm b/library/_vti_cnf/nulib2-preserve.htm new file mode 100644 index 0000000..5b9c3c2 --- /dev/null +++ b/library/_vti_cnf/nulib2-preserve.htm @@ -0,0 +1,24 @@ +vti_encoding:SR|utf8-nl +vti_author:SR|fadden +vti_modifiedby:SR|Administrator +vti_timecreated:TR|16 Jan 2000 19:37:01 -0000 +vti_timelastmodified:TR|09 Feb 2003 20:14:39 -0000 +vti_shadowfiles:VX| +vti_filesize:IR|18445 +vti_title:SR|NuLib2's ProDOS Attribute Preservation +vti_metatags:VR|HTTP-EQUIV=Content-Language en-us HTTP-EQUIV=Content-Type text/html;\\ charset=windows-1252 GENERATOR Microsoft\\ FrontPage\\ 4.0 ProgId FrontPage.Editor.Document +vti_progid:SR|FrontPage.Editor.Document +vti_generator:SR|Microsoft FrontPage 4.0 +vti_extenderversion:SR|4.0.2.6513 +vti_backlinkinfo:VX|nulib2-manual.htm library/index.htm +vti_nexttolasttimemodified:TR|09 Feb 2003 20:12:14 -0000 +vti_cacheddtm:TX|09 Feb 2003 20:14:40 -0000 +vti_cachedlinkinfo:VX|H|http://www.fadden.com/ H|http://www.nulib.com/ +vti_cachedsvcrellinks:VX|NHHS|http://www.fadden.com/ NHHS|http://www.nulib.com/ +vti_cachedtitle:SR|NuLib2's ProDOS Attribute Preservation +vti_cachedbodystyle:SR| +vti_cachedhasbots:BR|true +vti_cachedhastheme:BR|false +vti_cachedhasborder:BR|true +vti_botnavbits:SW|SHUB +vti_borderaggregate:SR|default diff --git a/library/_vti_cnf/nulib21.tar.gz b/library/_vti_cnf/nulib21.tar.gz new file mode 100644 index 0000000..b84c038 --- /dev/null +++ b/library/_vti_cnf/nulib21.tar.gz @@ -0,0 +1,4 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|13 Jan 2000 07:49:18 -0000 +vti_extenderversion:SR|4.0.2.2717 +vti_backlinkinfo:VX|library/index.htm diff --git a/library/_vti_cnf/nulib22.tar.gz b/library/_vti_cnf/nulib22.tar.gz new file mode 100644 index 0000000..b84c038 --- /dev/null +++ b/library/_vti_cnf/nulib22.tar.gz @@ -0,0 +1,4 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|13 Jan 2000 07:49:18 -0000 +vti_extenderversion:SR|4.0.2.2717 +vti_backlinkinfo:VX|library/index.htm diff --git a/library/_vti_cnf/nulib303.tar.gz b/library/_vti_cnf/nulib303.tar.gz new file mode 100644 index 0000000..b84c038 --- /dev/null +++ b/library/_vti_cnf/nulib303.tar.gz @@ -0,0 +1,4 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|13 Jan 2000 07:49:18 -0000 +vti_extenderversion:SR|4.0.2.2717 +vti_backlinkinfo:VX|library/index.htm diff --git a/library/_vti_cnf/nulib324.tar.gz b/library/_vti_cnf/nulib324.tar.gz new file mode 100644 index 0000000..ca1c907 --- /dev/null +++ b/library/_vti_cnf/nulib324.tar.gz @@ -0,0 +1,4 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|31 May 1998 22:31:04 -0000 +vti_extenderversion:SR|4.0.2.2717 +vti_backlinkinfo:VX|library/index.htm diff --git a/library/_vti_cnf/nulib324doc.txt b/library/_vti_cnf/nulib324doc.txt new file mode 100644 index 0000000..2465b8c --- /dev/null +++ b/library/_vti_cnf/nulib324doc.txt @@ -0,0 +1,11 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|05 May 2002 21:07:10 -0000 +vti_extenderversion:SR|4.0.2.2717 +vti_cacheddtm:TX|13 Jan 2000 07:43:44 -0000 +vti_filesize:IR|37824 +vti_cachedlinkinfo:VX| +vti_cachedsvcrellinks:VX| +vti_cachedhasbots:BR|false +vti_cachedhastheme:BR|false +vti_cachedhasborder:BR|false +vti_backlinkinfo:VX|library/index.htm diff --git a/library/_vti_cnf/nuview.tar.gz b/library/_vti_cnf/nuview.tar.gz new file mode 100644 index 0000000..b84c038 --- /dev/null +++ b/library/_vti_cnf/nuview.tar.gz @@ -0,0 +1,4 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|13 Jan 2000 07:49:18 -0000 +vti_extenderversion:SR|4.0.2.2717 +vti_backlinkinfo:VX|library/index.htm diff --git a/library/_vti_cnf/old-binary2-spec.htm b/library/_vti_cnf/old-binary2-spec.htm new file mode 100644 index 0000000..8c6baf6 --- /dev/null +++ b/library/_vti_cnf/old-binary2-spec.htm @@ -0,0 +1,20 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|08 Oct 2002 02:33:32 -0000 +vti_extenderversion:SR|4.0.2.5526 +vti_shadowfiles:VX| +vti_cacheddtm:TX|08 Oct 2002 02:34:00 -0000 +vti_filesize:IR|19119 +vti_cachedlinkinfo:VX|H|FTN.e08000.htm H|index.htm +vti_cachedsvcrellinks:VX|FHUS|library/FTN.e08000.htm FHUS|library/index.htm +vti_cachedtitle:SR|Binary ][ Protocol +vti_title:SR|Binary ][ Protocol +vti_cachedbodystyle:SR| +vti_cachedhasbots:BR|true +vti_cachedhastheme:BR|false +vti_cachedhasborder:BR|true +vti_metatags:VR|HTTP-EQUIV=Content-Type text/html;\\ charset=windows-1252 GENERATOR Microsoft\\ FrontPage\\ 4.0 ProgId FrontPage.Editor.Document +vti_progid:SR|FrontPage.Editor.Document +vti_generator:SR|Microsoft FrontPage 4.0 +vti_botnavbits:SW|SHUB +vti_borderaggregate:SR|default +vti_backlinkinfo:VX|library/index.htm diff --git a/library/_vti_cnf/old-nufx-spec.htm b/library/_vti_cnf/old-nufx-spec.htm new file mode 100644 index 0000000..4b463eb --- /dev/null +++ b/library/_vti_cnf/old-nufx-spec.htm @@ -0,0 +1,20 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|08 Oct 2002 02:33:32 -0000 +vti_extenderversion:SR|4.0.2.5526 +vti_shadowfiles:VX| +vti_cacheddtm:TX|08 Oct 2002 02:34:00 -0000 +vti_filesize:IR|48594 +vti_cachedlinkinfo:VX|H|index.htm H|FTN.e08002.htm +vti_cachedsvcrellinks:VX|FHUS|library/index.htm FHUS|library/FTN.e08002.htm +vti_cachedtitle:SR|NuFX Spec Final Rev 3 +vti_title:SR|NuFX Spec Final Rev 3 +vti_cachedbodystyle:SR| +vti_cachedhasbots:BR|true +vti_cachedhastheme:BR|false +vti_cachedhasborder:BR|true +vti_metatags:VR|HTTP-EQUIV=Content-Type text/html;\\ charset=windows-1252 GENERATOR Microsoft\\ FrontPage\\ 4.0 ProgId FrontPage.Editor.Document +vti_progid:SR|FrontPage.Editor.Document +vti_generator:SR|Microsoft FrontPage 4.0 +vti_botnavbits:SW|SHUB +vti_borderaggregate:SR|default +vti_backlinkinfo:VX|library/index.htm diff --git a/library/_vti_cnf/shrinkit.sdk b/library/_vti_cnf/shrinkit.sdk new file mode 100644 index 0000000..9956fa7 --- /dev/null +++ b/library/_vti_cnf/shrinkit.sdk @@ -0,0 +1,4 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|31 Jan 2000 05:35:40 -0000 +vti_extenderversion:SR|4.0.2.2717 +vti_backlinkinfo:VX|library/index.htm diff --git a/library/_vti_cnf/yanksrc.shk b/library/_vti_cnf/yanksrc.shk new file mode 100644 index 0000000..c1088bb --- /dev/null +++ b/library/_vti_cnf/yanksrc.shk @@ -0,0 +1,4 @@ +vti_encoding:SR|utf8-nl +vti_timelastmodified:TR|13 Jan 2000 07:43:34 -0000 +vti_extenderversion:SR|4.0.2.2717 +vti_backlinkinfo:VX|library/index.htm diff --git a/library/gshk11.sea b/library/gshk11.sea new file mode 100644 index 0000000..1849324 Binary files /dev/null and b/library/gshk11.sea differ diff --git a/library/index.htm b/library/index.htm new file mode 100644 index 0000000..d4f7eae --- /dev/null +++ b/library/index.htm @@ -0,0 +1,77 @@ + + + + + + + +NuLib Library + + + +
+ +

NuLib Library
+[ Home ] [ NuLib Downloads ] [ NuLib Library ] [ NuLib2 Manual ] [ NufxLib API ] [ Bugs & Features ]

+
+ +
+ +

 

+

This is a collection of "stuff" related to NuFX and NuLib, mainly +documents and outdated versions of programs.

+

Additional documentation related to NufxLib and NuLib2:

+ +

Technical documentation on NuFX and related subjects:

+ +Source code for recent versions of "classic" NuLib: + +

+Source code for really old versions of NuLib (not necessarily useful):

+ +

ShrinkIt and related software:

+ + 
+ + diff --git a/library/nufx-addendum.htm b/library/nufx-addendum.htm new file mode 100644 index 0000000..bccfba8 --- /dev/null +++ b/library/nufx-addendum.htm @@ -0,0 +1,539 @@ + + + + + + + +NuFX Addendum + + + +
+ +

NuFX Addendum
+[ Home ] [ Up ] [ NuFX Addendum ] [ ProDOS Attribute Preservation ]

+
+ +
+ +
 
+ +
NuFX Addendum - By Andy McFadden - Last revised 2004/09/26
+

This addendum clarifies and extends certain aspects of the NuFX +specification.  This was developed by Andy McFadden, and is not an +"official" modification of the original document.

+

Purpose

+

The NuFX specification defines a very loose structure, and +leaves much to the imagination of the implementer.  For example, "If a +utility finds a redundancy in a Thread Record, it must decide whether to skip +the record or to do something with that particular thread...".  +A strict specification would declare that the situation must never arise, and +define a standard approach for dealing with the anomalous condition.  The +current specification declares that the situation may arise, and +requires the application author to come up with a solution.

+

This document refines the NuFX specification and brings some of +the "fuzzy" areas into sharper focus.  Nothing in this document +contravenes the original document.

+

In the text below, "must" is an imperative that +has to be obeyed, and "should" is a recommendation that authors +are strongly encouraged to follow.

+
+

Clarifications

+

Pronunciation

+

What's the correct way to pronounce "NuFX"?  The +specification doesn't say.  There are two basic camps, letter-by-letter +("en you eff ecks") and minimal-syllable ("new fix" or +"new fuchs").  I don't recall how Andy Nicholas says it, so let +it be "new fix".

+

 

+

Use of ".SDK" suffix

+

Originally, only ".SHK" was used to represent a NuFX +archive.  Over time, a convention of using ".SDK" to represent +archives with a single disk image in them has arisen.  This is very +convenient for emulators on systems that rely on the file extension (e.g. +Windows), so use of ".SDK" is encouraged.

+

 

+

Archives with no records

+

An archive without records, i.e. nothing but a master header +block, serves no purpose.

+

Creating: Archives without any records in them must never +be created.  All archives must have at least one record.

+

Opening: If asked to open a record-less archive, +the application should recognize that the archive is empty and proceed as if it +were a new archive.

+

Modifying: If all records in an archive are deleted, the +archive file must be deleted as well.

+

 

+

Records with no threads

+

A record without threads is pretty pointless.

+

Creating: Records without threads must never be +created.  All records must have at least one thread.

+

Extracting: Empty records should be ignored.

+

 

+

Records with only a filename thread

+

GS/ShrinkIt v1.1 has a bug that prevents it from creating an empty +data thread when asked to add a zero-byte file.  This results in a thread +with a filename and nothing else.  (If it was the first new record added, +it will have an empty comment thread as well.)

There is no valid +reason for deliberately creating such a file. +

Creating: Records composed solely of a filename thread +must not be created. +

Extracting: Records with nothing but a filename thread +should be ignored.  For GSHK v1.1 bug compatibility: if a record has a filename +thread, and no other threads except "message" threads (i.e. no data +threads or control threads), then a zero-byte data fork file should be +created.  Otherwise, the record should be ignored.  If the ProDOS +storage type field indicates an extended file, a zero-byte resource fork should +also be created. +

  +

Records with no filename

+

A record without a filename thread is a curious beast.  +Ideally, there wouldn't be any such thing as a filename thread, since it doesn't +really make sense to have a record without one.  Expanding the record +header to hold a pre-sized buffer would've made many things simpler.

This +particular situation occurred with older versions of ShrinkIt (e.g. v1.1) that failed to store +a volume name when compressing a DOS 3.3 disk.  There was no filename in +the record header, nor one in a filename thread.

The only +situation where a record without a filename makes sense is if the record holds +nothing but comments or other archive "meta data", such as a +"create directory" control thread.

Creating: +Records without filenames must not be created, unless the record is intended +to contain +nothing but archive meta-data.  Deletion of the filename thread should only +be done if a new filename thread is being added.  If data threads are added +to a record without a filename, then a filename thread must be added as well.

Extracting: +If the record contains file data, the application may either prompt the user for +a filename to use, or supply a generated one.

  +

Records with more than one filename thread

+

This is an unusual situation that should only arise if an +application is buggy.  Every record created by a modern application should +have no more than one filename thread.

Creating: Records +with multiple filename threads must not be created.

Extracting: + Applications must use the first filename thread.  If a buggy application wants to +append an additional filename thread, their buggy filename will be ignored.

 

Records +with filenames in two places

+

The old way of storing filenames, used by NuLib and old versions of ShrinkIt, was to +put the filename in the record header.  To facilitate renaming, the +filename was moved into a thread.  Thus, there are two possible locations +where the filename may live, and no guarantee that only one will be used..

Creating: +Never put the filename in the record header when creating a new record.  +It's okay to leave existing records alone, but if an application has the +opportunity to rewrite the record header, the record filename must be removed.

Extracting: +The thread filename takes precedence over the record header filename.

 

File +system separator characters

+

Every record header has a "file system separator" +character ("fssep") in the "file_sys_info" word.  This +is usually something like ':' for GS/OS or '/' for UNIX.  It's necessary to +know what the separator is in order to break a pathname down into its individual +components.

Not all filesystems support subdirectories, however, +which means that not all filenames need to have a separator.  The +appropriate separator character for such a filesystem is not defined in the NuFX +spec.  Clearly it should be something illegal on the source filesystem, or +we could inadvertently see pathnames where they don't exist (e.g. a file called +"foo:bar" on DOS 3.3 if the fssep char were set to ':').

The +trouble is, DOS 3.3 doesn't actually have any illegal characters, just a field +of 30 characters padded with spaces.  Pascal disks are similar.  Since +we must define an fssep for every filename, our best choice is to use '\0' +(0x00), because it's unlikely to occur, and any program that stores names in C +strings will find it awkward to store and scan for '\0'.

This +situation also applies to archived disk images, which must be simple filenames.

(NOTE: +as of v2.0.3, NufxLib rejects 0x00 as an fssep character.  This is a bug.)

Creating: +When adding files directly from filesystems without subdirectories, use 0x00 as +the fssep char.

Extracting: An fssep char of 0x00 means +the pathname is just the filename.

 

Disk +image pathnames

+

While files may have multiple path components (e.g. +"subdir:subdir2:filename"), it makes no sense for disk images to have +them.  The stored filename for a disk is either the disk's ProDOS volume +name, or for non-ProDOS disks, a simple label defined by the user.  Since +the eventual target is a disk device, specifying a subdirectory path makes no +sense.

The issue becomes a little more confusing when storage of +disk images used for emulators is considered.  At first glance, it seems +useful to be able to store a hierarchy of disk images.  In practice, such +images would either be archived as a hierarchy of .PO files, or as an archive of +.SDK archives.

Adding/renaming Applications must +strip any leading path components from disk image "storage names".  +(The NuFX specification does explicitly forbid the use of a filesystem separator +character in a disk volume name.)

Extracting: +Applications extracting directly to a disk must strip leading path components +before assigning the ProDOS volume name.  Applications extracting images to +a file don't need to do anything unusual.

 

Filename +case sensitivity

+

There isn't a "filename is case-sensitive" flag in +NuFX archives.  Since it was designed primarily for ProDOS and HFS +filesystems, neither of which is case-sensitive, we should assume that case is +not meant to be significant when determining whether two records have the same +filename.  This becomes important when adding files (to test for +duplicates), extracting files by name, and when attempting +to display archive contents as a hierarchical tree.

Applications +should try to recognize that "foo/bar", "foo/BAR", and +"FOO/bar" are the same file, but it's probably not worth +"probing" a case-sensitive filesystem like Linux ext2 to guarantee +such.

 

Duplicate filenames

+

There is nothing in the NuFX specification that prevents having +more than one file with the same name in an archive.  In practice, this is +inconvenient, especially for users with command-line tools.  On the other +hand, if the underlying filesystem is case-sensitive, the extracted files may +not actually collide, so it may not make sense for all applications to treat +this as an iron-clad rule.

When comparing names, be sure to take +the filesystem separator character into account.  "foo:bar" could +be a simple filename or a partial pathname depending on whether ':' is the +separator.  Two names should be considered identical if each distinct path +component matches, so "foo/bar" and "foo:bar" are identical +if the separators are '/' and ':', respectively.  Comparisons should be +case-insensitive.

Adding/renaming: Applications +should prevent multiple records from having the same filename. +

 

+

Pre-sized or not pre-sized

+

The specification declares that filename threads and comments +use pre-sized buffers.  It does not define what other members of the +message and filename classes are, which makes it difficult to know what to do +with a request to create a heretofore undefined thread type.  The NuFX +format does not provide any definitive clue as to whether a thread is pre-sized, +so such decisions must be based on the thread class and thread kind.

+

Filename threads and comment threads are pre-sized.  All +other threads are not pre-sized (including other members of the +"filename" and "message" classes).

+

 

+

Proper pre-size for filename threads

+

ShrinkIt allocates a 32-byte pre-sized buffer for the +filename.  If the filename is larger than 32 bytes, the buffer grows to fit +the filename exactly.  If renaming files is considered useful, then the +buffer should always be slightly larger than is needed to hold the +filename.  (Filenames longer than 32 characters are most likely the result +of nested directories, so renaming the file itself is inhibited if the buffer +length is an exact match.) +

Side note: GSHK appears to have a bug where it can't deal with +32-byte HFS filenames (e.g. "foo:abcdefghijabcdefghijabcdefghijxy" +can't be added to an archive).  Emulating this behavior is discouraged. +

Creating: If GS/ShrinkIt compatibility is not important, +all filenames should have at least 8 bytes of free space in the filename thread.  +For GSHK compatibility, the filename thread compThreadEOF must be the greater of +32 and the filename length.

+

Renaming: It is acceptable to have fewer than 8 bytes of +free space remaining after a file is renamed.  However, if the filename +itself exceeds the buffer size and the thread must be rebuilt, the 8-byte +padding should be added.

+

 

+

Thread ordering

+

The NuFX specification does not require that threads appear in +any particular order.  However, writing them in a certain order can make +some operations significantly easier.

+

For example, if an archive is being unpacked as it is received, +it is important to know the filename before receiving the data.  If the +filename thread comes after the data threads, the application has to write the +incoming data into a temp file, and then rename it later when the filename +thread finally shows up.  It would also be nice to be able to display file +comments as the file is being downloaded.

+

Creating: The filename thread must precede all other +threads.  The recommended (but not required) ordering for common thread +types is:

+
    +
  • +

    Filename

    • +
  • +

    Message(s) (i.e. comments)

    • +
  • +

    Data fork

    • +
  • +

    Disk image

    • +
  • +

    Resource fork

    • +
  • +

    all other threads

    • +
+

Extracting: If the filename thread does not appear before +the first data-class thread, the record may be ignored.

+

 

+

Incompatible thread types

+

There are some combinations of threads that must never appear in +a single record.

+

Creating:

+
    +
  • +

    If a data fork is present, the record must not + contain another data fork or a disk image.

    • +
  • +

    If a resource fork is present, the record must not + contain another resource fork or a disk image.

    • +
  • +

    If a disk image is present, the record must not + contain another disk image, a data fork, or a resource fork.

    • +
  • +

    If a control-class thread is present, the record must + not contain any data-class threads.

    • +
+

Extracting: When incompatible threads are found, they +should be ignored in favor of the earlier threads.  For example, if two +data forks are found in the same record, only the first one should be extracted.  +If a data-class thread is found first, subsequent control-class threads should +be ignored, and vice-versa.

+

 

+

Compressed threads

+

Some threads are compressed, some aren't.  The +specification isn't very specific.

+

All data-class threads may be compressed.  All other +classes of threads must not be compressed.

+

 

+

ProDOS storage type

+

The ProDOS storage type has little meaning on most +systems.  However, certain values are significant.

+
    +
  • +

    For records with only a data fork, the storage type + must be one of 0, 1, 2, or 3.  The value "2" is recommended + for applications that don't wish to mimic ProDOS behavior exactly.

    • +
  • +

    For records with a resource fork, the storage type + must be "5" (ProDOS extended file).

    • +
  • +

    For records with a disk image thread, the storage + type must be equal to the disk block size (typically 512).

    • +
  • +

    For records without data-class threads, the storage + type must be "0".

    • +
+

Storage type 0x0d, which is used by ProDOS for directories, must +not be used.

+

It is important to update the storage type as threads are added +and deleted, so that it always accurately reflects the contents of the record.

+

 

+

Disk block size and block count

+

For a compressed disk image, the "storage_type" and +"extra_type" fields take on a different meaning, notably the block +size (typically 512) and block count (e.g. 280 for a 140K disk) of the disk.

+

These fields are more important than you might expect, because +some older versions of ShrinkIt would set the thread EOF to a strange value like +68096 (which, curiously enough, is 133 * 512).  These same versions of +ShrinkIt tended to leave the "storage_type" set to 2.  +Apparently, ShrinkIt just used extra_type * 512 as the uncompressed size when +trying to figure out what sort of disk it had.  An early version of +GS/ShrinkIt went one step further: it used a block count of 280 with a block +size of 256, resulting in archives that apparently held 70K disk images.

+

It is simple enough to disregard the thread EOF value, and +replace the storage_type when it is absurdly small, but there is a deeper +problem.  If you delete a 140K disk image thread and replace it with an +800K disk image thread, the block count stored in the extra_type no longer +accurately reflects the contents of the record.  (This linkage between the +record header and the thread contents is the reason why this document forbids +mixing of disk image threads with any other data-class thread, including other +disk images.)

+

Creating: Applications must update the extra_type +whenever a disk image thread is added.  The value (storage_type * +extra_type) must be equal to the uncompressed size.  The application may +wish to reject threads that are not a multiple of 512 bytes.

+

Extracting: The application must normalize storage_type +to 512 if it is less than 16 (0x0f is the largest possible ProDOS storage +type).  The value storage_type * extra_type must then be used as the +uncompressed size.  If the uncompressed size is zero, the thread may be +ignored.

+

 

+

Access permissions

+

NuFX supports four boolean access permission flags (read, write, +destroy, rename) and two boolean attributes (backup needed, invisible) in the +"access" field.  This matches up with ProDOS capabilities nicely, +but very few other operating systems support all six.

+

Applications authors should consider the following approaches:

+
    +
  1. +

    Preserve all.  All flags in the access field + must be preserved.  It is not required that the extracted files obey + the original semantics -- an "invisible" file might be visible, + and a file with "rename" disabled might still be rename-able -- + but when the files are re-added, the permissions must match.

    2. +
  2. +

    Locked/unlocked.  A file with read enabled, and + write, destroy, rename, and invisible disabled, is considered + "locked" (access 0x01 or 0x21).  All other files are + considered "unlocked".  When a file is extracted and then + added to an archive, the locked/unlocked status must be preserved.  + Locked files are added with access 0x21, and unlocked files are added with + access 0xe3.

    3. +
+

It is acceptable for an application to find a middle ground +between these two, and preserve more of the flags accurately than approach #2 +does, but approach #2 should be considered the minimum acceptable level of +support.

+

 

+

Empty directories

+

Directories do not need to be stored explicitly unless they are +empty.  The NuFX specification manages to avoid describing how directories +are actually supposed to be stored, saying only: "A Thread Record must exist to inform a utility that a directory is to +be created through the use of the proper control_thread value."

+

What is in a "create directory" control thread?  +It appears that the intent was to have the thread contain the pathname that +needed to be created.  In theory, you could have several of these things, +and create an entire hierarchy from a single record.  Such threads should +not be compressed, but their compThreadEOF should always match their threadEOF +(i.e. they're not pre-sized).

+

It's a little tricky to say, "add a control thread whenever +you find a directory with nothing in it".  What if the directory has +files in it, but you don't have the access permissions necessary to read the +files?

+

Does such a record require a filename?  Probably not.  +However, if it doesn't have a filename, ShrinkIt might not display the record, +and you'd have no way to manipulate it.  Adding a "record label" +is easy and useful.

+

(I'm strongly tempted to punt on the control threads and just +use storage type 0x0d to indicate that a directory should be created.  This +is in direct opposition to the NuFX specification, however, so I'm reluctant to +do so.)

+

Creating: Applications not interested in preserving empty +directories need do nothing.  Otherwise, the application must add a +"create directory" control thread whenever a directory is encountered +for which no files are added to the archive.

+

Extracting: A directory must be created when a control +thread is present.  As noted in the NuFX specification, the application +must also create any directories listed in the record's pathname that don't yet +exist.

+

 

+

Message thread format

+

The specification says that message threads are ASCII text, but +doesn't specify an EOL character.  For the benefit of Apple II utilities, +it's best to use a carriage return (ctrl-M).

+

Creating: Convert any EOL markers to CR.

+

Extracting: Assume that the comment may be using CR, LF, +or CRLF, and convert as needed for display.  GS/ShrinkIt used a +proportional font, so there is no need to worry about "ASCII art" in +comments.

+

 

+

GS/OS option lists

+

Files archived from HFS AppleShare volumes come with +"option lists", a GS/OS feature that provides a way for non-ProDOS +filesystem information to be preserved.  GS/ShrinkIt tries to save this +information, but it doesn't seem to do a very good job.  It appears to drop +a big chunk of the data without altering the size (e.g. the size field says 36 +bytes, but there's only space for 18 bytes in the record header).

+

GS/ShrinkIt seems to work correctly whether the option list size +is correct or not, so other applications should do the same.

+

Opening: Assume the option_size field is correct +unless it exceeds attrib_count-2.  If it's too large, clip it down to size.

+

Updating: Always use the actual size.  Do not +propagate incorrect values.  Discarding existing option lists is +discouraged but allowed.

+

 

+

Master EOF

+

For the most part, ShrinkIt correctly sets the MasterEOF field +in the Master Header block.  A very old version of ShrinkIt left it set to +zero (this is the same version that completely omitted the filename for DOS 3.3 +disk images).  GS/ShrinkIt appears to initialize it to 48 (the size of the +MH block), and if the creation process is interrupted you can end up with a +partial archive with a nonzero EOF.

+

Opening: Accept a MasterEOF of zero, but reject a +MasterEOF of 48.  Don't assume the MasterEOF is accurate.

+

Updating: Applications must write the correct MasterEOF +value if an archive is modified.

+
+

Extensions

+

Unofficial extensions to the NuFX specification.  Anyone +working with NuFX archives should take heed.

+

New compression formats

+

Thread formats 0x0000 through 0x0005 are already defined.  The +following thread format values have been added:

+
    +
  • +

    0x0006 - deflate.  The thread contains data conforming + to RFC 1951 (deflate1.3 specification).  A more practical way of + putting it is it contains exactly the data that zlib v1.1.4 outputs.  Visit http://www.zlib.org/ + for more details.

    • +
  • +

    0x0007 - bzip2.  The thread contains BWT+Huffman + compressed data as output by Julian Seward's "libbz2" v1.0.2.  Visit + http://sources.redhat.com/bzip2/ + for more information.

    • +
+

Support for these formats is nonexistent on the Apple II, so +they should not be used except in situations where compatibility is unimportant +(e.g. collections of disk archives for use with A2 emulators).

+ +

I found that "deflate" generally does as well or +better than "bzip2" on Apple II binaries, disk images, and small text +files.  Deflate is also faster and uses less memory, and you're more likely +to find libz installed on a given system than you are libbz2  For these +reasons, use of deflate should be encouraged in favor of bzip2.

+ +
+

NuFX Quirks

+

This section identifies some quirks in NuFX or ShrinkIt that, +while not bugs, are worth noting.

+

Filename separator character

+

Originally, the filename was stored in the record header, so it +made sense that the filename separator character ("fssep char") should +also be there.  When the filenames were moved into threads, the fssep char +got left behind.  If a record has two filenames, they'd better have the +same fssep char, or interpreting one of them will be impossible.  (This is +one of the reasons why it's important to clearly define which filename takes +precedence in all circumstances.)

+

Files with zero or two CRCs

+

The "threadCRC" field in the thread header block can +have one of three meanings: nothing (v0, v1), the CRC of the compressed data +(v2), or the CRC of the uncompressed data (v3).  The version 2 meaning +wasn't used in anything significant, and can be ignored.

+

Version 1 records generally have threads compressed with LZW/1 +data.  The LZW/1 compression format includes a 16-bit CRC at the start of +the thread.  Version 3 records generally have threads compressed with LZW/2 +data, which does not include a CRC.

+

Applications like P8 ShrinkIt and NuLib creation v1 records and +compress with LZW/1, while GS/ShrinkIt and NuLib2 create v3 records and compress +with LZW/2.  This means that each compressed thread has exactly one CRC.  +So what happens if you tell NuLib2 to create a new record with +LZW/1, or tell it to add a new LZW/2 thread to an existing v1 record?

+

In one case, you end up with two CRCS; in the other, you end up +with no CRC on your data at all.  For some bizarre reason, the v3 thread +CRC is computed with a different initial value, so it is necessary to compute +the CRC twice, not merely store the same value twice.

+

Please select your compression methods appropriately.  +Also, bear in mind that uncompressed data stored with P8 ShrinkIt has no CRC +whatsoever.

+

Extra data in compressed threads

+

ShrinkIt adds an extra byte at the end of all LZW compressed +data, probably due to an off-by-one bug in the compression code.  It turns +out that it's possible to get even more "extra" bytes at the end.

+

ShrinkIt's LZW-I algorithm always operates on a 4K buffer, +largely because it was originally designed for compressing 5.25" disks with +4K tracks.  +On small files, or at the end of a large one, the last bit of data is padded out +to 4K and then compressed.  Ordinarily this is barely noticeable, because +the compression routines do an RLE (Run-Length Encoding) pass before applying +LZW.

+

However, if both RLE and LZW fail to make the 4K block any +smaller, it is stored without compression.  This means the whole 4K, +complete with padding, gets written to the archive.  This doesn't cause any +problems, but can make you wonder where all the extra bits came from.

+

The SQ compression algorithm, as implemented by Don Elton's SQ3, +appears to add an extra 0xff to the end of the compressed data.  It can +safely be ignored.

+

Preserving BXY and SEA wrappers

+

Preserving BXY wrappers is pretty easy, since the Binary II +format is well documented.  Updating block counts and file lengths is all +that is required.

+

Preserving SEA wrappers is a little harder, since (as far as I +can tell) there is no documentation on the format.  A little +experimentation shows that the SEA header is always 12005 bytes long, and the +only part that changes from file to file is a short piece right before the NuFX +archive begins.

+

It is necessary to update the file length in three different +places, all right next to each other, one of which is offset by 64 bytes.  +I would guess the header allows for more than one archive to be present, but +since such things have never actually been created, the possibility can be +ignored.

+

Y2K

+

The NuFX standard says that the Date/Time format is the same as +that returned by the IIgs ReadTimeHex toolbox call.  That call returns the +year as (year - 1900), so the year 2000 is stored as "100".  +ProDOS 8 clock drivers, on the other hand,  return 40-99 for 1940-1999, and +0-39 for 2000-2039.  As a result, archives created with P8 ShrinkIt use 0 +for the year 2000 instead of 100.

+

When creating archives, always use 100 for the year 2000, but +also accept the year 0.  However, if you find a Date/Time with zero in all +useful fields (second, minute, hour, day, month, year), treat it as an +unspecified date rather than midnight of January 1, 2000.

+
+

This document is Copyright © 2000-2004 by Andy +McFadden.  All Rights Reserved.

+

The latest version can be found on the NuLib web site at +http://www.nulib.com/.

+
+ + diff --git a/library/nulib2-preserve.htm b/library/nulib2-preserve.htm new file mode 100644 index 0000000..0e39805 --- /dev/null +++ b/library/nulib2-preserve.htm @@ -0,0 +1,389 @@ + + + + + + + +NuLib2's ProDOS Attribute Preservation + + + +
+ +

ProDOS Attribute Preservation
+[ Home ] [ Up ] [ NuFX Addendum ] [ ProDOS Attribute Preservation ]

+
+ +
  +
NuLib2's ProDOS Attribute Preservation - By Andy McFadden - Last revised +2003/02/08
+

This document describes how NuLib2 preserves file types and identifies +resource forks and disk images when such things aren't handled by the filesystem. +

  +

+File Type Preservation

+

+The overriding goal is to provide a way to preserve filetypes and auxtypes +when extracting files to "typeless" filesystems like those supported by +UNIX or Windows. A secondary goal is to make the preservation attractive. +As it turns out, these goals tend to conflict. +

+First, a simple example of a ProDOS text file named "fubar". Here's a +trivial way of preserving the file type when extracting the file from an +archive: +

Archive      :  FUBAR           TXT $0000
+Extract to   :  FUBAR.TXT
+
+ +When adding files to the archive, we'd just do the opposite: +
Original     :  FUBAR.TXT
+Rearchive to :  FUBAR           TXT $0000
+
+ +This works out pretty well under Windows, since "fubar.txt" is recognized with +the correct file type.  (It might get confused by the carriage returns, but +that's a different problem.)  If we happened to find a file called "fubar.txt" +that didn't come from an archive, we still do the right thing, and store it as a +file with type "TXT".  All well and good. +

Now suppose we have an auxtype that we don't want to lose. We have to +make things a little more ugly. +

Archive      :  FUBAR           TXT $0100
+Extract to   :  FUBAR.TXT#0100
+
+This isn't going to open with a double-click under Win95, but at least +we're not losing the type. +

+Now imagine we have something that doesn't use a standard type, like: +

Archive      :  FUBAR           LBR $8002
+Extract to   :  FUBAR.SHK
+Rearchive to :  FUBAR           LBR $8002
+
+We happen to know that $E0 (LBR) with auxtype of $8002 is a ShrinkIt +archive. So, when we extract it, instead of making it FUBAR.LBR#8002, we +change it to FUBAR.SHK. When we archive such a file, we apply the same +process in reverse. We don't *have* to do this, but it certainly makes +the results more attractive, and would allow a Windows-based ShrinkIt +application to identify the file. +

+Now things start to get a little ugly. Suppose, like most ShrinkIt +archives, it already ends with ".SHK"? Now we have: +

Archive      :  FUBAR.SHK       LBR $8002
+Extract to   :  FUBAR.SHK.SHK
+Rearchive to :  FUBAR.SHK       LBR $8002
+
+This is annoying, but it won't stop anything from working (unless the file +extension is too long!). The alternative would be to realize that there's +already a ".SHK" extension on the file, and not add another one, but then +when we went to rearchive it we'd end up with something different: +
Archive      :  FUBAR.SHK       LBR $8002
+Extract to   :  FUBAR.SHK
+Rearchive to :  FUBAR           LBR $8002
+
+We've lost the file extension.  For a ShrinkIt archive this wouldn't be so bad, but for a library or +executable launched with a hardcoded path ("foo.s16") it could be fatal. +


+In some cases we just want to be "nice" and put file types on things +that weren't extracted from a ShrinkIt archive. For example, suppose +we're archiving a bunch of source code ("foo.c" and "foo.h"). We can +give them specific file types, e.g. the APW "SRC" type $b0/$000a. We +can't convert back from those types though, since *.c and *.h are +both $b0/$000a. With .txt files we could strip off ".txt" and give them +a unique type, but with source files we have to leave ".c" and ".h" on +them. +

+The situation gets more confusing when we re-extract the files from the new +archive. If their types are NON/$0000, then they will get extracted as +"foo.c" and "foo.h". If we were nice and gave them file types, then when +we extracted them from the new archive they'd come out with preserved file +types, named "foo.c.SRC#000a" and "foo.h.SRC#000a". We may actually make +things more ugly by trying to be nice! +

+There are also cases where we may want to be "mean" and lose information, +such as when extracting a BIN file called "foo.gif" or "foo.jpg". In most +cases, these are GIF or JPEG images that should not have type information +appended. Storing the file as "foo.gif.BIN" is counterproductive if we +want to use the file, but it's the right thing to do if we want to +re-archive the files in the same way that we extracted them. +


+One other bit of difficulty arises if the archiver application gets +updated. Maybe a file type was misnamed, so what used to be type "AST" +becomes "AJT". Now, when we try to add "FUBAR.AST#0100", we don't recognize +the file type. To avoid problems recognizing file types written by older +versions of NuLib2, we always want to use the numeric file type values. However, +this prevents us from ever being able to double-click on an extracted file in +Windows, unless we set up mappings for the numeric types (e.g. associate +"$04" with the same thing ".TXT" uses). +

+Bill North gave me some interesting ideas about how to preserve the +file type and still keep extension-oriented operating systems like Windows +happy. The format proposed below is based largely on his ideas. +


+There are three levels of file type preservation: +

+
None (equivalent to the original NuLib):
+
+ When extracting, no file type information is stored in the name extension.
+
+ +
+ When adding, file type information in the extension is ignored (in fact, + it's regarded as part of the filename).
+ +
+
+
Basic (preserves reliably):
+
+ When extracting, all files have their type and auxtype appended at the + end of the filename, in hexadecimal. "fubar.txt" becomes "fubar.txt#040000".  + Resource forks and disk images are annotated with + single-letter codes.
+
+
+ When adding in "basic" mode, all files are checked for file type + information, and (if found) everything after the last '#' is removed. + If a full type isn't found ("foo.c"), the file is added as NON/$0000. + Care is taken to treat files like "blah#123" and "foo#040000xyz" as + typeless, so we don't get confused by files that legitimately have a '#' in + the filename. +
+
+
+
Extended (preserves reliably, works better with Windows)
+
+ This works like "basic", but a redundant file extension is added to + the filename. "fubar.txt" becomes "fubar.txt#040000.txt". Special + care is taken to preserve existing extensions, so "foo.c" would become + "foo.c#b0000a.c", not "foo.c#b0000a.src". If no extension is present + on the original, and no ProDOS three-letter extension is known + (e.g. $f7), then no redundant extension is added.  Type TXT is + special-cased, so text files are always ".TXT".
+
+
+ Adding of preserved files works like "basic" mode, where everything after the last '#' + is removed. The redundant file extension is simply ignored.  If a file + was not preserved, but it has a + file extension, an attempt is made to determine the file type based + solely on the extension (e.g. "fubar.jpeg" gets stored as BIN rather + than NON). +
+
+

Examples

+
Extracting "fubar", type=TXT, auxtype=$0000
+  none:     fubar
+  basic:    fubar#040000
+  extended: fubar#040000.txt
+
+Extracting "fubar.txt", type=TXT, auxtype=$0000
+  none:     fubar.txt
+  basic:    fubar.txt#040000
+  extended: fubar.txt#040000.txt
+  
+Extracting "fubar.doc", type=TXT, auxtype=$0000
+  none:     fubar.doc
+  basic:    fubar.doc#040000
+  extended: fubar.doc#040000.txt
+
+Extracting "fubar.doc", type=BIN, auxtype=$0000
+  none:     fubar.doc
+  basic:    fubar.doc#060000
+  extended: fubar.doc#060000.doc
+
+Extracting "fubar", type=S16, auxtype=$0100
+  none:     fubar
+  basic:    fubar#b30100
+  extended: fubar#b30100.s16
+
+Extracting "fubar.gif", type=BIN, auxtype=$2000
+  none:     fubar.gif
+  basic:    fubar.gif#062000
+  extended: fubar.gif#062000.gif
+
+Extracting "fubar.c", type=SRC, auxtype=$000a
+  none:     fubar.c
+  basic:    fubar.c#b0000a
+  extended: fubar.c#b0000a.c
+
+Extracting "fubar", type=LBR, auxtype=$8002
+  none:     fubar
+  basic:    fubar#e08002
+  extended: fubar#e08002.lbr
+
+Extracting "fubar.shk", type=LBR, auxtype=$8002
+  none:     fubar.shk
+  basic:    fubar.shk#e08002
+  extended: fubar.shk#e08002.shk
+
+
+
Adding file "fubar"
+  none:     fubar/NON/$0000
+  basic:    fubar/NON/$0000
+  extended: (same as basic)
+
+Adding file "fubar.txt"
+  none:     fubar.txt/NON/$0000
+  basic:    fubar.txt/NON/$0000
+  extended: fubar.txt/TXT/$0000
+
+Adding file "fubar#B30100"
+  none:     fubar#B30100/NON/$0000
+  basic:    fubar/S16/$0100
+  extended: (same as basic)
+
+Adding file "fubar.c"
+  none:     fubar.c/NON/$0000
+  basic:    fubar.c/NON/$0000
+  extended: fubar.c/SRC/$000a
+
+Adding file "fubar.gif"
+  none:     fubar.gif/NON/$0000
+  basic:    fubar.gif/NON/$0000
+  extended: fubar.gif/PNT/$8006
+
+Adding file "fubar.gif#060000.txt"
+  none:     fubar.gif#060000/NON/$0000
+  basic:    fubar.gif/BIN/$0000
+  extended: (same as basic)
+
+Adding file "fubar.shk#045678.s16-wahoo"
+  none:     fubar.shk/TXT/$5678
+  basic:    fubar.shk/TXT/$5678
+  extended: (same as basic)
+
+ +

+Files extracted in either "basic" or "extended" mode can be re-added in +"basic" mode. Files extracted in "none" mode shouldn't be re-added if you +care about file types. Files that didn't originate from a NuFX archive, +such as text files or source code on disk, can be added in "extended" +mode if you'd like to have NuLib2 guess at their file types. + +

+Because GS/OS supports the HFS filesystem, we may have items in an +archive that have full Macintosh HFS types rather than ProDOS types. +If the file type is larger than 0xff, or the auxtype is larger than 0xffff, +then the type will be a 16-digit hex value (#1234567812345678) instead of +the usual 6-digit value. This may strain the limits on some filesystems, +so preserving the types of Mac files may not be practical everywhere. +

 

+
+

+Special Characters and Long Names

+

+Filesystems don't generally allow every possible byte value to be included +in a filename. The typical UNIX filesystem is very forgiving, but it +won't allow '/' or '\0'. Win32 won't accept \/:*?"<>| . If we are to +preserve the filenames as well as the filetypes, we have to provide a +way to include special characters. ProDOS only uses A-Z, 1-9, and '.', +so preserving special characters may not be possible. +

+Some filesystems, such as MS-DOS and ISO-9660 (level 1), restrict the +filename format as well as the character set, e.g. names limited to +"8.3" form. It's not generally possible to preserve complex names on +such systems, so we don't even try. Hybrid CD-ROMs can be created with +Joliet, Rock Ridge, and HFS filenames, so the appropriate target system +can see the correct name. (Of course, stuff written to a CD-ROM should +be inside an SHK archive anyway, not expanded into separate files.) +

+In the "none" preservation mode, filenames will be converted into something +acceptable for the target filesystem. No effort will be made to create +something that can be converted back. When files are added in the "none" +mode, no conversion will take place. +

+In "basic" and "extended" modes, characters invalid on the current +filesystem will be written as "%xx", where "xx" is the two-digit hex +value for the character. If the '%' character appears in a filename, +it will be stored as "%%".  The "%00" sequence, added in some +unusual circumstances, should be removed entirely rather than converted to '\0'.

Character +preservation shouldn't often be necessary, unless the files were archived +from an HFS or UNIX volume, and the archive creator used characters like "/" or +"*". Win32, HFS, and UNIX can all handle the short names and restricted +set of characters that ProDOS filesystems support. +


+Another situation where filenames can be twisted is when they are too +long to fit on a filesystem. The character escaping and addition of type +information can make a filename much longer than it was originally, so +a name that was kinda long before will be really long when it's extracted. +

+In the "none" mode, filenames will be truncated silently. In the "basic" +and "extended" modes, an error will be returned, and you will be given +the opportunity to skip or rename the file. +


+Another problem area has to do with the path separators.  Consider a file +named "foo/bar" in a folder called "subdir" on an HFS +volume.  It would be archived as "subdir:foo/bar".  When +extracted to a UNIX volume, you would get a file called "foo%2fbar" in +"subdir".  When added back to an archive, however, if '/' is used +as the path separator, you would get "subdir/foo/bar", which is not +what was intended.  Similar examples can be created for other pathname +separators.

In general, restoring a filename to its original status requires +encoding not only the special characters but also the path separators.  +Ideally the gunk added to the filename would include some indication, either an +enumerated value or a two-digit hex ASCII value.  In practice, ':' is +illegal on all Apple II filesytems (except DOS 3.3) as well as Win32, so using +it as the default path separator should work well.  Only files created on a +UNIX system will have problems, and these can be screened (replacing ':' with, +say, 'X').

Since NuLib2 isn't intended to be a general-purpose file +archiver, there's not much need to support all possible UNIX filenames.  +There's little advantage to adding an additional character to every filename for +this rare case. +

+  +

+

+Resource Forks, Disk Images, and Comments

+

+A forked file "FINDER.SYS16" with filetype S16/$0100 would be extracted +into "FINDER.SYS16#b30100" and "FINDER.SYS16#b30100r". The "r" is +added in both "extended" and "basic" modes, but as with everything else +is unused in "none" mode. This used to result in "file already exists, +overwrite?" messages when the resource fork was extracted, because both +the data and resource forks will be written to "FINDER.SYS16".  The current +version of NuLib2 appends the rather obvious "_rsrc_" to resource +forks in "none" mode. +


+The earlier discussion on file type preservation has meaning for disk +archive preservation as well. In general, people don't combine file and +disk archives, or have more than one disk image in an archive, but there's +nothing in the NuFX format that prevents it. It is useful to transparently +handle disk images as well. +

+The trouble is with identifying disk image files as such. Formats with +unique extensions, such as 2IMG (.2MG) are fairly safe, but a raw disk +image entitled "system.raw" could be confused with other forms of data. +This can make it tricky to do the right thing. +

+The presence of an explicit "this file is a disk" option, which treats all +files as disk images no matter what they're called, guarantees that we can +always do *something* useful with a disk image file. Even when this option +isn't being used, we can identify .2MG files by the extension and (to be +rigorous) the file contents. Extracting and re-adding a .2MG file multiple +times shouldn't result in any degradation, unless we try to convert the +sector interleave from DOS to ProDOS, but even that is a reversible +transformation. +

+The explicit flag for a disk image works similarly to the flag for a +resource fork. After the type info, which for a disk is always $00 with +the number of blocks in the auxtype, we add 'i'. A 5.25" disk image +stored as "SYSTEM" would be extracted in "none" mode as "SYSTEM", and in +"basic" or "extended" mode as "SYSTEM#000118i". +

+No flag is added for a data fork. If a flag were added, it probably +wouldn't be 'd', since that could be confused with "disk" and also happens +to be a valid hexadecimal digit. +

+

Comments are another special case.  Preserving archive comments requires +extracting them into separate files.  NuLib2 doesn't currently do this, but +if it were to do so the file would look like "SYSTEM#0000c8n", where +0x00c8 is the pre-allocated size for the comment thread.  I'm using 'n' as +the comment designator (for "note") because 'c' is a valid hexadecimal +digit. +

+
+

This document is Copyright © 2000-2003 by Andy +McFadden.  All Rights Reserved.

+

The latest version can be found on the NuLib web site at +http://www.nulib.com/.

+
+ + diff --git a/library/nulib21.tar.gz b/library/nulib21.tar.gz new file mode 100644 index 0000000..b8cbf03 Binary files /dev/null and b/library/nulib21.tar.gz differ diff --git a/library/nulib22.tar.gz b/library/nulib22.tar.gz new file mode 100644 index 0000000..de31cfe Binary files /dev/null and b/library/nulib22.tar.gz differ diff --git a/library/nulib303.tar.gz b/library/nulib303.tar.gz new file mode 100644 index 0000000..9764571 Binary files /dev/null and b/library/nulib303.tar.gz differ diff --git a/library/nulib324.tar.gz b/library/nulib324.tar.gz new file mode 100644 index 0000000..e4b2c6e Binary files /dev/null and b/library/nulib324.tar.gz differ diff --git a/library/nulib324doc.txt b/library/nulib324doc.txt new file mode 100644 index 0000000..8d64e2e --- /dev/null +++ b/library/nulib324doc.txt @@ -0,0 +1,889 @@ + + +NuFile eXchange (NuFX) Archive Utility +NuLib v3.2 Documentation + +By Andy McFadden + + + +Updated September 23, 1992 + + + + + + + + + + + + + +Overview +-------- + +NuLib is a shell-based NuFX archive utility, based loosely on "ARC" +for the IBM PC and "ar" under UNIX. It allows you to perform certain +operations on the same archives used by ShrinkIt, including view +archive contents, add to archive, extract from archive, and delete +from archive. In addition, it will list and unpack files from Binary +II archives. + +This program is primarily targeted at users of non-Apple II computer +systems, from IBM PC compatibles to Sun workstations to Amdahl +mainframes. It will also be of use to people who use a GS/OS shell on +the Apple //gs, such as APW, ORCA, or ECP-16 (although it is +considerably slower than ShrinkIt). NuLib may require more than 256K +of free RAM to function properly. Apple //gs users can use YankIt +instead, which is much smaller and many times faster (though +considerably less powerful). + + +Background +---------- + +For our purposes, an archive is simply a collection of one or more +files stored as a single file. This allows groups of related files to +be stored under a single filename, reducing directory clutter, and +makes it possible to transfer groups of files without the use of a +batch transfer utility. + +To reduce the space required to store the archives and the time it +takes to transfer archives, most popular archiving programs +automatically compress files as they are stored. Two popular +compression methods are Huffman (variable-size codes are used, with +smaller codes assigned to bytes that appear frequently) and +Lempel-Ziv-Welch (LZW; finds common substrings in the file). + +Popular archiving programs include "BLU" (Apple II), "ShrinkIt" (Apple +II), "GS/ShrinkIt" (Apple IIgs), "ARC"/"PKARC" (MS-DOS), "PKZIP" +(MS-DOS), "lharc" (MS-DOS/Amiga), "zoo" (MS-DOS/Amiga), "PackIt" +(Macintosh), "StuffIt" (Macintosh), "tar" (UNIX), and "ar" (UNIX). +UNIX archivers don't generally compress files; a separate program +(called "compress" of all things) is usually used. NuLib works with +archives that ShrinkIt produces (NuFX). + + +New Features/Bug Fixes +---------------------- + +Major changes since last release: + ++ Compression speed improvement ++ Can add a file as if it were a disk image + + +How to Use NuLib +---------------- + +Usage: nulib option[suboption] archive [filespec1] [filespec2] [...] + +"option" is a single character, as specified below. +"suboption" is one or more characters, which modify the performance of +the option (see each specific option for details). +"archive" is the name of a NuFX archive. It doesn't need to exist for +the add, create, and move options. +"filespecN" is the name of a file, either on disk or in the archive. +More than one file may be named on the command line. + +The option/suboption string may be entered in upper or lower case, and +may be preceeded by a hyphen (ex: "nulib -XT0 ..." is equivalent to +"nulib -xt0 ..." and "nulib xt0 ..."). Suboptions may be entered in +any order. + +Side note: Under APW, both the archive filename and all file +specifications relating to files ON DISK are expanded by NuLib (device +names, wildcards, etc). Under UNIX, all filename expansion and +wildcards are handled by the shell or the kernal. The filenames of +files in the ARCHIVE do not undergo wildcard expansion on any system. + +Examples: +$ nulib av foo.shk file1 subdir/file2 +Adds "file1" and "subdir/file2" to "foo.shk", listing each file as it +is archived. +$ nulib av foo.shk subdir +Recursively adds all files in subdir and all files in subdirectories +of subdir. +$ nulib mv foo.shk =.doc (APW) +$ nulib mv foo.shk *.doc (UNIX) +$ nulib mv foo.shk *.doc (MS-DOS) +Adds all files with the suffix ".doc" to the archive "foo.shk", +listing them as it goes, then deletes them from the disk. +$ nulib d+ foo.shk work/ +Deletes all files in "foo.shk" that start with "work/"; this would +likely be used to delete all files archived from the directory "work". +No output is produced. + + +The various options are: + +*** Add/Append files: + nulib A[V][U|C|S][R][F[/]][D] archive files +Quickly appends the specified files to the end of the archive. This +option does not scan the archive to see if the files already exist, +and will create a new archive file if one is not found. This option +corresponds roughly to N)ew archive or A)dd files on the ShrinkIt +menu. + +If you add files from the current directory or a subdirectory of the +current directory, then the entire partial pathname is stored (ex: +"nulib av fubar.shk foo/bar/filename" will store "foo/bar/filename" in +the archive "fubar.shk.") If you add files from directory that is not +a subdirectory of the current directory, only the filename will be +stored (ex: "nulib av fubar.shk ../zip/bang/filename" will store +"filename" in the archive). Unfortunately, this does not work under +UNIX, since filenames are expanded by the kernel instead of by NuLib. + +Wildcards are allowed, and subdirectories are recursively descended +(the depth is limited only by the maximum length of a pathname and the +number of files in the subdirectories), unless the R suboption is +used. The R suboption will prevent subdirectories from being +processed, so that "nulib avr fubar.shk *" under UNIX will add all +files in the current directory, but will not descend into any +subdirectories. The V suboption displays the filenames as they are +added. + +The U suboption adds a file without compressing it. The C suboption +allows you to insert a file in the archive with a specific compression +algorithm (specified by the argument "value", which must immediately +follow the suboption - no spaces between. Ex: "nulib ac5v fubar.shk +file1" would store "file1" in "fubar.shk" using 16-bit UNIX-style LZW +compression). "value" corresponds to the appopriate thread_format as +specified in the section on compression methods. + +The S suboption does not compress the file, but will store it as if it +were by setting the thread_format field to "value". One use for this +is adding compressed files without having to uncompress them first +(ex: "nulib as1 foo.shk file1.qq" adds a squeezed file). This +suboption could conceivably cause a NuFX extractor to crash; please +don't use it unless you are sure of what you are doing. Note also +that some compression methods (like ShrinkIt LZW) require both the +"compressed length" and "uncompressed length" values to be correct; +use of this suboption will (incorrectly) set them both to the current +file length. + +Only one of U, C, or S may be specified. + +The F suboption allows specification of the filetype attribute. It +should be three characters long, and will be matched against a list of +standard ProDOS file types. Common ones are "BIN" (binary), "TXT" +(text), "SRC" (source code), "OBJ" (object code), "EXE" (executable), +and "NON" (typeless, the default). Both upper and lower case letters +will match. + +If the filetype is followed by a "/", then the auxtype field will also +be set. NuLib expects the auxtype to be a four-byte hexadecimal +number; entering less than four digits could cause the auxtype to be +misinterpreted. A good example is storing APW C source files: "nulib +cvfSRC/000a nulib.shk file1.c file2.c ...". + +If the D suboption is given, the entry will be tweaked to make it +appear that the file is actually a disk image. This may be useful +somewhere. Don't use it if you don't know what you're doing. + +Adding comments is not currently supported. + + +*** Binary II Operations + nulib B[X][V][T][I] binary2-archive [archived-files] +This option is included for compatibility with the older Binary II +standard, since it is still in use by many communications programs. + +With no suboptions, this will list the information on the archived +files specified by "archived-files". The X suboption extracts the +named files from the archive, unsqueezing them if necessary. If the +archived-files parameter is omitted, then all files will be listed or +extracted. Note that matches are case-independent. + +When the V suboption is used in conjunction with the X suboption, the +names of the archived files are printed as they are extracted. The V +suboption has no effect otherwise. + +The T suboption will perform a text substitution as it works; see the +section of conversion of line terminators for more information. + +If the filename of an extracted file would conflict with an existing +file, the file will be overwritten and a message will be printed. If +the "I" suboption is used, then NuLib will give an "overwrite (y/n)?" +prompt. An affirmative answer overwrites the file; a negative answer +will skip the file and continue processing. + +Attempting to perform NuFX operations on a Binary II archive will +fail. If the file appears to be Binary II format, then a message +indicating this will be printed. Providing transparent support for +Binary II archives is not impossible, but isn't needed often enough to +be worth doing. + + +*** Create Archive + nulib C[V][U|C|S][R][F[/]][D] archive files +This is identical to the A option, but the "creating archive" message +is suppressed. This behavior is similar to "ar". + + +*** Delete from Archive + nulib D[V][+] archive archived-files +Deletes the named files from the archive. NuLib scans the archive, +marking all records that match the names in the file specification +(case-independent). If all files are marked for deletion, then the +archive file itself is deleted. Otherwise, a new archive is created, +the unmarked records are transferred, and the old archive is deleted. +An error during this process will leave the original archive +unmodified. + +Note that this does not require an exact match if the "+" suboption is +used; "nulib d+ fubar.shk foo" will delete "foo", "Foozle", +"food/recipies" and "food/stock." The V suboption prints the list of +marked records as it works. + + +*** Freshen Archive + nulib F[V][U|C|S][R][F[/]][D] archive files +Updates files in the archive, but doesn't add any new files. Creates +a new archive file, and either transfers the old record or adds the +file depending on which is more recent. Only exact filename matches +are allowed (case-independent), including partial paths. The archive +being updated must already exist. + +Wildcards are allowed, and subdirectories are automatically expanded +unless the R suboption is used. The V suboption displays the +filenames as they are added. The U/C/S/F suboptions, explained under +A)dd, only apply to files being added to the archive or being updated +(files that aren't updated are left unaltered). + +Files that are updated will retain their previous filetype. New files +will get either the default filetype, the filetype specified by the F +suboption, or the actual filetype (under APW only) in that order. + + +*** Command Help + nulib H[NWS] +Displays a help screen. Three screens are available. + +"nulib h" alone displays help on the options. The N suboption gives +help with numbers; it lists the known compression methods and text +translation types. The W suboption gives a brief listing of +contributors, and how to contact me. The S suboption gives help on +the suboptions. + + +*** Integrity Check + nulib I[V] archive +Verifies that the archive is intact. Does not modify the archive in +any way. The V suboption prints a list of CRCs for each entire record +(this is different from those listed by the TZ option, which are only +for the record headers and (sometimes) threads; this includes not only +the headers but *all* data as well). + +Please note that this doesn't do much more than read the file, unless +the record_version is $0002 or greater (which means that the data has +a checksum stored; currently these records are only generated by +GS/ShrinkIt), AND no compression was used. This merely scans the +records and verifies the header CRCs, NOT the data CRCs. The main +purpose of the V suboption is to make a list of CRCs that can be sent +along with the archive. + + +*** Move Files to Archive + nulib M[V][U|C|S][R][F[/]][D] archive files +This is identical to the A option, but the files are deleted after +they are added to the archive. Note that the actual directory files +are NOT deleted, unless they were given distinct record entries. + +Care should be taken to avoid trying to M)ove an archive into itself. +The act of adding may (depending on the OS and the archive) go into an +infinite loop creating a huge file, and the coup de grace is when +NuLib then deletes the archive you were adding to. + + +*** Print an Archived File + nulib P[V][T][+] archive files +Print the contents of an archived file without extracting it. Useful +for viewing archived text files without having to actually unpack +them. Note this only allows viewing of data_threads; resource forks +and disk images will not be displayed, and comments are not shown. I +take no responsibility for pagination or filtering of funky control +characters... + +The V suboption will print the file's name right before it is +extracted. The + suboption allows you to specify the first part of a +pathname; see the D or X options for details. + +The T suboption will perform a text substitution as it works; see the +section on conversion of line terminators for more information. + + +*** Table of Contents + nulib T[VAZ] archive +With no suboptions, this lists only the filenames of the archived +files. Not only does this make it easier to view the archive contents +(the ShrinkIt format filename field is about 20 characters wide; this +is as wide as it has to be), but the output is suitable for +transmission via a pipe to other utilities. + +Using the V suboption will make it use ShrinkIt v2.0 output format +(same as using the V option); it is included as a suboption mainly for +people used to "ar". Using the A suboption will produce a list +similar to the output of ARC or ZOO. + +Using the Z suboption will dump everything known about the archive, +including all information in headers, CRCs, relative file position, +sizes of individual threads, etc. + + +*** Update Archive + nulib U[V][U|C|S][R][F[/]][D] archive files +Updates files in the archive, keeping the archived file or the file +listed on the command line, whichever is most recent (or exists). +Unlike freshen, this will add new files as necessary. Creates a new +archive file, and either transfers the old record or adds the file. +Only exact filename matches are allowed (case-independent), including +partial pathnames. The archive being updated must already exist. + +Wildcards are allowed, and subdirectories are automatically expanded +unless the R suboption is used. The V suboption displays the +filenames as they are added. The U/C/S/F suboptions, explained under +A)dd, only apply to files being added to the archive (files that +aren't updated are left unaltered). Note that the order of files in +the archive will be preserved. + +Files that are updated will retain their previous filetype. New files +will get either the default filetype, the filetype specified by the F +suboption, or the actual filetype (under APW only) in that order. + + +*** Verbose Archive Listing + nulib V archive +Lists the archive contents in a format indentical to that used by the +ShrinkIt v2.0 L)ist archive contents option. Same thing as "-tv". + + +*** Extract from Archive + nulib X[V][T][U][I][M][+] archive [archived-files] + nulib E... +The X and E options are synonymous. Extract the archived-files from +the archive. If the file already exists, you are asked if you want to +overwrite it. If part of a partial pathname does not exist, the +subdirectory will be created. Omitting the "archived-files" +specification will cause the entire archive to be unpacked. + +When files are archived, a pathname separator is stored (e.g., "/" for +ProDOS and UNIX, "\" for MS-DOS, ":" for Mac HFS). During extraction, +the pathnames are broken down into component file names, converted to +names legal under the current operating system, and then recombined +using the pathname separator for the current OS. This facilitates +extraction of files archived under any OS, but can lead to filename +conflicts that didn't exist when the files were added (e.g., a UNIX +file that contained a backslash is unpacked under MS-DOS). + +If the filename of an extracted file would conflict with an existing +file, the file will be overwritten and a message will be printed. If +the "I" suboption is used, then NuLib will give an "overwrite (y/n)?" +prompt. An affirmative answer overwrites the file; a negative answer +will skip the file and continue processing. + +Note that extraction does not require an exact match if the "+" +suboption is specified; "nulib x+ fubar.shk foo" will extract "FOO", +"Foozle", "food/recipies" and "food/stock." This makes it possible to +extract entire subdirectories at a time. The V suboption prints the +list of marked records. + +If comments are present, then the V suboption also prints those as it +extracts. Using the M suboption will cause the comments to be +printed, but the files will not be extracted. **NOTE**: the exact use +of the M suboption has not been entirely settled. Since it may be +desirable to extract comments into a file, future versions of NuLib +may use eXtract to put them in a file and Print to view them. +Comments welcome, caveat emptor, have a nice day. + +The T suboption will perform a text substitution as it extracts; see +the section on converson of line terminators for more information. + +The U suboption will extract the files without uncompressing them. +This is rarely useful, but is easy to implement and is present for the +curious. + + +Non-NuFX files +-------------- + +NuLib will only work with Binary II and NuFX archives. If you try to +view some other kind of file, you will get an error message and an +indication of what kind of file NuLib thinks it is. NuLib can +recognize files processed with compress, Zip, Zoo, StuffIt, ar, shar, +GIF, and many others. + + +Compression Methods +------------------- + +The following methods are defined in the NuFX documentation: +(# = method number, Name = method name [abbreviation], Pack? and +Unpack? refer to the ability of NuLib to perform that operation using +the given compression method. + +# Name Pack? Unpack? +00 Uncompressed [unc] Y Y +01 SQueezed [squ] N Y +02 Dynamic LZW I (ShrinkIt) [shk] Y Y +03 Dynamic LZW II (ShrinkIt) [sh2] N Y +04 12-bit UNIX Compress [u12] Y Y +05 16-bit UNIX Compress [u16] Y Y + +Attempting to use a compression method that does not exist will result +in an error message like "[can't squeeze; storing]...". This means +that the compression method you requested is unavailable, and it +simply stored the file without compression. + +If you try to extract a file that has been compressed with an +algorithm that NuLib is not familiar with, an error message will be +printed and the file will not be extracted. + + +Converting Line Terminators +--------------------------- + +Different operating systems use different line terminators in text +files. The table below shows them for some popular systems: + +Operating System Line Terminator +Apple ProDOS CR ($0d) +UNIX LF ($0a) +MS-DOS CRLF ($0d0a) + +While NuLib will know what kind of terminators the operating system it +is running under uses, it cannot reliably determine what kind an +archived file uses. Thus, the terminator used on the system where the +file was created must be specified. + +Note that text translation should *not* be performed on non-ASCII +files (non-ASCII means anything other than pure text; word processing +files and executables are two examples of files that should *never* +have text translation performed on them). Doing so will probably ruin +the file, causing strange errors. Because of the wide range of files +that NuLib must handle, it is impossible to automatically determine +which files are binaries and which are text. + +In order to tell NuLib what format to expect, you have to specify a +value parameter (although it will default to zero if you don't). The +following examples illustrate their usage: + +nulib xt0 ... Convert from CR (Apple II -> current system) +nulib xt1 ... Convert from LF (UNIX -> current system) +nulib xt2 ... Convert from CRLF (MS-DOS -> current system) + + +Shell Variables +--------------- + +NuLib looks for a shell variable called "NULIBOPT" to get default +values for certain things. This must be an environment variable +("setenv" using csh, "set" and "export" using sh or APW), and may +contain the following: + + verbose Default to Verbose output (otherwise Silent). + type=xxx Default type for storing files (under UNIX or + MS-DOS). "xxx" should match the 3-letter + ProDOS abbreviation (NON, BIN, TXT, SRC). + Normally "NON". + aux=xxxx Default auxtype for storing files (under UNIX + or MS-DOS). "xxxx" is a four-character hex + number (000a, 8002, abcd). Normally "0000". + interactive Default to Interactive mode (prompt before + overwriting files). + compress=n Set default compression to type n. This is + useful if you prefer 16-bit UNIX compress + (which would be compress=5). + +Note that the type= and aux= settings do NOT apply when running under +APW. Files will be stored with their actual filetypes, regardless of +the variable setting. + +Also, the 'F' suboption will override these settings. + +An example from csh: + setenv NULIBOPT type=BIN,compress=5,interactive + +** WARNING: As of this writing, GS/ShrinkIt 1.0.6 and ShrinkIt v3.4 +are not able to extract archived files compressed with UNIX compress. +It is likely that GS/ShrinkIt will be able to in the future, but the +//e version of ShrinkIt will probably never handle them. + + +Error Handling +-------------- + +Many errors simply cause the program to exit, leaving the archive in +an uncertain state (which sounds fairly evil). If you were +extracting, deleting, viewing, or updating when the error occurred, +the worst that can happen is you will be left with a bogus temporary +file in the current directory (something like "nulib.tmp5610"). + +If you were adding to an existing archive, the files that were there +will be unharmed, but additional files will not appear, and the +archive will be oversized. This is because the master header block +(which keeps a count of the number of records in the archive) is +written last. + +If you were creating a new archive, the file will be guano. This is +because, as mentioned before, the Master Header Block is not written +until the very end. Since NuLib identifies NuFX archives by looking +at certain bytes in the MHB, the file will not be identifiable as +NuFX. Note that the M)ove option is safer than it looks, because +files on disk are not deleted until the archive is safely closed. + + +Revision History +---------------- + +NuLib v3.22 (September 1992) +- improved speed of LZW-I compression +- added ability to add files as if they were disks (patches provided +by somebody whose name I've lost) +- updated the list of 3-letter filetype abbreviations + +NuLib v3.21 (August 1992) +- minor stuff + +NuLib v3.2 (April 1992) +- fixed two bugs in the LZW-II uncompression. +- made it compatible with SysV Expanded Fundamental Types (stuff like +eight byte file offsets on lseek()s). + +NuLib v3.11 - v3.14 (November 1991) +- fixed a problem with LZW-II uncompression. +- added fixes to make it compile on a NeXT and with MS C 6.0. +- fixed XENIX directory compatibility problems. + +NuLib v3.1 (October 1991) +- added ability to uncompress LZW-II compression (GS/ShrinkIt). +- improved speed of compression routines. +- improved System V compatibility. +- cleaned up code a bit and fixed minor bugs. + +Nulib v3.01 - v3.03 (February 1991) +- fixed XENIX problems with includes and libs. +- fixed bug in directory expansion. +- silenced screaming about bad dates. +- fixed glitches in nulib.lnk and nulib.mak +- fixed non-compression bug in ShrinkIt LZW. + +NuLib v3.0 (Sep 1990): +- added ability to compress files using ShrinkIt LZW-I compression +(replaces "fake" compression used previously). +- added "compress" parameter to NULIBOPT environment variable. +- added ARCZOO output format for people used to MS-DOS archivers. +- added M suboption for comment ("message") printing. + +NuLib v2.3 (May 1990 - not released): +- addition of UNIX compress. +- threw in some benchmarks. + +NuLib v2.2 - v2.22 (Apr 1990): +- second release as NuLib (after a brief vacation). +- fixed incompatibility problems with GS/ShrinkIt and ShrinkIt v3.0. +- added some support for comments. +- unpacks disk archives to files (good for the UNIX Apple ][+ +simulator). +- fake compression correctly handles files that don't compress (it +used to store them as compressed whether or not they got smaller). +- various minor changes/improvements (more help screens, etc). + +NuLib v2.1.1 (Nov 1989): +- first wide distribution as NuLib. +- fixed command processing bugs. + +NuLib v2.1 (Nov 1989): +- yet another name change (thanks loads, L&L). +- ShrinkIt LZW uncompression added. +- MS-DOS code added. +- shell variable used for defaults. +- CRLF translation. +- added R, I, and F suboptions. + +CShrink v2.08 (Oct 1989): +- altered help screens, some commands. +- added recognition of other kinds of files (compressed, shar, etc). +- switched to table lookups for calculating CRC (much faster). + +CShrink v2.07 (Sep 1989): +- Another name change (legal reasons). +- UNIX port completed. +- Binary II operations are fully functional. +- some compression code added (unSQueeze, fake ShrinkIt pack). +- '+' suboption added. +- text translation improved. +- printing of archived files (P option) now works. + +NuARC v2.03 (Aug 1989): +- first NuARC distribution (APW executable only). +- added subdirectory expansion. +- added suboption processing. +- replaced buffered I/O (fopen(), fread(), etc) on files with +lower-level read()/write() routines. +- added automatic byte-order determination. +- implemented move, extract all, and update/freshen. +- added Print archived file option. +- added Verbose, Text translation, and Uncompressed storage +suboptions. +- wrote this documentation. + +NuARC v2.0 (Aug 1989): +- added archive manipulation routines (EXtract, Add and Delete for +uncompressed archived files). +- added filename-only output format. +- added CRC verification. +- added byte order and data element size checks. +- removed the LAMESEEK option. + +NuView v1.2 (July 1989): +- major overhaul of all source code to make it work under APW C. +- new //gs-specific routines added. +- minor alterations to output format. + +NuView v1.1 (June 1989): +- major rewrite of the way archives are read (had problems with +machines requiring word-alignment of data). +- added thread file position storage to internal archive structure. +- fixed non-(void) sprintf() bug. + +NuView v1.0 (May 1989): +- initial release. +- works only on a Sun 3/50 running BSD UNIX. + + +Limitations +----------- + +NuLib works just fine with records containing more than one thread +(i.e., comments and resource forks). However, while comments can be +printed, they can't be added. + +The big problem this program has is speed. Since it is meant to be +portable first and efficient second, it won't run as fast as something +like ShrinkIt (written entirely in assembly). What I envision is +people using ShrinkIt at home on their Apple //s, but NuLib on UNIX +systems or other microcomputers. This will facilitate transfers of +large compressed files, which can then be quickly unpacked on the +destination system (which will likely have greater computing power, or +a C compiler more efficient than APW). + + +System Dependencies +------------------- + +When compiling this on a Sun 3/50, I noticed a problem: the byte +ordering for the //gs (65816) and the Sun (68020) are backward. The +present version of NuLib will automatically determine the byte +ordering and treat data appropriately (although this may fail if the +data size definitions are wrong). + +There are definitions for one byte, two byte, and four byte variable +types; my compiler uses char, short, and long. If these are different +for your compiler, be sure to change the typedefs in "nudefs.h". If +you don't have 8-bit bytes, though, it may not work (most machines +do). + +Notes on UNIX implementation: +If this is being run under UNIX, you should #define UNIX in +"nudefs.h". This will enable certain UNIX functions (like system() +calls and time routines), and disable others (like file types and +"binary mode" for file I/O). If this is being run under BSD UNIX, you +should also #define BSD43 in "nudefs.h"; certain small differences +were unavoidable (strrchr() vs rindex()). System V users should +#define SYSV. See "nudefs.h" for examples. + +Under APW, breaking down the full pathnames into relative pathnames is +easy (and should be under MS-DOS as well...). Under UNIX ".." is an +actual directory link, so obtaining a fully expanded pathname with no +redundancies is difficult. Thus, storing "dir1/../dir1/foo" would +appear as "dir1/foo" under APW but "dir1/../dir1/foo" under UNIX. +Care must be taken when extracting such files on non-UNIX systems; +they are best avoided entirely. One nasty pitfall is using "~/foo" +under the C shell... + +If the UNIX owner access permissions include write permission, the +file will be stored as unlocked. If write permission is denied, it +will be stored as locked. As far as dates are concerned, the +modification and access times will both be set to the modification +time. + +Notes on //gs implementation: +There is very little //gs-specific code, except where absolutely +necessary (time routines, file type handling, etc). GS/OS support is +absent (extended files are not handled yet, different file systems are +not recognized, and pseudo-mixed-case ProDOS filenames are stored as +upper case). Some of the faults belong to APW... + +Unfortunately, the startup code (2/start) provided with APW and the +methods of argument passing used by ECP-16 and ProSel-16 aren't +compatible. This means that you can run NuLib from ECP or ProSel, but +you can't pass it any arguments (which pretty much defeats the +purpose...). + +At any rate, the program is linked with the compression routines in +dynamic load segments, so that simple operations should run faster. +Additionally, it should be restartable from memory. [actually, the +ZapLink version doesn't have dynamic segments...] + +There is a bug (not my fault) when using the P option to print a file. +The file gets printed on one line for some reason... If you redirect +the I/O to a file, everything comes out fine. Weird. + +APW shell-specific code has been avoided where possible. Places where +I couldn't easily work around it include wildcard expansion and +ERROR(). It was deliberately used in several places to allow the user +to STOP() processing with Apple-period. + +NOTE: NuLib was written to work with APW C, ** NOT ** Orca/C. +Attempting to compile it with Orca/C would be a major undertaking (as +several people who have tried and failed can attest to). + +Notes on MS-DOS implementation: +The user interface may be slightly different from the UNIX and APW +versions, so that MS-DOS users will feel more at home when using the +program. The only major deviation is in the handling of subdirectory +expansion. + +A future version may select the ARC/ZOO output format as default. + + +Bugs / Glitches +--------------- + +UNIX lseek uses longs, which are usually four bytes. Signed. If an +archive is larger than 2 gigabytes, there may be a problem (cough). +This isn't a problem if the system has EFT. + +Pathnames longer than the #defined maximum (1024 bytes) will not be +processed. This is the limit on most machines, and is well in excess +of most people's sanity. Pathnames with a null ('\0') in them should +generally be avoided. + +Some partial pathname comparisons may fail because pathname separators +vary between operating systems. File naming conventions can result in +collisions (ex: "foo+" and "foo~" are unpacked under ProDOS, where +both are translated to "foo."). Also, you will probably need to be in +the same directory each time you U)pdate an archive, or else the +partial pathnames won't match (update requires an EXACT match), and +you'll end up with two different copies of the same file. + +A maximum of 255 files may be added/deleted/whatever at at time. +Expanded subdirectories count. This is an arbitrary number and is +easy to change if someone can convice me that you'd need to archive +more than 255 files at a time. + +The same file may be added/extracted/whatever more than once if the +user enters multiple file selectors on the command line ("nulib av +foo.shk file1 file1 file1"). This means that U)pdate could insert the +same file twice, etc. + +NuLib does not at present prevent an archive from being added to +itself. This can have unfortunate consequences, especially in +conjunction with the M)ove option. + +The ProDOS three-letter filetype names may or may not be the offical +Apple versions. Some users may have problems with things like "$00" +because '$' is a csh metacharacter (job control). + +Error output is informative but ugly (mostly debugging-type messages). +Error handling is somewhat fatal in most cases. + +The code is fairly large, currently around 260K (around 7000 lines of +C). It takes a while to compile it, and can be difficult if you only +have 1.25MB of RAM (kill your DAs and purge before you link!) + + +In the Works +------------ + +This is a wishlist of sorts. Don't hold your breath: +- Allow CR <-> LF translation when ADDING files. +- Improve GS/OS handling. Need to handle resource forks and file +system IDs. Would probably help if I had a GS/OS reference... +- Add to thread methods (insert ASCII messages, etc). +- Add different compression methods (LZH?). + + +Author Info +----------- + +I'm currently an Associate Software Engineer for Amdahl Corp. in Santa +Clara, California. + +e-mail: fadden@uts.amdahl.com (Andy McFadden) + +No relation to the author of ShrinkIt and NuFX, Andy Nicholas. This +was and continues to be my idea; throw all kudos and blame in my +general direction. All code herein is mine, developed from the NuFX +Documentation Revisions three through five (the latter from Call +-A.P.P.L.E.), except as noted below. + +Dynamic LZW-I (ShrinkIt) by Kent Dickey. LZW-II compression (LZW-I +with table clears) designed by Andy Nicholas. C implementation by +Frank Petroski and Kent Dickey (independently and simultaneously). + +MS-DOS code by Robert B. Hess and Bruce Kahn. + +The Binary II routines and the unSQueeze code were adapted from the C +source by Marcel J.E. Mol (usq.c based on usq2/sq3 by Don Elton). + +UNIX compress code from COMPRESS v4.3 (see source for detailed author +information). + +If you have suggestions or comments and can't send mail to my Internet +address, you can send US mail to: +Andy McFadden +1474 Saskatchewan Dr. +Sunnyvale, CA 94087 + +Andy Nicholas, the author of ShrinkIt and affiliated products +(ShrinkIt ][+, UnshrinkIt ][+, GS/ShrinkIt), may be contacted at: + Internet : shrinkit@Apple.COM + CompuServe : 70771,2615 + GEnie and + America OL : shrinkit + + +Legal Stuff +----------- + +(I'm not sure about all of these, but I'd like to cover all the +bases): + +ShrinkIt is a trademark of L&L Productions, Inc (...or is it?). +ARC is owned by SeaWare. +Apple is a tradmark of Apple Computer, Inc. +MS-DOS is (probably) a trademark of Microsoft, Inc. +PKZIP is (probably) a trademark of PKWare. + +I have not seen source code for ShrinkIt(tm) or for the compression +routines used within it, except as provided to me in C by Kent Dickey. + +Disclaimers, borrowed from Dave Whitney and Andy Nicholas: + +This program is FREEWARE; both source and binaries may be distributed +freely, but not sold. If you wish to include NuLib in the +distribution of a commercial product, you will need to get my +permission beforehand. + +Distribute as widely as possible, but please include this +documentation with the binaries and/or sources. + +Copyright (C) 1989-1991 by Andy McFadden. All rights reserved. + +I (Andy McFadden) MAKE NO WARRANTY ON THIS MANUAL OR SOFTWARE, EITHER +EXPRESS OR IMPLIED, WITH RESPECT TO QUALITY, MANUAL'S ACCURACY, +MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. + +IN NO EVENT WILL I BE HELD RESPONSIBLE FOR DIRECT, INDIRECT, SPECIAL, +INCIDENTAL, OR CONSEQUENTIAL DAMAGES RESULTING FROM ANY DEFECT OF THE +SOFTWARE OR INACCURACY IN THE MANUAL. + +In other words, I have tested the program to the best of my ability, +and I find that as distributed by me, it is safe for general use. It +isn't necessarily bug-free, and as a result, loss of data, however +unlikely, is entirely possible. Use at your own risk, but also for +your own enjoyment. diff --git a/library/nuview.tar.gz b/library/nuview.tar.gz new file mode 100644 index 0000000..7ef3de7 Binary files /dev/null and b/library/nuview.tar.gz differ diff --git a/library/old-binary2-spec.htm b/library/old-binary2-spec.htm new file mode 100644 index 0000000..462bd64 --- /dev/null +++ b/library/old-binary2-spec.htm @@ -0,0 +1,405 @@ + + + + + + +Binary ][ Protocol + + + +
+ +


+

+
+ +
+

Binary ][ Specification -- Initial Release

+

This is an older version of the specification. Please use the +File Type Note instead.

+

Back to nulib.com library

+
+                           Binary ][ protocol
+ 
+                              developed by
+                             Gary B. Little
+ 
+ Version History
+ ---------------
+ November 24, 1986  : Initial release.
+ 
+ Background
+ ----------
+ Transferring Apple II files in binary form to commercial information
+ services like CompuServe, Delphi, GEnie, and The Source is, to put it
+ mildly, a frustrating exercise. (For convenience, I'll refer to such
+ services, and any other non-Apple II systems, as "hosts.") Although
+ most hosts are able to receive a file's *data* in binary form (using
+ the Xmodem protocol, for example), they don't receive the file's all-
+ important attribute bytes. All the common Apple II operating systems,
+ notably ProDOS, store the attributes inside the disk directory, not
+ inside the file itself.
+ 
+ The ProDOS attributes are the access code, file type code, auxiliary
+ type code, storage type code, date of creation and last modification,
+ time of creation and last modification, the file size, and the name of
+ the file itself. (All these terms are defined in Apple's "ProDOS
+ Technical Reference Manual" or in the book "Apple ProDOS: Advanced
+ Features for Programmers" by Gary Little.) It is usually not possible
+ to use a ProDOS file's data without knowing what the file's attributes
+ are (particularly the file type code, auxiliary type code, and size).
+ This means ProDOS files uploaded in binary form to a host are useless
+ to those who download them. The same is true for DOS 3.3 and Pascal
+ files.
+ 
+ Most Apple II communications programs use special protocols for
+ transferring file attributes during a binary file transfer, but none
+ of these protocols have been implemented by hosts. These programs are
+ only useful for exchanging files with another Apple II running the
+ same program.
+ 
+ At present, the only acceptable way to transfer an Apple II file to a
+ host is to convert it into lines of text and send it as a textfile.
+ Such a textfile would contain a listing of an Applesoft program, or a
+ series of Apple II system monitor "enter" commands (e.g., 0300:A4 32
+ etc.). Someone downloading such a file can convert it to binary form
+ using the Applesoft EXEC command.
+ 
+ The main disadvantage of this technique is that the text version of
+ the file is over three times the size of the original binary file,
+ making it expensive (in terms of time and $$$) to upload and download.
+ It is also awkward, and sometimes impossible, to perform the binary-
+ to-text or text-to-binary conversion.
+ 
+ The solution to the problem is to upload an encoded binary file which
+ contains not just the file's data, but the file's attributes as well.
+ Someone downloading such a file, say using Xmodem, can then use a
+ conversion program to strip the attributes from the file and create a
+ file with the required attributes.
+ 
+ To make this technique truly useful, however, the Apple II community
+ must agree on a format for this encoded binary file. A variety of
+ incompatible formats, all achieving the same general result, cannot be
+ allowed to appear.
+ 
+ It is proposed that the Binary II format described in this document be
+ adopted. What follows is a description of the Binary II format in
+ sufficient detail to allow software developers to implement it in
+ Apple II communications programs.
+ 
+ The Binary II File Format
+ -------------------------
+ The Binary II form of a standard file consists of a 128-byte file
+ information header followed by the file's data. The data portion of
+ the file is padded with nulls ($00 bytes), if necessary, to ensure the
+ data length is an even multiple of 128. As a result, the Binary II
+ form of a file is never more than 255 bytes longer than the original
+ file.
+ 
+ The file information header contains four ID bytes, the attributes of
+ the file (in ProDOS 8 form), and some control information. Here is the
+ structure of the header:
+ 
+       Offset  Length                  Contents
+       ------  ------   ---------------------------------------
+        +0       1      ID byte: always $0A
+        +1       1      ID byte: always $47
+        +2       1      ID byte: always $4C
+        +3       1      access code
+        +4       1      file type code
+        +5       2      auxiliary type code
+        +7       1      storage type code
+        +8       2      size of file in 512-byte blocks
+        +10      2      date of modification
+        +12      2      time of modification
+        +14      2      date of creation
+        +16      2      time of creation
+        +18      1      ID byte: always $02
+        +19      1      [reserved]
+        +20      3      end-of-file (EOF) position
+        +23      1      length of filename/partial pathname
+        +24      64     ASCII filename or partial pathname
+        +88      23     [reserved, must be zero]
+        +111     1      ProDOS 16 access code (high)
+        +112     1      ProDOS 16 file type code (high)
+        +113     1      ProDOS 16 storage type code (high)
+        +114     2      ProDOS 16 size of file in blocks (high)
+        +116     1      ProDOS 16 end-of-file position (high)
+        +117     4      disk space needed
+        +121     1      operating system type
+        +122     2      native file type code
+        +124     1      phantom file flag
+        +125     1      data flags
+        +126     1      Binary II version number
+        +127     1      number of files to follow
+ 
+ Multi-byte numeric quantities are stored with their low-order bytes
+ first, the same order expected by ProDOS. All reserved bytes must be
+ set to zero; they may be used in future versions of the protocol.
+ 
+ To determine the values of the attributes to be put into a file
+ information header for a ProDOS file, you can use the ProDOS
+ GET_FILE_INFO and GET_EOF MLI commands.
+ 
+    Note: Some file attributes returned by ProDOS 16 commands
+          are one or two bytes longer than the attributes
+          returned by the corresponding ProDOS 8 commands. At
+          present, these extra bytes are always zero, and
+          probably will remain zero forever. In any event,
+          place the extra bytes returned by ProDOS 16 in the
+          header at +114 to +119. ProDOS 8 communications
+          programs should zero these header locations.
+ 
+ The "disk space needed" bytes contain the number of 512-byte disk
+ blocks the files inside the Binary II file will occupy after they've
+ been removed from the Binary II file. (The format of a Binary II file
+ containing multiple files is described below.) If the number is zero,
+ the person uploading the file did not bother to calculate the space
+ needed. The "disk space needed" must be placed in the file information
+ header for the first file inside the Binary II file; it can be set to
+ zero in subsequent headers. A downloading program can inspect "disk
+ space needed" and abort the transfer immediately if there isn't enough
+ disk free space.
+ 
+ The value of the "operating system type" byte indicates the native
+ operating system of the file:
+ 
+         $00 = ProDOS 8, ProDOS 16, or SOS
+         $01 = DOS 3.3
+         $02 = Pascal
+         $03 = CP/M
+         $04 = MS-DOS
+ 
+ Note that even if a file is not a ProDOS file, the attributes in the
+ file information header, including the name, must be inserted in
+ ProDOS form. Instructions on how to do this for DOS 3.3 files are
+ given later in this document. Similar considerations apply for the
+ files of other operating systems.
+ 
+ The "native file type code" has meaning only if the "operating system
+ type" is non-zero. It is set to the actual file type code assigned to
+ the file by its native operating system. (Some operating systems, such
+ as CP/M and MS-DOS, do not use file type codes, however.) Contrast
+ this with the file type code at +4, which is the closest equivalent
+ ProDOS file type code. The "native file type code" is needed to
+ distinguish files which have the same *ProDOS* file type, but which
+ may have different file types in their native operating system. Note
+ that if the file type code is only byte long (the usual case), the
+ high-order byte of "native file type code" is set to zero.
+ 
+ The "phantom file flag" byte indicates whether a receiver of the
+ Binary II file should save the file which follows (flag is zero) or
+ ignore it (flag is non-zero). It is anticipated that some
+ communications programs will use phantom files to pass non-essential
+ explanatory notes or encoded information which would be understood
+ only by a receiver using the same communications program. Such
+ programs must not rely on receiving a phantom file, however, since
+ this would mean they couldn't handle Binary II files created by other
+ communications programs.
+ 
+ The first two bytes in a phantom file *must* contain an ID code unique
+ to the communications program. Developers must obtain ID codes from
+ Gary Little to ensure uniqueness (see below for his address). Here is
+ a current list of approved ID codes for phantom files used by Apple II
+ communications programs:
+ 
+         $00 $00  =  [generic]
+         $00 $01  =  Point-to-Point
+         $00 $02  =  Tele-Master Communications System
+ 
+ Developers of communications programs are responsible for defining and
+ publishing the structures of their phantom files.
+ 
+ The ID bytes appear in the first two bytes of the phantom file.
+ Phantom files having a generic ID code of zero must contain lines of
+ text terminated by a $00 byte. The text must begin at the third byte
+ in the file.
+ 
+ The "data flags" byte is a bit vector indicating whether the data
+ portion of the Binary II file has been compressed, encrypted, or
+ packed. If bit 7 (the high-order bit) is set to 1, the file is
+ compressed. If bit 6 is 1, the file is encrypted. If bit 0 is 1, the
+ file is a sparse file that is packed. A Binary II downloading program
+ can examine this byte and warn the user, when necessary, that the file
+ must be expanded, decrypted, or unpacked. The person uploading a
+ Binary II file may use any convenient method for compressing,
+ encrypting, or packing the file but is responsible for providing
+ instructions on how to restore the file to its original state.
+ 
+ This initial release of Binary II has a "Binary II version number" of
+ $00.
+ 
+ Handling Multiple Files
+ -----------------------
+ An appealing feature of Binary II is that a single Binary II file can
+ hold multiple disk files, making it easy to keep a group of related
+ files "glued" together when they're sent to a host.
+ 
+ The structure of a Binary II file containing multiple disk files is
+ what you might expect: it is a series of images of individual Binary
+ II files. For example, here is the general structure of a Binary II
+ file containing three disk files:
+ 
+  start                                                           end
+  -------------------------------------------------------------------
+  | Header #1 | #1 Data | Header #2 | #2 Data | Header #3 | #3 Data |
+  -------------------------------------------------------------------
+    +127 = 2              +127 = 1              +127 = 0
+ 
+ The data areas following each header end on a 128-byte boundary.
+ 
+ The "number of files to follow" byte (at offset 127) in the file
+ information header for each disk file contains the number of disk
+ files that follow it in the Binary II file. It will be zero in the
+ header for the last disk file in the group.
+ 
+ Filenames and Partial Pathnames
+ -------------------------------
+ Notice that you can put a standard ProDOS filename or a partial
+ pathname in the file information header (but never a complete
+ pathname). *Beware!* Don't use a partial pathname unless you've
+ included, earlier on in the Binary II file, file information headers
+ for each of the directories referred to in the partial pathname. Such
+ a header must have its "end of file position" bytes set to zero, and
+ no data blocks for the subdirectory file must follow it.
+ 
+ For example, if you want to send a file whose partial pathname is
+ HELP/GS/READ.ME, first send a file information header defining the
+ HELP/ subdirectory, then one defining the HELP/GS/ subdirectory. If
+ you don't, someone downloading the Binary II file won't be able to
+ convert it because the necessary subdirectories will not exist.
+ 
+ Filename Convention
+ -------------------
+ Whenever a file is sent to a host, the host asks the sender to provide
+ a name for it. If it's a Binary II file, the name provided should end
+ in .BNY so that its special form will be apparent to anyone viewing a
+ list of filenames.
+ 
+ Identifying Binary II Files
+ ---------------------------
+ ose the ProDOS
+ file. You would repeat this for each file contained inside the Binary
+ II file.
+ 
+    Note: The number of 128-byte data blocks following the
+          file information header must be derived from the
+          "end-of-file position" attribute (EOF) not the "size
+          of file in blocks" attribute. Calculate the number
+          by dividing EOF by 128 and adding one to the result
+          if EOF is not 0 or an exact multiple of 128.
+ 
+ Exception: If the file information header defines a subdirectory (the
+ file type code is 15), simply CREATE the subdirectory file. Do not
+ OPEN it and do not set its size with SET_EOF.
+ 
+ Ideally, all this conversion work will be done automatically by a
+ communications program during an Xmodem (or other binary protocol)
+ download. If not, a separate conversion program will have to be run
+ after the Binary II file has been received and saved to disk. Gary
+ Little has published a public domain program, called BINARY.DWN, that
+ will do this for you. (A related program, BINARY.UP, combines multiple
+ ProDOS files into one Binary II file which can then be uploaded to a
+ host.)
+ 
+ DOS 3.3 Considerations
+ ----------------------
+ With a little extra effort, you can also convert DOS 3.3 files to
+ Binary II form. This involves translating the DOS 3.3 file attributes
+ to the corresponding ProDOS attributes so that you can build a proper
+ file information header. Here is how to do this:
+ 
+    (1) Set the name to one that adheres to the stricter ProDOS naming
+        rules.
+ 
+    (2) Set the ProDOS file type code, auxiliary type code, and access
+        code to values which correspond to the DOS 3.3 file type:
+ 
+           DOS 3.3  |   ProDOS     ProDOS    ProDOS
+          file type | file type   aux type   access
+         -----------|----------- ---------- --------
+          $00 ( T)  | $04 (TXT)    $0000      $E3
+          $80 (*T)  | $04 (TXT)    $0000      $21
+          $01 ( I)  | $FA (INT)    $0C00      $E3
+          $81 (*I)  | $FA (INT)    $0C00      $21
+          $02 ( A)  | $FC (BAS)    $0801      $E3
+          $82 (*A)  | $FC (BAS)    $0801      $21
+          $04 ( B)  | $06 (BIN)     (*)       $E3
+          $84 (*B)  | $06 (BIN)     (*)       $21
+          $08 ( S)  | $06 (BIN)    $0000      $E3
+          $88 (*S)  | $06 (BIN)    $0000      $21
+          $10 ( R)  | $FE (REL)    $0000      $E3
+          $90 (*R)  | $FE (REL)    $0000      $21
+          $20 ( A)  | $06 (BIN)    $0000      $E3
+          $A0 (*A)  | $06 (BIN)    $0000      $21
+          $40 ( B)  | $06 (BIN)    $0000      $E3
+          $C0 (*B)  | $06 (BIN)    $0000      $21
+ 
+          (*) Set the aux type for a B file to the
+              value stored in the first two bytes
+              of the file (this is the default load
+              address).
+ 
+     (3) Set the storage type code to $01.
+ 
+     (4) Set the size of file in blocks, date of creation, date of
+         modification, time of creation, and time of modification to
+         $0000.
+ 
+     (5) Set the end-of-file position to the length of the DOS 3.3
+         file, in bytes. For a B file (code $04 or $84), this number is
+         stored in the third and fourth bytes of the file. For an I
+         file (code $01 or $81) or an A file (code $02 or $82), this
+         number is stored in the first and second bytes of the file.
+ 
+     (6) Set the operating system type to $01.
+ 
+     (7) Set the native file type code to the value of the DOS 3.3 file
+         type code.
+ 
+ Attribute bytes inside a DOS 3.3 file (if any) must *not* be included
+ in the data portion of the Binary II file. This includes the first
+ four bytes of a B (Binary) file, and the first two bytes of an A
+ (Applesoft) or I (Integer BASIC) file.
+ 
+ Acknowledgements
+ ----------------
+ Thanks to Glen Bredon for suggesting that partial pathnames be allowed
+ in file information headers. Thanks also to Shawn Quick for suggesting
+ the "phantom file" byte, to Scott McMahan for suggesting the
+ compression and encryption bits in the "data flags" byte, and to
+ William Bond for suggesting the "disk space needed" bytes. Finally, a
+ big thank you to Neil Shapiro, Chief Sysop of MAUG, for supporting the
+ development of the Binary II format and helping it become a true
+ standard.
+ 
+ Feedback and Support
+ --------------------
+ Send any comments or questions concerning the Binary II file format
+ to:
+ 
+    Gary B. Little
+    #210 - 131 Water Street
+    Vancouver, British Columbia
+    Canada  V6B 4M3
+    (604) 681-3371
+ 
+    CompuServe : 70135,1007
+    Delphi     : GBL
+    MCI Mail   : 658L6
+ 
+ Gary developed the Point-to-Point telecommunications program published
+ by Pinpoint Publishing. He has also written several books on how to
+ program Apple computers: "Inside the Apple IIe," "Inside the Apple
+ IIc," "Apple ProDOS: Advanced Features for Programmers," and "Mac
+ Assembly Language: A Guide for Programmers." He is currently a
+ Contributing Editor for A+ magazine and writes A+'s monthly Rescue
+ Squad column. Gary has also published articles in Nibble, Micro, Call
+ -A.P.P.L.E, and Softalk.
+ 
+
+ +
+ + diff --git a/library/old-nufx-spec.htm b/library/old-nufx-spec.htm new file mode 100644 index 0000000..163e540 --- /dev/null +++ b/library/old-nufx-spec.htm @@ -0,0 +1,1873 @@ + + + + + + +NuFX Spec Final Rev 3 + + + +
+ +


+

+
+ +
+

NuFX Specification -- Final Revision 3

+

Back to nulib.com library

+

This is an older version of the specification. Please use the +File Type Note instead.

+
+ -------------------------------------------------------------------------------
+ |   The New Archive Standard for the Apple II -- "NuFX" -- NuFile eXchange    |
+ -------------------------------------------------------------------------------
+ 
+                 NuFX Documentation - 12/2/88 by Andy Nicholas
+                       Final Revision Three - 2/3/89
+ 
+                   Please distribute as widely as possible
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ Send questions/comments to:
+ 
+ Paper Bag Productions          CSNET: nicholaA@moravian.edu
+ c/o Andy Nicholas           InterNET: nicholaA%batman.moravian.edu@relay.cs.net
+ Box 435                               nicholaA@batman.moravian.edu.csnet
+ Moravian College             ProLine: andyn@pro-sol.cts.com     [619-670-5379]
+ Bethlehem, PA  18018    AppleLink PE: ShrinkIt
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ NuFX Documentation  2/3/89                             - Page 2 -
+ 
+ 
+ Preface
+ -------
+ 
+ The first copy of this proposal that was circulated was preliminary
+ revision 10.   Since then I have had many requests for a more flexible
+ solution to the archival problem.  To do so meant the removal of such
+ features as the alignment to 128-byte boundaries of all data blocks and
+ so forth, and the re-definition of the header block so that it is less rigid
+ in its format (extensible).
+ 
+ To do this makes it at least a factor more complex for programs which can
+ extract records from NuFX archives on-the-fly (such as telecommunications
+ programs).  While this may be seen as a hardship by some, I believe the
+ additional power provided by all formats after revision 10 justifies the
+ additional complexity.  Standalone utility programs should have no problems
+ adjusting to the new format.
+ 
+ This final revision of the documentation fixes the master_header's size
+ at 48 bytes, eliminates the archive_create_program and archive_mod_program
+ fields, increases the size of the access field to a longword, and introduces
+ a new name for the archive, NFX, or NuFX, for "NuFile eXchange."
+ 
+ I am also almost finished an archive program which places files or disks
+ into NuFX archives using dynamic LZW compression, ShrinkIt.  Any reference
+ to ShrinkIt in the text is a reference to this program.
+ 
+ Final Rev 2 corrects the all the algorithms to include the length of the
+ thread list in their calculations, includes some notes on how to properly
+ include threads and directories, and standardizes on the use of "NuFX" to
+ describe the archive name so as to distinguish it from Sun Microsystems'
+ "NFS" network product.
+ 
+ Final Rev 3 changes some of the terminology used, changes the date labels
+ to xx_when instead of xxx_date_time, assigns the resource_fork of a file
+ to be stored as a thread_kind in a data_thread instead of having it's own
+ thread available.  An attempt at further clarity has been undertaken.
+ 
+ andy (2/3/89)
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ NuFX Documentation  2/3/89                             - Page 3 -
+ 
+ 
+ History
+ -------
+ 
+ The Apple II community has always been sorely lacking a well-defined method
+ for archiving files.  NuFX is an attempt to rectify the situation by providing
+ for a flexible, consistent standard for archiving files, disks, and other
+ computer medium.
+ 
+ The Binary II standard, authored by Gary B. Little, for placing multiple files
+ within a single file has been rendered obsolete by the (now) recent release of
+ GS/OS(tm) which provides access to multiple filing systems.  Since GS/OS can
+ (or will) use HFS files, Binary II does NOT provide for:
+ 
+ o  filenames larger than 64 characters
+    (GS/OS can create 8000 character filenames)
+ 
+ o  a convenient way to add to, remove from, and other ways work on an archive.
+ 
+ o  including HFS-style files which contain resource forks.
+ 
+ o  including of entire disk images.
+ 
+ o  including messages along with a file.
+ 
+ o  a convenient way to represent that a file is compressed or encrypted by
+     a specific application.  (an excess secondary header must be attached to
+     the beginning of files, such is the case with SQueezed files)
+ 
+ o  a true archive standard.  Binary II's original intent was to make transfer
+     of Apple II files from local machines to large information services like
+     The Source, Delphi, CompuServe, and GEnie, possible.  Otherwise, a file's
+     attribute information would be lost.  Binary II is now being stretched
+     beyond what it was originally meant to do.
+ 
+ o  no support for multiple data threads or structures.
+ 
+ Adding all of these features to the existing Binary II standard would not only
+ be nerve-racking, but nearly impossible without violating the existing standard
+ and causing a great deal of confusion (ie, "did you say it was Binary II, first
+ revision or the second one?")  Although Binary II is flexible, it is simply
+ unable to address all of these concerns without alienating existing Binary II
+ extraction programs.
+ 
+ So, to provide some differentiation between standards and provide a better
+ functioning format, I have defined a new standard called "NuFX" (NuFile
+ eXchange for the Apple II).  NuFX fixes the problems that Apple IIgs(tm)
+ users would soon be experiencing as other filing systems become available for
+ GS/OS(tm).  I am trying to stop a set of problems before they have a chance to
+ develop.  NuFX provides all of the features of Binary II, but doesn't stop
+ there...  it goes farther to allow the user the ultimate in flexibility,
+ usefulness and performance.
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ NuFX Documentation  2/3/89                             - Page 4 -
+ 
+ 
+ Implementation
+ --------------
+ 
+ The basic structure of a NuFX archive is as follows:
+ 
+                 [First record]                    [Next Record]
+ +---------------------------------------------------------------------------+
+ | Master Header | Header | Data  . . . . . . . .  | Header | Data  . . . .  |
+ +---------------------------------------------------------------------------+
+ 
+ A single master header block contains values which describe the entire archive
+ (those of you who are into structured programming can consider them archive
+ globals).  Each of the succeeding header blocks contain only information about
+ the record they precede (consider them archive locals).
+ 
+ Each header block may be followed by a series of "threads."  Each thread may
+ be a portion of data, a message, the resource part of an extended file, a
+ control sequence for a NuFX utility program, or almost any sort of
+ sequential data.  The number of threads is described as a longword (32-bit
+ word), so it is also possible to properly archive and store the data
+ portions of sparse files.
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ NuFX Documentation  2/3/89                             - Page 5 -
+ 
+ 
+ Master header block contents:
+ 
+ All word and double-word values are byte-reversed.
+ 
+ Offset  Length  Content
+ ------  ------  ---------------------------------------------------------------
+ +0      1       $4E  Master ID Byte #1
+ +1      1       $F5  Master ID Byte #2
+ +2      1       $46  Master ID Byte #3   spells the word "NuFile" in
+ +3      1       $E9  Master ID Byte #4   alternating ASCII (high, low) for
+ +4      1       $6C  Master ID Byte #5   uniqueness.
+ +5      1       $E5  Master ID Byte #6
+ 
+ +6      2       master_crc
+ 
+                 16-bit CRC of the remaining fields in this block.
+                 (bytes +8 through +47)
+ 
+                 Any programs which modify the master header block *MUST*
+                 recalculate the CRC for the master header.
+ 
+ +8      4       total_records
+ 
+                 Total number of records in this archive file.
+ 
+                 It is possible to chain multiple records (Files or
+                 Disks) together.  It is also possible to chain
+                 different types of records together (Files and
+                 Disks mixed).
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ NuFX Documentation  2/3/89                             - Page 6 -
+ 
+ 
+ +12     8       archive_create_when
+ 
+                 The date and time on which this archive was initially
+                 created.  This field should never be changed once initially
+                 written.
+ 
+                 The format of this field is as follows:
+ 
+                 +12     second - 0 through 59
+                 +13     minute - 0 through 59
+                 +14     hour - 0 through 23
+                 +15     current Year minus 1900
+                 +16     day - 0 through 30
+                 +17     month - 0 through 11, with 0=January
+                 +18     filler byte - reserved must be null (00).
+                 +19     weekDay - 1 through 7, with 1=Sunday
+ 
+                 The format of this field is identical to that described
+                 in the _ReadTimeHex ($0D03) call described on page 14-14
+                 of the Apple_IIgs_Toolbox_Reference:_Volume_1.
+ 
+                 If the date is not known, or is unable to be calculated, this
+                 field should be set to null (00).  If the weekDay is not
+                 known, or is unable to be calculated, this field should be
+                 set to null (00).
+ 
+ 
+ +20     8       archive_mod_when
+ 
+                 The date of the last modification to this archive. This field
+                 should be changed every time a change is made to any of the
+                 records in the archive.
+ 
+                 The format of this field is as follows:
+ 
+                 +20     second - 0 through 59
+                 +21     minute - 0 through 59
+                 +22     hour - 0 through 23
+                 +23     current Year minus 1900
+                 +24     day - 0 through 30
+                 +25     month - 0 through 11, with 0=January
+                 +26     filler byte - reserved, must be null (00).
+                 +27     weekDay - 1 through 7, with 1=Sunday
+ 
+                 The format of this field is identical to that described
+                 in the _ReadTimeHex ($0D03) call described on page 14-14
+                 of the Apple_IIgs_Toolbox_Reference:_Volume_1.
+ 
+                 If the date is not known, or is unable to be calculated, this
+                 field should be set to null (00).  If the weekDay is not
+                 known, or is unable to be calculated, this field should be
+                 set to null (00).
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ NuFX Documentation  2/3/89                             - Page 7 -
+ 
+ 
+ +28
+  .
+  .              *** RESERVED, MUST BE SET TO NULL (00) ***
+  .              Do NOT use any of these fields.
+  .
+ +47
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ NuFX Documentation  2/3/89                             - Page 8 -
+ 
+ 
+ The following header block must precede each record within the NuFX archive.
+ The cyclic redundancy check (CRC) has been provided to detect archives which
+ have possibly been corrupted.  The only time the CRC should be included in
+ in a block is for the master header and for each of the regular header blocks.
+ The CRC functions to ensure reliability and record integrity.
+ 
+ Header Block contents:
+ 
+ All word and double-word values are byte-reversed.
+ 
+ Offset  Length  Content
+ ------  ------  ---------------------------------------------------------------
+ +0      1       $4E - Header ID Byte #1
+ +1      1       $F5 - Header ID Byte #2  Spells "NuFX" in alternating ascii
+ +2      1       $46 - Header ID Byte #3  (high/low) for uniqueness.
+ +3      1       $D8 - Header ID Byte #4
+ 
+ +4      2       header_crc
+ 
+                 16-bit CRC of the remaining fields of this block.
+                 (bytes 6 through the end of the attributes, filename, and
+                  any threads.)
+ 
+                 This field is used to verify the integrity of the rest of the
+                 block.
+ 
+                 Programs which make NuFX archives *MUST* include this
+                 in every header.  It is up to the discretion of the extracting
+                 program to check the validity of these bytes.  Any programs
+                 which might modify the header of a particular record *MUST*
+                 recalculate the CRC for the header block.
+ 
+ 
+ +6      2       attrib_count
+ 
+                 This field describes the length of the attribute section of each
+                 header in bytes.  This count measures the distance in bytes
+                 from the first field (offset +0) to and including the
+                 filename_length field.  By convention, the filename_length
+                 field will always be the last 2 bytes of the attribute
+                 section regardless of what has preceded it.
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ NuFX Documentation  2/3/89                             - Page 9 -
+ 
+ 
+ +8      2       version_number
+ 
+                 Minimum NuFX version number needed for extraction.
+                 (Currently $0000)
+ 
+                 This field is used to detect the possible existence of
+                 other as-of-yet undefined fields and features.  Utility
+                 programs should check this value to be certain that they
+                 are capable of extracting a record with this minimum version.
+ 
+ 
+ +10     4       total_threads
+ 
+                 The number of thread sub-records which should be expected
+                 immediately following the end of the file/path name.  This
+                 field is extremely important because it contains the
+                 information about the length of the last 1/3 of the header.
+ 
+ 
+ +14     2       file_sys_id
+ 
+                 Native file system identifier:
+ 
+                 $0000   reserved
+                 $0001   ProDOS/SOS
+                 $0002   DOS 3.3
+                 $0003   DOS 3.2
+                 $0004   Apple II Pascal
+                 $0005   Macintosh(tm) (HFS)
+                 $0006   Macintosh (MFS)
+                 $0007   LISA(tm) file system
+                 $0008   Apple CP/M
+                 $0009   reserved, do not use
+                 $000A   MS-DOS
+                 $000B   High-Sierra/ISO 9660
+ 
+                 $000C
+                 .
+                 .       reserved
+                 .
+                 $FFFF
+ 
+                 Disk: if the file system of a disk is not known, then
+                  this field should be set to null (0000).
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ NuFX Documentation  2/3/89                            - Page 10 -
+ 
+ 
+ +16     2       file_sys_info
+ 
+                 Information about the current filing system:
+ 
+                 [$00xx]
+ 
+                 Native file system separator.  Under Prodos, the "/" ($2F)
+                 character is used to separate paths.  Under HFS, the
+                 ":" ($3A) character is used to separate paths.  Under
+                 MS-DOS, the "\" ($5C) character is used to separate paths.
+                 The low byte of this word is used to store the file system's
+                 separator.
+ 
+                 The primary reason for including this field is that the
+                 receiving file system (say, to Prodos 8 from GS/OS running an
+                 HFS File System Translator) must know how to parse a valid
+                 file/path name from the filename field for the receiving file
+                 system.
+ 
+                 [$xx00]
+ 
+                 Sparse byte.  If the high-byte of this word is $01, then
+                 the image which follows is a sparse file (and the threads
+                 should have been filled in properly to indicate this).  If
+                 a more "normal" image follows, this byte will be null (00).
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ NuFX Documentation  2/3/89                            - Page 11 -
+ 
+ 
+ +18     4       access          [0000 0000 0000 0000 DRB00IWR]
+ 
+                 bits 31-8       reserved, must be zero
+                 bit 7           D=0, destroy disabled
+                                 D=1, destroy enabled
+                 bit 6           R=0, rename disabled
+                                 R=1, rename enabled
+                 bit 5           B=0, backup not needed
+                                 B=1, backup needed
+                 bits 4-3        reserved, must be zero
+                 bit 2           I=0, file is visible
+                                 I=1, file is invisible
+                 bit 1           W=0, write disabled
+                                 W=1, write enabled
+                 bit 0           R=0, read disabled
+                                 R=1, read enabled
+ 
+                 Disk: this field should be set to null (00).
+ 
+ 
+ +22     4       file_type
+ 
+                 Disk: this field should be set to null (00).
+ 
+ 
+ +26     4       extra_type
+ 
+                 ProDOS aux_type or HFS creator_type
+ 
+                 Disk: this field *MUST* be set to the total number of blocks
+                   on the device.  This information must be present so that
+                   the extracting program can place the record on the proper
+                   type of device.
+ 
+ 
+ +30     2       storage_type
+ 
+                 $0 - $3 = standard file
+                 $5      = extended (gs/os) file
+                 $d      = subdirectory
+ 
+          Disk:  file_sys_block_size
+ 
+                 This should only be used if a disk is being archived. The block
+                 size used by the device should be placed in this field.  For
+                 example, under Prodos, this field will be 512, while HFS
+                 would set it to 524.
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ NuFX Documentation  2/3/89                            - Page 12 -
+ 
+ 
+ +32     8       create_when
+ 
+                 The date and time on which this record was initially created.
+                 If the creation date and time is available from a disk device,
+                 this information should be included.
+ 
+                 The format of this field is as follows:
+ 
+                 +32     second - 0 through 59
+                 +33     minute - 0 through 59
+                 +34     hour - 0 through 23
+                 +35     current Year minus 1900
+                 +36     day - 0 through 30
+                 +37     month - 0 through 11, with 0=January
+                 +38     filler byte - reserved, must be set to null (00).
+                 +39     weekDay - 1 through 7, with 1=Sunday
+ 
+                 The format of this field is identical to that described
+                 in the _ReadTimeHex ($0D03) call described on page 14-14
+                 of the Apple_IIgs_Toolbox_Reference:_Volume_1.
+ 
+                 If the date is not known, or is unable to be calculated, this
+                 field should be set to null (00).  If the weekDay is not
+                 known, or is unable to be calculated, this field should be
+                 set to null (00).
+ 
+ 
+ +40     8       mod_when
+ 
+                 The date and time on which this record was last modified.
+                 If the modification date is available from a disk device,
+                 this information should be included.
+ 
+                 The format of this field is as follows:
+ 
+                 +40     second - 0 through 59
+                 +41     minute - 0 through 59
+                 +42     hour - 0 through 23
+                 +43     current Year minus 1900
+                 +44     day - 0 through 30
+                 +45     month - 0 through 11, with 0=January
+                 +46     filler byte - reserved, must be set to null (00).
+                 +47     weekDay - 1 through 7, with 1=Sunday
+ 
+                 The format of this field is identical to that described
+                 in the _ReadTimeHex ($0D03) call described on page 14-14
+                 of the Apple_IIgs_Toolbox_Reference:_Volume_1.
+ 
+                 If the date is not known, or is unable to be calculated, this
+                 field should be set to null (00).  If the weekDay is not
+                 known, or is unable to be calculated, this field should be
+                 set to null (00).
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ NuFX Documentation  2/3/89                            - Page 13 -
+ 
+ 
+ +48     8       archive_when
+ 
+                 The date and time on which this record was placed in this
+                 archive.
+ 
+                 The format of this field is as follows:
+ 
+                 +48     second - 0 through 59
+                 +49     minute - 0 through 59
+                 +50     hour - 0 through 23
+                 +51     current Year minus 1900
+                 +52     day - 0 through 30
+                 +53     month - 0 through 11, with 0=January
+                 +54     filler byte - reserved, must be set to null (00).
+                 +55     weekDay - 1 through 7, with 1=Sunday
+ 
+                 The format of this field is identical to that described
+                 in the _ReadTimeHex ($0D03) call described on page 14-14
+                 of the Apple_IIgs_Toolbox_Reference:_Volume_1.
+ 
+                 If the date is not known, or is unable to be calculated, this
+                 field should be set to null (00).  If the weekDay is not
+                 known, or is unable to be calculated, this field should be
+                 set to null (00).
+ 
+ Any other attributes which are needed may be added at the discretion of the
+ NuFX application programmer.  The attrib_count field should be modified
+ accordingly.
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ NuFX Documentation  2/3/89                            - Page 14 -
+ 
+ 
+ attrib_count-2  filename_length (xx bytes)
+ 
+                 Length of filename.  Under Prodos, this will not exceed 64
+                 characters.  If HFS or another filing system is used, this
+                 field may exceed 64 characters.  This is the last field
+                 considered included in the attributes section.  To allow the
+                 inclusion of future additional parameters in the attributes
+                 section,  NuFX utility programs should rely on the
+                 attribs_count field to find the filename_length field.
+ 
+ -------------------------
+ End of attributes section
+ 
+ 
+      xx Bytes   Filename or partial pathname if applicable.
+ 
+ 
+ 
+                 If this is a disk which is being archived, then the
+                 volume_name should be included in this field.  If a volume
+                 name is included in this field, a separator ("/" or ":")
+                 should *NOT* be included in, or precede the name.  If a volume
+                 name is not available, then this field should be set to nulls.
+                 (00's)
+ 
+                 If a partial pathname is specified, the directories to which
+                 the current pathname refers need not have preceded this
+                 particular record.  The extraction program must test each
+                 referenced directory individually.  If the directory in
+                 question does not exist, the extracting program should create
+                 it.
+ 
+                 Any utility which extracts files from a NuFX archive *MUST NOT*
+                 assume that this field will be in a format it is able to
+                 handle.  In particular, extraction programs should check for
+                 mixed case text in a file/path name and do whatever
+                 conversions are necessary to parse a legal file/path name.
+                 In general, assume nothing.
+ 
+ -----------------------
+ End of filename section
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ NuFX Documentation  2/3/89                            - Page 15 -
+ 
+ 
+ Threads
+ -------
+ 
+ Thread records are 16 byte records which immediately follow the filename and
+ describe the types of data structures which are included with a given record.
+ The number of thread records is described in the attribute section by a
+ longword, total_threads.
+ 
+ Each thread record should be checked for the type of information that a given
+ utility program can extract.  If a utility is incapable of extracting a
+ particular thread, that thread should be skipped.  If a utility finds a
+ redundancy in a thread_record, it must decide whether to skip the record or to
+ do something with that particular thread (ie, if a utility finds 2
+ message_threads it can either ignore the second thread or display it. Likewise,
+ if a utility finds 2 resource threads, it can either overwrite the first thread
+ which was extracted, or warn the user and skip the errant thread).
+ 
+ 
+ A thread record can be represented as follows:
+ 
+ Offset  Length  Content
+ ------  ------  ---------------------------------------------------------------
+ +0      2       thread_class
+ +2      2       thread_format
+ +4      2       thread_kind
+ +6      2       reserved
+ +8      4       thread_eof
+ +12     4       comp_thread_eof
+ 
+ 
+ "thread_class" describes the classification of the thread
+ ---------------------------------------------------------
+ 
+ $0000   =       message_thread
+ $0001   =       control_thread
+ $0002   =       data_thread
+ $0003   =       sparse_thread
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ NuFX Documentation  2/3/89                            - Page 16 -
+ 
+ 
+ "thread_format" is the format of the data within the thread.
+ ------------------------------------------------------------
+ 
+ $0000   =       Uncompressed                    [Not application specific]
+ $0001   =       SQueezed (SQ/USQ)               [Not application specific]
+ 
+ $0002   =       Dynamic LZW                     [ShrinkIt]
+ 
+ $0003
+  .
+  .      RESERVED, contact the author
+  .
+ 
+ $FFFF
+ 
+ 
+ "thread_kind" describes the kind of data which is contained in the thread
+ -------------------------------------------------------------------------
+ 
+ if thread_class
+    $0000   =       message_thread
+                    thread_kind $0000 = ASCII text
+                                $xxxx = all others undefined
+ 
+    $0001   =       control_thread
+                    thread_kind $0000 = create directory
+                                $xxxx = all others undefined
+ 
+    $0002   =       data_thread
+                    thread_kind $0000 = data_fork of file
+                    thread_kind $0001 = disk image
+                    thread_kind $0002 = resource_fork of file
+                                $xxxx = all others undefined
+ 
+ 
+ "thread_eof" is the length of the uncompressed thread
+ -----------------------------------------------------
+ 
+ 
+ "comp_thread_eof" is the length of the compressed thread
+ --------------------------------------------------------
+ 
+ 
+ Current ideas for messages include static pictures, sounds, sound & pictures,
+ animations, and possibly executable files.  I encourage writers of NuFX
+ utility programs to be able to handle messages of the lowest common
+ denominator, ASCII text.
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ NuFX Documentation  2/3/89                            - Page 17 -
+ 
+ 
+ Finding the start of the thread list
+ ------------------------------------
+ 
+ The beginning of the thread records can be found by using the following
+ algorithm:
+ 
+ Threads := (mark at beginning of header) + (attrib_count) +
+             (filename_length)
+ 
+ 
+ Finding the end of the thread list
+ ----------------------------------
+ 
+ The end of the thread records can be found by applying the following
+ algorithm:
+ 
+ endOfThreads := (mark at beginning of header) + (attrib_count) +
+                  (filename_length) + (16 * total_threads)
+ 
+ 
+ Finding the start of a data_thread
+ ----------------------------------
+ 
+ The beginning of a data_thread can be found using the following algorithm:
+ 
+ Data Mark := (mark at beginning of header) + (attrib_count) +
+                (filename_length) + (16 * total_threads) +
+                  (comp_thread_eof of all threads in the thread list which
+                   are not data prior to finding a data_thread)
+ 
+ 
+ Finding the start of a resource_thread
+ --------------------------------------
+ 
+ The beginning of a resource_thread can be found using the following algorithm:
+ 
+ Resource Mark := (mark at beginning of header) + (attrib_count) +
+                    (filename_length) + (16 * total_threads) +
+                      (comp_thread_eof of all the threads in the thread list
+                       which are not resources prior to finding a
+                       resource_thread)
+ 
+ 
+ Finding the next record
+ -----------------------
+ 
+ The next record can be found using the following algorithm:
+ 
+ Next Mark := (mark at beginning of header) + (attrib_count) +
+               (filename_length) + (16 * total_threads) +
+                (comp_thread_eof of each thread)
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ NuFX Documentation  2/3/89                            - Page 18 -
+ 
+ 
+ Misc notes on threads
+ ---------------------
+ 
+ There must *ALWAYS* be at least 1 thread attached to each record, whether
+ the thread has any physical length or not.  Phantom files and directories will
+ have both the thread_eof and comp_thread_eof fields set to null (00).
+ 
+ If a control_thread indicates that a directory should be created on the
+ destination device, the path to be created must take the form of a prodos
+ partial pathname.  That is, the path must *NOT* be preceded with a volume
+ name.  ie, /STUFF/SUBDIR is an invalid path, while SUBDIR/ANOTHERSUB is
+ a legal path.
+ 
+ If a control_thread indicates that a directory is to be created, *ALL*
+ the subdirectories that are contained in the pathname must be created.
+ 
+ Control_threads will be eventually used to control the execution of utility
+ programs by allowing for directory creation, renaming, deleting, moving,
+ or modifying files.  A form of scripting language will eventually be available
+ to allow utility programs to perform these actions automatically.
+ Control_threads will allow extraction programs to perform operations akin to
+ Apple's GS/OS installer program, allowing updates to program sets dependent
+ upon such things as date of creation or modification, version number, etc.
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ NuFX Documentation  2/3/89                            - Page 19 -
+ 
+ 
+ Normal Files
+ ------------
+ 
+ Normal Prodos files (sub_types $01,$02,$03) should be handled in the
+ following manner: the data portion of the file will occupy the first
+ data_thread.
+ 
+ Sample header block for a normal file record:
+ 
+       00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
+     +-------------------------------------------------+  created 01:10:00
+  00 | 4E F5 46 D8 55 34 3A 00 00 00 01 00 00 00 01 00 |          10/22/88
+  10 | 2f 00 00 00 c3 00 04 00 00 00 00 00 00 00 01 00 |          saturday
+  20 | 00 0a 01 58 16 0a 00 07 00 10 0b 58 11 0b 00 05 |
+  30 | 00 0c 01 28 16 0a 00 07 05 00 53 54 55 46 46 02 | last mod 11:16:00
+  40 | 00 02 00 00 00 00 00 00 20 00 00 00 10 00 00    |          11/17/88
+     +-------------------------------------------------+          thursday
+ 
+                                                         archived 01:12:00
+     header_id             = "NuFX"                               10/22/88
+     header_crc            = $3455                                saturday
+     attrib_count          = $003A (58 bytes in attrib section)
+     version               = $0000
+     total_threads         = $00000001
+     file_sys_id           = $0001 (Prodos)
+     file_sys_info         = $002f (not sparse, / = separator)
+     access                = $000000C3 (full access, not invis)
+     filetype              = $00000004 (Prodos BIN)
+     aux_type              = $00000000
+     storage_type          = $0001 (Prodos sapling file)
+     create_when           = 00 0a 01 58 16 0a 00 07
+     mod_when              = 00 10 0b 58 11 0b 00 05
+     archive_when          = 00 0c 01 58 16 0a 00 07
+ 
+     filename_length       = $0005
+     filename              = "STUFF"
+ 
+     thread_class          = $0002 (data_thread)
+     thread_format         = $0002 (compressed with ShrinkIt)
+     thread_kind           = $0000 (file)
+     reserved              = $0000
+     thread_eof            = $00002000
+     comp_thread_eof       = $00001000 (file is 50% of original size)
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ NuFX Documentation  2/3/89                            - Page 20 -
+ 
+ 
+ Extended Files
+ --------------
+ 
+ Sample header block for an extended file record:
+ 
+       00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
+     +-------------------------------------------------+  created 01:10:00
+  00 | 4E F5 46 D8 65 78 3A 00 00 00 02 00 00 00 01 00 |          10/22/88
+  10 | 2f 00 00 00 c3 00 b3 00 00 00 00 00 00 00 05 00 |          saturday
+  20 | 00 0a 01 58 16 0a 00 07 00 10 0b 58 11 0b 00 05 |
+  30 | 00 0c 01 58 16 0a 00 07 09 00 45 58 54 2e 53 54 | last mod 11:16:00
+  40 | 55 46 46 02 00 02 00 00 00 00 00 00 20 00 00 00 |          11/17/88
+  50 | 08 00 00 02 00 02 00 00 00 00 00 00 10 00 00 00 |          thursday
+  60 | 08 00 00                                        |
+     +-------------------------------------------------+ archived 01:12:00
+                                                                  10/22/88
+     header_id             = "NuFX"                               saturday
+     header_crc            = $7865
+     attrib_count          = $003A (58 bytes in attrib section)
+     version               = $0000
+     total_threads         = $00000002
+     file_sys_id           = $0001 (Prodos)
+     file_sys_info         = $002f (not sparse, / = separator)
+     access                = $000000C3 (full access, not invis)
+     filetype              = $000000B3 (Prodos S16)
+     aux_type              = $00000000
+     storage_type          = $0005 (extended file)
+     create_when           = 00 0a 01 58 16 0a 00 07
+     mod_when              = 00 10 0b 58 11 0b 00 05
+     archive_when          = 00 0c 01 58 16 0a 00 07
+ 
+     filename_length       = $0009
+     filename              = "EXT.STUFF"
+ 
+     thread_class          = $0002 (data_thread)
+     thread_format         = $0002 (compressed by ShrinkIt)
+     thread_kind           = $0000 (file)
+     reserved              = $0000
+     thread_length         = $00002000
+     comp_thread_length    = $00000800 (data_fork is 25% of original size)
+ 
+     thread_class          = $0002 (data_thread)
+     thread_format         = $0002 (compressed by ShrinkIt)
+     thread_kind           = $0002 (resource_fork)
+     reserved              = $0000
+     thread_eof            = $00001000
+     comp_thread_eof       = $00000800 (resource is 50% of original size)
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ NuFX Documentation  2/3/89                            - Page 21 -
+ 
+ 
+ Disks
+ -----
+ 
+ If the file system of a particular disk is not known, the file_sys_id field
+ should be set to null, the volume name should also be set to null, and all
+ the other fields pertaining only to files should be set to null.
+ 
+ If the file system of a particular disk *IS* known, as many of the fields
+ as possible should be filled with the correct information.  Fields which do not
+ pertain to an archived disk should remain set to null.
+ 
+ If an entire disk is added to the archive without some form of compression
+ (ie, record_format = uncompressed), then the blocks which comprise the disk
+ image *MUST* be added sequentially from the first through the last block.
+ Since there will be no character included in the data stream to mark the
+ end/beginning of a block, extraction programs should rely on the
+ file_sys_block_size field to determine how many bytes to read from the record
+ to properly fill a block.
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ NuFX Documentation  2/3/89                            - Page 22 -
+ 
+ 
+ Sample header block for a disk record:
+ 
+       00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
+     +-------------------------------------------------+  created 01:10:00
+  00 | 4E F5 46 D8 67 05 3A 00 00 00 01 00 00 00 01 00 |          10/22/88
+  10 | 2f 00 00 00 00 00 00 00 00 00 40 06 00 00 00 02 |          saturday
+  20 | 00 0a 01 58 16 0a 00 07 00 10 0b 58 11 0b 00 05 |
+  30 | 00 0c 01 58 16 0a 00 07 04 00 44 49 53 4B 02 00 | last mod 11:16:00
+  40 | 02 00 01 00 00 00 00 00 00 00 51 45 07 00       |          11/17/88
+     +-------------------------------------------------+          thursday
+ 
+                                                         archived 01:12:00
+     header_id             = "NuFX"                               10/22/88
+     header_crc            = $0567                                saturday
+     attrib_count          = $003A (58 bytes in attrib section)
+     version               = $0000
+     total_threads         = $00000001 (one thread)
+     file_sys_id           = $0001 (Prodos)
+     file_sys_info         = $002f (not sparse, / = separator)
+     access                = $00000000 (none)
+     filetype              = $00000000 (none)
+     aux_type/creator_type = $00000640 (1600 blocks on device -- 3.5" disk)
+     storage_type          = $0200 (block size = 512 bytes)
+     create_when           = 00 0a 01 58 16 0a 00 07
+     mod_when              = 00 10 0b 58 11 0b 00 05
+     archive_when          = 00 0c 01 58 16 0a 00 07
+ 
+     filename_length       = $0004
+     filename              = "DISK"
+ 
+     thread_class          = $0002 (data_thread)
+     thread_format         = $0002 (compressed with ShrinkIt)
+     thread_kind           = $0001 (disk image)
+     reserved              = $0000
+     thread_eof            = $00000000 (unknown size before compressing)
+     comp_thread_eof       = $00074551 (size after compression)
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ NuFX Documentation  2/3/89                            - Page 23 -
+ 
+ 
+ Directories
+ -----------
+ 
+ Directories are handled almost the same way that normal files are handled with
+ the exception that there will be no data in the thread which follows the entry.
+ A thread_record *MUST* exist to inform a utility that a directory is to be
+ created through the use of the proper control_thread value.
+ 
+ Directories do not necessarily have to precede a record which references a
+ directory.  ie, if a record contains STUFF/MY.STUFF, the directory "STUFF"
+ need not exist for the extracting program to properly extract the record.  The
+ extracting program must check to see if each of the directories referenced
+ exist, and if they do not exist, create them.  While this method places a great
+ burden on the abilities of the extraction program, it avoids the anomalies
+ associated with the deletion of directories within an archive.
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ NuFX Documentation  2/3/89                            - Page 24 -
+ 
+ 
+ Cyclic Redundancy Checks (CRC's)
+ --------------------------------
+ 
+ Many people are not aware of how to calculate a CRC, so to provide this
+ function, I am providing source code to a very fast routine which does the crc
+ calculation.  The routine "makeLookup" needs to be called only once.  After
+ this, the routine "doByte" should be called repeatedly with each new byte in
+ succession to generate the cumulative CRC for the block.  The CRC word should
+ be reset to null (0000) before beginning each new CRC.
+ 
+ This is the same CRC calculation which is done for CRC/Xmodem, and Ymodem.  The
+ code is easily portable to a 16-bit environment like the Apple IIgs.  The only
+ detrimental factor with this routine is that it requires 512 bytes of main
+ memory to operate.  If you can spare the space, this is one of the fastest
+ routines I know to generate a CRC-16 on a 6502-type machine.
+ 
+ 
+ *-------------------------------
+ * fast crc routine based on table lookups by
+ * Andy Nicholas - 03/30/88 - merlin 'c02 - easily portable to nmos 6502 also.
+ * easily portable into orca/m format, just snip and save.
+ 
+          xc                             turn 65c02 opcodes on
+ 
+ *-------------------------------
+ * routine to make the lookup tables
+ *-------------------------------
+ 
+ makeLookup
+          LDX   #0                       zero first page
+ zeroLoop STZ   crclo,x                  zero crc lo bytes
+          STZ   crchi,x                  zero crc hi bytes
+          INX
+          BNE   zeroLoop
+ 
+ *-------------------------------
+ * the following is the normal bitwise computation
+ * tweeked a little to work in the table-maker
+ 
+ docrc
+          LDX   #0                       number to do crc for
+ 
+ fetch    TXA
+          EOR   crchi,x                  add byte into high
+          STA   crchi,x                  of crc
+ 
+          LDY   #8                       do 8 bits
+ loop     ASL   crclo,x                  shift current crc-16 left
+          ROL   crchi,x
+          BCC   loop1
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ NuFX Documentation  2/3/89                            - Page 25 -
+ 
+ 
+ * if previous high bit wasn't set, then don't add crc
+ * polynomial ($1021) into the cumulative crc.  else add it.
+ 
+          LDA   crchi,x                  add hi part of crc poly into
+          EOR   #$10                     cumulative crc hi
+          STA   crchi,x
+ 
+          LDA   crclo,x                  add lo part of crc poly into
+          EOR   #$21                     cumulative crc lo
+          STA   crclo,x
+ loop1    DEY                            do next bit
+          BNE   loop                     done? nope, loop
+ 
+          INX                            do next number in series (0-255)
+          BNE   fetch                    didn't roll over, so fetch more
+          RTS                            done
+ 
+ crclo    ds    256                      space for low byte of crc table
+ crchi    ds    256                      space for high bytes of crc table
+ 
+ 
+ *-------------------------------
+ * do a crc on 1 byte/fast
+ * on initial entry, CRC should be initialized to 0000
+ * on entry, A = byte to be included in CRC
+ * on exit, CRC = new CRC
+ *-------------------------------
+ 
+ doByte
+          EOR   crc+1                    add byte into crc hi byte
+          TAX                            to make offset into tables
+ 
+          LDA   crc                      get previous lo byte back
+          EOR   crchi,x                  add it to the proper table entry
+          STA   crc+1                    save it
+ 
+          LDA   crclo,x                  get new lo byte
+          STA   crc                      save it back
+ 
+          RTS                            all done
+ 
+ crc      dw    0000                     cumulative crc for all data
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ NuFX Documentation  2/3/89                            - Page 26 -
+ 
+ 
+ Possible Block Combinations
+ ---------------------------
+ 
+ The blocks *MUST* occur in the following fashion:
+ 
+         Master Header block containing N entries
+ 
+         Header block
+         threads (message, control, data, or resource)
+ 
+         .
+         .
+         .
+ 
+         Next Header block (notice no second Master Header block)
+         threads (message, control, data, or resource)
+ 
+         .
+         .
+         .
+ 
+         Nth Header Block
+         threads (message, control, data, or resource)
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ NuFX Documentation  2/3/89                            - Page 27 -
+ 
+ 
+ Known NuFX utility programs:
+ ---------------------------
+ 
+ Name           Author          Description                          Current Ver
+ -------------  --------------  ------------------------------------ -----------
+ ShrinkIt       Andy Nicholas   Compresses files/disks, provides        0.95
+                                 archive and file utilities.
+ NuList         Andy Nicholas   Lists contents of NuFX files for        1.1
+                                 the GBBS "Pro" (tm) BBS, online.
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ NuFX Documentation  2/3/89                            - Page 28 -
+ 
+ 
+ Legal Stuff
+ -----------
+ 
+ Apple, Apple IIGS, AppleLink, GS/OS, Macintosh, and Lisa are registered
+  trademarks of Apple Computer, Inc.
+ 
+ GBBS "Pro" is a registered trademark of L&L Productions.
+ 
+ 
+ About the Author
+ ----------------
+ 
+ I am currently a Junior attending Moravian College in Bethlehem, Pennsylvania,
+ majoring in Computer Science.
+ 
+ Any comments or suggestions you have about NuFX are more than welcome, or if
+ you wish to request that any of the fields be assigned your own value, or if
+ you would like to inform me of a NuFX utility you have written, you can contact
+ me at:
+ 
+ 
+         Paper Bag Productions
+         c/o Andy Nicholas
+         Box 435
+         Moravian College
+         Bethlehem, PA  18018
+ 
+         CSNET       :  nicholaA@moravian.edu
+         InterNET    :  nicholaA%batman.moravian.edu@relay.cs.net
+                        nicholaA@batman.moravian.edu.csnet
+         ProLine     :  andyn@pro-sol.cts.com      [619-670-5379]
+         AppleLink PE:  ShrinkIt
+ 
+ 
+ I would like to thank the following people for their help and input during the
+ design phase of the NuFX proposal:
+ 
+ Jason Blochowiak, Morgan Davis, Don Elton, Dave Lyons, Jon Davidson,
+ Vince Cooper, Lance Taylor-Warren, Floyd Zink, Kent Dickey, John Brooks,
+ Doug Brandon, Todd South, Larry Hawkins, Kevin Keller
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+
+ +
+ + diff --git a/library/shrinkit.sdk b/library/shrinkit.sdk new file mode 100644 index 0000000..f12bbe8 Binary files /dev/null and b/library/shrinkit.sdk differ diff --git a/library/yanksrc.shk b/library/yanksrc.shk new file mode 100644 index 0000000..fa8adf9 Binary files /dev/null and b/library/yanksrc.shk differ diff --git a/nufxlibapi.htm b/nufxlibapi.htm new file mode 100644 index 0000000..63ade1b --- /dev/null +++ b/nufxlibapi.htm @@ -0,0 +1,2006 @@ + + + + + + + +NufxLib API + + + +
+ +

NufxLib API
+[ Home ] [ NuLib Downloads ] [ NuLib Library ] [ NuLib2 Manual ] [ NufxLib API ] [ Bugs & Features ]

+
+ +
+ +
 
+
NufxLib v2.2.0 API - By Andy McFadden - Last revised 2005/09/17
+

Table of contents

+ +

Introduction

+

NuFX, short for "New File Exchange", is a file format developed by +Andy Nicholas for archiving files and disks on the Apple II series of +computers.  The format was devised in tandem with the +development of ShrinkIt, which became the standard archive software for the +Apple II soon after it's release in 1989.  NuFX archives usually have filenames that end in ".SHK".

+

This document describes the API (Application Program Interface) for NufxLib, +a library of functions that manipulate NuFX archives.

+

Good engineering practices dictate that an API should be minimal and complete.  The +confusion generated by redundant and overlapping interfaces can be as harmful as +an omitted vital feature.  I feel pretty good about the +"complete" part, since NufxLib provides a way to do pretty much +everything that I can reasonably expect somebody to want to do, but in some +cases "minimal" has been swept aside in the name of convenience.

+

The NuFX specification is extremely general, and does not explicitly allow or +forbid unusual conditions like having a record with two filenames in it.  +NufxLib follows the NuFX specification on everything that is spelled out, but +restricts some of the undefined behaviors to the subset defined in the NuFX +Addendum.

+

This document is a bit long for a single page, but it's the sort of thing you +want to print out.  Whenever you see "threads" think NuFX +threads, not POSIX threads.

+

Goals

+
    +
  • Provide a complete set of function calls for manipulating NuFX archives.
    • +
  • Be portable.  Keep most of the system-dependent code in the application rather than the + library, and use plain old ANSI C.
    • +
  • Define the functions so that command-line, GUI, and embedded applications + can be developed easily.
    • +
  • Obey "nice library" semantics.  No globals, no + "static" local variables, avoid non-reentrant libc calls, + etc.  It should be possible for the application to have more than one + NuFX archive open at a time, and be able to perform operations on both + simultaneously.
    • +
  • Be efficient.  Don't allocate unneeded memory, don't waste time + reading and writing unnecessarily, and don't assume that archives are small + relative to other system resources.
    • +
  • When in doubt, do what ShrinkIt does.  There are aspects of the NuFX + file format (notably regarding Threads) where some things aren't well + defined.  Emulating ShrinkIt avoids compatibility problems.
    • +
+

Not Goals

+
    +
  • Don't try to provide an interface that works for all archive + formats.  NuFX is a very complex format, developed for a specific + platform, and attempting to describe it and other archive formats in a + general way would result in a big mess.  It is assumed + that anyone using this library has at least a passing familiarity with the + NuFX file format, so instead of passing around generalized structures, the + application gets full read access to all fields in Records, Threads, and the + Master Header.
    • +
  • Don't try to provide everything an Apple II archiver needs.  Binary + II and BinSCII support are useful, but they don't need to be part of the NufxLib + library.
    • +
  • Don't do things the application should be doing.  Recursive directory + tree traversals are system-dependent and may not even be desirable.  + Apple II disk images come in a plethora of formats, but there's no need for + the library to understand them all.  And things like filesystem-dependent + filename remapping are best left to the application.
    • +
+

That explains what I set out to do.  Here's a quick summary of what I +accomplished.

+

Features

+
    +
  • Full set of calls for adding, extracting, listing, updating, renaming, + deleting, and testing Records and Threads.
    • +
  • All write operations are queued up and deferred until a "flush" + call is issued, then executed in the most efficient manner possible.
    • +
  • Automatically handles Binary II and GSHK Self-Extracting Archive + wrappers.  BXY, SEA, and BSE files are handled as easily as SHK.  + Can also skip over junk left at the start of the file, such as MacBinary + headers.
    • +
  • Supports compression and expansion with the ShrinkIt LZW/1 and LZW/2 + algorithms, the SQueeze algorithm used by BLU (and nominally supported by + ShrinkIt), LZC (i.e LZW in UNIX "compress"), and optionally zlib's + "deflate" algorithm and libbz2's BWT+Huffman.
    • +
  • Can (optionally) update an archive in place (i.e. without writing to and + later renaming a + temp file) if the only operations are limited to pre-sized thread updates + and/or the addition of new files.
    • +
  • Can perform most of the read-only operations on a "streaming" + archive, i.e. one where the archive is presented as a pipe or shell + redirect.
    • +
  • Data can be added from or extracted to a file on disk, FILE*, or memory + buffer.
    • +
  • End-of-line conversions can be performed during extraction.  Text + files can be detected automatically or identified explicitly.
    • +
  • Has no hard-coded size restrictions other than those imposed by C library + functions.  (It + has some "reasonableness" tests based on compile-time constants, + but they're set pretty high and are easy to change.  They're intended + to spot corrupt archives, not define limits.)
    • +
+

The library is protected by copyright, but can be distributed under the terms +of theBSD License.  See the file "COPYING-LIB" for +full details.

+

Interface changes from v1.x to v2.x

+

Some changes were made during the development of NufxLib that broke binary +compatibility with version 1.1 of the library.  The changes were:

+
    +
  • NufxLib no longer call free() or fclose() on resources allocated by the + application.  (This was done for some DataSource objects that were part + of "bulk" adds and would've been inconvenient for the application + to manage.)  Instead, the application passes a pointer to a + "resource release" function that will be called when + appropriate.  This makes Win32 DLL implementations more likely to work, + and allows C++ applications to use customized "new" and + "delete".  If your application creates DataSource objects, it + will need to change the order of some of the arguments.
    • +
  • Callback setters no longer return NuError.  They now return the + previous callback function.
    • +
  • A new progress state, kNuProgressAborted, was added so the progress + updater can correctly handle a "cancel" button.
    • +
  • A new field, intended to allow applications to access the original + filename in "progress" and "error" callbacks, has been + added to NuFileDetails and NuErrorStatus.
    • +
+

In addition, a NuTestRecord call was added.

+

Applications written against v1.x may need to be updated.  Check the +NufxLib "samples" directory for examples of programs that use the +updated calls.

+

To make version management easier, v2.x includes the version number in the +NufxLib.h header file.  This allows dynamically-linked applications to +compare a "compiled" version against a "linked" version.

+
+

NuFX Archive Format Overview

+

This document assumes that you are already familiar with the NuFX archive +format, as described in the Apple II File Type +Note for $e0/8002 and the Winter 1990 issue of Call-A.P.P.L.E.  For those unwilling to wade through the technical +documentation, here is a quick overview.

+

A NuFX archive is composed of a Master Header followed by a series of +Records.  Each Record is composed of a Record Header and one or more Threads.  The general +idea is to store one file per Record.

+

Each Thread holds a blob of data.  The data can be a data or resource +fork of a file, a disk image, a comment, or the filename for the +Record.  The Threads are identifed by a "class" and a +"kind".  The "class" tells you if it's a data thread, +comment, filename, or something else, and the "kind" refines the +class.  For example, the resource fork of a file is a data-class thread with a +"kind" of 2.

+

Some Threads, notably filenames and comments, are pre-sized, meaning that the +space allocated for them in the archive is larger than what is actually +used.  Filenames usually have at least 32 bytes set aside for them, though +in practice a simple ProDOS filename will be shorter.  This makes it +possible to rename files and update comments without having to reconstruct the +archive.

+

The archive Master Header has only a few bits of information, such as the +number of records and the date the archive was created.  Unlike a ZIP +archive, NuFX has no central table of contents.  If you want to display the +contents of an archive, you have to read the first Record header, pull the +filename out (usually by finding and reading a filename Thread), compute the total +size of the Record, and seek forward past the data.  Repeat the +process with each subsequent record, until you reach the end of the archive. 

+

The predominant compression algorithm is a slightly modified LZW (Ziv-Lempel-Welch).  +It's +fast, but not very effective compared to the standard methods used in modern +archivers.

+

API Overview

+

There are five basic categories of API calls.

+

ReadOnly calls do not modify the archive in any way.  The +operations include things like listing and extracting files.  These can be +used on archive files opened read-only or read-write.

+

StreamingReadOnly calls are a subset of ReadOnly calls that can be +made on a streaming archive.  A "streaming" archive is one that +cannot be seeked, e.g. an archive being received over a network socket or a pipe +from stdin.  The same functions are invoked as for ReadOnly archives, but +in some rare cases the results may be different.

+

ReadWrite calls change the archive contents.  Functions that add +and delete files are here.  These can only be used on archive files opened +read-write.

+

General calls can be made regardless of how the archive was +opened.  Functions included here can get and set archive parameters and +define callbacks.

+

Helper functions don't do anything to the archive.  They're functions +or macros that do useful things with some of the data types returned.  +They're included as a convenience.

+

The library does everything it can to aid multi-threading.  You should +be able to perform simultaneous operations on multiple archives (assuming you +have the reentrant versions of certain libc calls available).  You cannot, +however, invoke multiple simultaneous operations on a single archive.

+

There is a general philosophy of laziness employed.  For example, +opening an archive does not cause the entire table of contents to be read.  +(In a NuFX archive, that would require scanning through most of the file.)  +As a result, there are actually three different ways to get the table of +contents out of an archive:

+
    +
  1. Read-as-you-go, forgetful.  On StreamingReadOnly files, we handle an + individual record and then throw the data away.  We can't seek back to + deal with the record again, so there's no point in holding onto it.  + This feature allows applications with very limited memory to list the + contents of and extract files + from very large archives.
    2. +
  2. Read-as-you-go.  While performing a whole-archive operation (e.g. + getting the complete list of contents, or extracting all files) on a + non-streaming file, we read and save the table of contents as we go.  + This saves us from having to scan through the archive once to get the + contents, and then running through it again to extract the files.  This + might seem silly, but if you're extracting from an archive on a slow medium + (e.g. floppy or a sluggish CD-ROM drive), it cuts + the time required nearly in half.
    3. +
  3. Read up front.  For operations like single-file extractions and + anything in the ReadWrite category, we want to have full knowledge of the + archive up front.  There are cases where this will be less efficient + than a more cleverly designed algorithm, but it's much simpler this way.
    4. +
+

For write operations, a certain form of laziness is again employed.  If +you want to delete three records from various points in an archive, you don't +want to have to update the archive three times.  NufxLib handles this by +deferring all write operations until a "flush" call is made.  In most cases, a +"flush" results in a new archive being constructed in a temp file, +which is subsequently renamed over the original.  The flush call does not +close the archive, so it is possible to do things like:

+ +
    +
  1. Open the archive.
    2. +
  2. Queue up a bunch of operations.
    3. +
  3. Flush changes.
    4. +
  4. Queue up some more operations.
    5. +
  5. Abort changes.
    6. +
  6. Queue up yet more operations.
    7. +
  7. Flush changes.
    8. +
  8. Close archive.
    9. +
 +

In certain restricted cases, such as updating a comment or appending new +records, +the original archive can (optionally) be updated in place, saving a potentially +lengthy copying of data.

+

As a final example of laziness, NufxLib does not re-read the archive it has +just written after a Flush.  It would have been easier to write all +changes, throw out all data structures, and re-read the archive from scratch, +but that could be slow.  Instead, the library keeps track of the changes it +has made -- something that gets a little tricky when filename threads are +updated.  Being lazy is often more work.

+

Data Types and Source Conventions

+

All API calls and data types begin with "Nu", and all constants start with "kNu".  +All internal functions start with "Nu_", and any internal data tables with +global scope start with "gNu".  Hopefully these rules will avoid +compile-time and link-time name conflicts.

+ +

For details about the fields available in different structures, see the +NufxLib.h header file.  Everything in NufxLib.h is +public.  Most of these types have a direct analog with a +field or structure in the NuFX specification.

+ +

NuError (enum): Most library functions return NuError.  A value +of zero (kNuErrNone) indicates success, anything else indicates failure.  +Values less than zero are NufxLib errors, while values greater than zero are +system errors (like ENOENT).

+ +

NuResult (enum): Callback functions return these values to tell +NufxLib how things went.  For example, an error callback can tell the +library to Abort, Retry, or Skip.  (Okay, it can Ignore too.)

+ +

NuRecordIdx and NuThreadIdx (unsigned long): These are used to +identify a specific record or thread in API calls.  Their values are assigned when the archive file is +read.  They aren't reused, so if you delete some records and add some new +ones, the indices of the deleted records won't appear again.  Do not assume +that the indices start at a specific value or are assigned in a particular +order.  The indices are assigned when the archive is opened, and if you +close and reopen the archive, they may be completely different.

+ +

NuThreadID (unsigned long): This is a combination of the 16-bit +"thread class" and the 16-bit "thread kind".  Constants +are defined for common values, e.g. kNuThreadIDDataFork (0x00020000) indicates a data fork.

+ +

NuThreadFormat (enum): An enumeration of constants representing the +16-bit "thread format" value.  This is used to specify a type of +compression (uncompressed, LZW/1, LZW/2, etc).

+ +

NuFileSysID (enum): An enumeration of GS/OS file system identifiers.

+ +

NuStorageType (enum): An enumeration of ProDOS storage types.  +There are extended (forked) files, directories, and three types of plain files.

+ +

NuArchive (opaque struct): This is the fundamental state structure for +all API calls.  Every call takes one of these as an argument.  The +structure contains all of the information about the archive and pending +operations.

+ +

NuCallback (pointer to function): Callback function declarations must +match this type.  An example would be "NuResult MyFunction(NuArchive* pArchive, void* args)".

+ +

NuValueID (enum): An identifier for settable values.  You can +change certain NufxLib parameters after opening an archive.  This enum is +how you specify which parameter you want to change.

+ +

NuValue (unsigned long): The new value for the parameter specified by +the NuValueID.

+ +

NuAttrID (enum): An identifier for archive attributes.  You can +get information about archive attributes (characteristics of the archive itself) +through a NufxLib interface.  This type has an enumeration of the legal +values.

+ +

NuAttr (unsigned long): The value for the attribute specified by the +NuAttrID is placed in one of these.

+ +

NuDataSource (opaque struct): Some of the fancier NufxLib calls allow +you to use data from a file on disk, a file that's already open, or a buffer of +memory.  This struct contains that specification.

+ +

NuDataSink (opaque struct): Like NuDataSource, this specifies a data +location.  This struct is for data being extracted.

+ +

NuDateTime (struct): This holds the date and time in an expanded +format, using the same structure as TimeRec from "misctool.h" on the +IIgs.

+ +

NuThread (struct): The fields from the thread header, as well as a few +new ones like the absolute file offset, are accessible.

+ +

NuRecord (struct): This has all of the fields from the NuFX Record +structure, as well as some convenience fields (like "filename", which +always points to the right filename whether it was stored in the record header +or came out of a thread).  Some calls cause a NuRecord structure to be +passed to a callback function, where it can be accessed directly.  The +Threads are represented as an array of NuThread structures attached to the +NuRecord.

+ +

NuMasterHeader (struct): This holds the data from the archive's master +header block.

+ +

NuRecordAttr (struct): Some of the fields in a NuRecord can be +changed, such as the file type and modification date.  This structure +contains the modifiable fields, and is used as an argument to two of the API +calls.

+ +

NuFileDetails (struct): When adding files, it is up to the application +to supply many of the details about the file, such as the file type, access +permissions, and modification date.  This structure provides a way to pass +those values into the library.

+ +

NuSelectionProposal (struct): Selection callback functions receive one +of these.

+ +

NuPathnameProposal (struct): Pathname filter callback functions +receive one of these.

+ +

NuProgressData (struct): Progress update callback functions receive +one of these.

+ +

NuProgressState (enum): A component of NuProgressData, this tells the +callback function what the library is doing.

+ +

NuErrorStatus (struct): Error handling callback functions receive one +of these.

+ +

Files are referenced with standard libc FILE* pointers.  The +library uses fseek and ftell, which are defined by POSIX to take a signed long +integer for the offset argument, so archives larger than 2GB cannot be handled.

+ +
+

ReadOnly Interfaces

+ +

These interfaces can be used on read-only and read-write archives.  A +subset, described later, can also be used on streaming-read-only archives.

+ +

NuError NuOpenRO(const char* archivePathname, NuArchive** ppArchive)

+

Creates a new NuArchive structure for the "archivePathname" +file.  The file will be opened in read-only mode.

+

Attempting to use ReadWrite interfaces on a read-only archive will fail.

+

NuError NuContents(NuArchive* pArchive, NuCallback contentFunc)

+

Read the list of entries from the archive.  If the full table of +contents has already been read, the in-memory copy will be used.

+

"contentFunc" +is a callback function that will be called once for every record in the archive.  +The callback function should look something like this:

+
+

NuResult EntryListing(NuArchive* pArchive, const NuRecord* pRecord)

+
+

(Depending on your compiler, you may have to declare "pRecord" as a void* +and cast it in the function.)

+

The record passed to the callback function does not reflect the results of +any un-flushed changes.  Additions, deletions, and updates will not be +visible until NuFlush is called.

+

The application must not attempt to retain a copy of "pRecord" +after the callback returns, as the structure may be freed.  Anything of interest should be copied out.

+

NuError NuExtract(NuArchive* pArchive)

+

Try to extract all files from the archive.  Each entry is passed through +the SelectionFilter callback, if one has been supplied, to determine whether or +not it should be extracted.  The OutputPathnameFilter callback is invoked +to covert the filenames to something appropriate for the target filesystem.

+

On systems that support forked files, a record with both data and resource +forks can be extracted to the individual forks of the same file.  On +systems without native support for forks, the data can be extracted into two +different files by using the OutputPathnameFilter.  If the system doesn't +support forks, and no OutputPathnameFilter is specified, then the forks will be +extracted into the same file.  Depending on the value of the +kNuValueHandleExisting parameter, this could result in one fork overwriting the +other, in one fork not getting extracted, or in the HandleError callback getting +invoked.  (The HandleError callback can choose to rename the file, +overwrite it, skip the current entry, or abort the entire process.)

+

The global EOL conversion setting is applied to all threads, but is +automatically turned off for disk image threads.

+

NuError NuExtractRecord(NuArchive* pArchive, NuRecordIdx recordIdx)

+

Extract a single record.  Otherwise identical to NuExtract.  The +SelectionFilter callback, if specified, will be invoked.

+

There are a number of ways to get the recordIdx.  You can call +NuContents and use the callback to find the one you want.  You can get the recordIdx by the filename stored in the +archive, with +NuGetRecordIdxByFilename.  Or, you can get it by the record's offset in the archive, +using NuGetRecordIdxByPosition.

+

NuError NuExtractThread(NuArchive* pArchive, +NuThreadIdx threadIdx, NuDataSink* pDataSink)

+

Extract a single thread.  Specify the thread index and a place to put +the data.  The SelectionFilter callback, if specified, will be invoked.

+

Remember that, if EOL conversion is enabled in the data sink, the amount of data that comes +out of a thread may not match pThread's "actualThreadEOF" value.

+

(In some ways it doesn't really make sense to call the SelectionFilter callback +when a specific thread has been singled out for extraction.  However, it's +easy to disable (set the callback to NULL), it may prove useful, and it keeps +the interface consistent.)

+

NuError NuTest(NuArchive* pArchive)

+

The NuTest call is functionally equivalent to NuExtract in every way but one: +it doesn't actually extract anything.  If you want to test a subset of the +files, supply a SelectionFilter callback.

+

This won't test filenames or comments because those aren't extracted by +NuExtract.  However, since such threads don't have CRCs, there's really +nothing to test anyway.  The parts that can be tested for correctness are +verified automatically when the archive table of contents is read.

+

NuError NuTestRecord(NuArchive* pArchive, NuRecordIdx recordIdx)

+

A single-record version of NuTest.

+

NuError NuGetRecord(NuArchive* pArchive, NuRecordIdx recordIdx, +const NuRecord** pRecord)

+

Get a pointer to the record header.  The thread array can be accessed +through this pointer.

+

As with callbacks, when you get a const pointer, it is very important that +you don't try to modify it.  The structure pointed to is part of the +current archive state, so the effects of changes are unpredictable.  If you +wish to alter fields in the Record header, use the NuSetRecordAttr call.

+

IMPORTANT: you must discard this pointer if you call NuFlush or NuClose.

+

NuError NuGetRecordIdxByName(NuArchive* pArchive, const char* name, +NuRecordIdx* pRecordIdx)

+

Get the recordIdx for the first record in the archive whose case-insensitive +filename matches "name".  The value retrieved can be used with +any call that takes a NuThreadIdx argument.

+

The "name" string must match the record's filename exactly, +including the filename separator character.

+

If you know what you want to extract from an archive by name, use this.

+

NuError NuGetRecordIdxByPosition(NuArchive* pArchive, unsigned long +position, NuRecordIdx* pRecordIdx)

+

Get the recordIdx for nth record in the archive.  "position" +is zero-based, meaning the very first record in the archive is at position 0, +the next is at position 1, and so on.  The value retrieved can be used with +any call that takes a NuRecordIdx argument.

+

This could be useful when an application is certain that it is only +interested in the very first record in the archive, e.g. an Apple II emulator +opening a disk image.

+
+

StreamingReadOnly Interfaces

+

A streaming +archive is presented to the library as a FILE* that can't be seeked, generally +because it was handed to the application via a pipe or shell redirect.  A +subset of the ReadOnly interfaces are supported.  All of them leave the +stream pointed at the first byte past the end of the archive.

+

This calls are also useful for files on disk in situations where memory is +at a premium.  Because it's impossible to seek backwards in the archive, no +attempt is made to remember anything about records other than the one most +recently read.

+

The interfaces supported are:

+
+
NuContents +
Behaves just like the non-streaming version. +
NuTest
+
Behaves just like the non-streaming version.
+
NuExtract
+
Behaves like the non-streaming version for well-formed archives.  If a filename is stored in a thread, and the filename thread comes after a data + thread, NufxLib would need to extract the data before it knows what filename to + use.  This currently results in an error.
+
+ +

There is one interface that only applies to StreamingReadOnly archives:

+ +

NuError NuStreamOpenRO(FILE* infp, NuArchive** ppArchive)

+

Creates a new NuArchive structure for "infp".  The file must +be positioned at the start of the archive.

+

It should be possible to concatenate multiple archives together, and use them +by issuing consecutive NuStreamOpenRO calls.

+

If your system requires fopen(filename, "rb") instead of +"r" (e.g. Win32), make sure the archive file was opened with +"b", or you may get "unexpected EOF" complaints.

+
+

ReadWrite Interfaces

+

NuError NuOpenRW(const char* archivePathname, const char* +tempPathname, unsigned long flags, NuArchive** ppArchive)

+

Open a file for read-write operations.  A pointer to the new archive is +returned via "ppArchive".

+

"archivePathname" is the name of the archive to open.  If the +file has zero length, the archive will be treated as if NufxLib had just created +it.

+

"tempPathname" is the name of the temp file to use.  The call +will fail if the temp file already exists.  The temp file must be in a +location that allows it to be renamed over the original archive when a +"flush" operation has completed.  If "tempPathname" +ends in six 'X's, e.g. "tmpXXXXXX", the name will be treated as a +mktemp-style pattern, and a unique six-character string will be substituted +before the file is opened.  Note that the temp file will be opened +even if "kNuValueModifyOrig" is set.

+

"flags" is a bit vector of boolean flags that affect how the +archive is opened.  If no flags are set, and the archive doesn't exist, the +call will fail.  If "kNuOpenCreat" is set, the archive will be +created if it doesn't exist.  If "kNuOpenCreat" and "kNuOpenExcl" +are both set, the call will fail if "archivePathname" already exists +(i.e. the archive *must* be created).

+

If the archive was just created, "kNuValueModifyOrig" will be set +to "true".

+

NufxLib can tell the difference between a BXY file (NuFX in a Binary II +wrapper) and a BNY file with several entries whose first entry happens to be a NuFX +archive.  Access to BNY files that happen to have a ShrinkIt archive in +them isn't supported.

+

NuError NuFlush(NuArchive* pArchive, long *pStatusFlags)

+

Commits all pending write operations.

+

"pStatusFlags" gets a bit vector of flags regarding the status of +the archive.  If a non-kNuErrNone result is returned, "pStatusFlags" +may contain one or more of the following:

+
    +
  • kNuFlushSucceeded - the flush succeeded, but something failed after the + archive was written (e.g. the archive file couldn't be removed, or the temp + file couldn't be renamed over the original)
    • +
  • kNuFlushAborted - the flush failed, and all of your changes were thrown + out
    • +
  • kNuFlushCorrupted - a flush to the original archive file failed, possibly + resulting in corruption to the archive (kNuValueModifyOrig must be set for + this to happen)
    • +
  • kNuFlushReadOnly - because of the failure, the archive has been placed + into read-only mode
    • +
  • kNuFlushInaccessible - because of the failure, the archive is no longer + accessible, and should be closed (can happen if we successfully rename the + temp file over the original, but can't reopen the archive)
    • +
+

Some of the above are mutually exclusive, e.g. only one of kNuFlushSucceeded, +kNuFlushAborted, and kNuFlushCorrupted will be set.

+

Any records without threads -- either created that way or having had all +threads deleted -- will be removed.  Newly-created records without filename +threads will have one added.  (Existing records without filenames are +frowned upon but left alone.)

+

Normally, the archive is reconstructed in the temp file, and the temp file is +renamed over the original archive after all of the operations have completed +successfully.  As a performance optimization, if kNuValueModifyOrig is +"true", NuFlush will try to modify the archive in place.  This is +only possible if the changes made to the archive consist entirely of additions of new +files, updates to pre-sized threads, and/or setting record attributes.  If +other changes have been made, the update will be done through the temp file.

+

If an operation fails during the flush, all changes will be aborted.  If +something fails in a way that can't be recovered from, such as failing to rename +the temp file after a successful flush or failing partway through an update to +the original archive, the archive may be switched to read-only mode to prevent +future operations from compounding the problem.

+

NuError NuAddRecord(NuArchive* pArchive, const NuFileDetails* +pFileDetails, NuRecordIdx* pRecordIdx)

+

Add a new record with no threads.  The index of the created record is +returned in "pRecordIdx".  This always creates a "version +3" record, and expects that the filename will be stored in a thread.

+

"pFileDetails" is a pointer to a NuFileDetails structure.  +This contains most of the interesting fields in a record, such as access flags, +dates, file types, and the filename.  The "threadID" field is +ignored for this call.

+

"pRecordIdx" may be NULL.  However, the only way to add +threads to the record is with NuAddThread, which requires the record index as a +parameter, so you almost certainly want to get this value.

+

If no filename thread is added, the NuFlush call will use the "storageName" +field from the "pFileDetails" parameter to create a filename thread +for it.

+

If no threads are added at all, the NuFlush call will throw the record away.

+

The "pFileDetails->storageName" may not start with the filename separator +argument, e.g. "/tmp/foo" is illegal but "tmp/foo" is okay.

+

If a disk image thread is added to the record, and the "storageType" +and "extraType" values set by "pFileDetails" aren't +compatible, the entries will be replaced +with values appropriate for the thread.  For records with non-disk data-class +threads, the storageType will be adjusted when necessary.

+

Depending on the values of kNuValueAllowDuplicates and kNuValueHandleExisting, +this may replace an existing record in the archive.  See Replacing +Existing Records and Files for details.

+

NuError NuAddThread(NuArchive* pArchive, NuRecordIdx recordIdx, +NuThreadID threadID, NuDataSource* pDataSource, NuThreadIdx* pThreadIdx)

+

Add a new thread to a record.  You may add threads to an existing record +or a newly created one.  Some combinations of threads are not allowed, and +will cause an error to be returned.  (See the NuFX +Addendum for details.)

+

"recordIdx" is the index of the record being added to.

+

"threadID" is the class and kind of the thread being added.  +This defines how the data is labeled in the archive, and whether the contents of +"pDataSource" are to be regarded as pre-sized or not.

+

"pDataSource" is where the data comes from.  If the source is +uncompressed, the thread will be compressed with the compression value currently +defined by kNuValueDataCompression.  (You can set the value independently +for each call to NuAddThread.)  Only data-class threads will be compressed.  +If you're adding a pre-sized thread, such as a comment or filename, set the +"otherLen" field in the data source.

+

"pThreadIdx" gets the thread index of the newly created +thread.  This parameter may be set to NULL.

+

Threads will be arranged in an appropriate order that may not be the same as +the order in which NuAddThread was called.

+

If "threadID" indicates the thread is a disk image, then the +uncompressed length must either be a multiple of 512 bytes, or must be equal to +(recExtraType * recStorageType) in the record header.

+

On successful completion, the library takes ownership of "pDataSource".  +The structure will be freed after a NuFlush call completes successfully or all +changes are aborted.  Until NuFlush or NuAbort completes, it is vital that +you don't free the underlying resource.  That is, don't close the FILE*, delete the file, or +free the buffer that the data source references.  If you don't want to keep +track of the resources used by FP and Buffer sources, you can specify "fcloseFunc" +or "freeFunc" functions to have them released automatically.  See the explanation of +NuDataSource for details.

+

NuError NuAddFile(NuArchive* pArchive, const char* pathname, const +NuFileDetails* pFileDetails, short fromRsrcFork, NuRecordIdx* pRecordIdx)

+

Add a file to the archive.  This is a combination of NuAddRecord and +NuAddThread, but goes a little beyond that.  If you add a file whose +pFileDetails->threadID indicates a data fork, and another file whose +pFileDetails->threadID indicates a resource fork, and both files have the +same pFileDetails->storageName, then the two files will be combined into a +single record. 

+

"pathname" is how to open the file.  It does not have any +bearing on the filename stored in the archive.  Because all write +operations are deferred, NufxLib will not open or even test the existence of the +file before NuFlush is called.

+

"pFileDetails" describes the file types, dates, and access flags +associated with the file, as well as the filename that will be stored in the +archive ("storageName").  If two forks are placed in the same +record, whichever was added first will determine the record's characteristics.

+

"fromRsrcFork" should be set if NufxLib should get the data out of +the "pathname" file's resource fork.  If the underlying +filesystem doesn't support resource forks, then the argument has no +effect.  It does not have any impact on whether the data is stored as a +data fork thread or resource fork thread -- that is decided by the "threadID" +field of "pFileDetails".

+

"pRecordIdx" gets the record index of the new (or existing) +record.  This argument may be NULL.

+

The "pFileDetails->storageName" may not start with the filename separator +argument, i.e. "/tmp/foo" is illegal but "tmp/foo" is okay.

+

If "pFileDetails->threadID" indicates the thread is a disk +image, then the uncompressed length must either be a multiple of 512 bytes, or +must be equal to recExtraType * recStorageType.

+

On systems with forked files, such as GS/OS and Mac OS, it will be necessary +to call NuAddFile twice on forked files.  The call will automatically join +forks with identical names.

+

Depending on the values of kNuValueAllowDuplicates and kNuValueHandleExisting, +this may replace an existing record in the archive.  See Replacing +Existing Records and Files for details.

+

Adding a directory will not cause NufxLib to recursively descend through the +directory hierarchy.  That's the application's job.  Requests to add +directories are currently ignored.  [A future release may add a +"create directory" control thread, so we can store empty directories.]

+

NuError NuRename(NuArchive* pArchive, NuRecordIdx recordIdx, const +char* pathname, char fssep)

+

Rename an existing record.  Pass in the index of the record to update, +the new name, and the filename separator character.  Setting the name to an +empty string is not permitted.

+

This call will do one of three things to the archive.  If a filename +thread is present in the record, and it has enough room to hold the new +filename, then the existing thread will be updated.  If a filename thread +is present, but doesn't have enough space to hold the new name, then the +existing thread will be deleted and a new filename thread will be added.  +Finally, if no filename thread is present, a new one will be added, and the +filename in the record header (if one was set) will be dropped.

+

NufxLib does not currently test for the existence of records with an +identical name.  This is probably a bug (ought to obey the +kNuValueAllowDuplicates setting).

+

NuError NuSetRecordAttr(NuArchive* pArchive, NuRecordIdx recordIdx, +const NuRecordAttr* pRecordAttr)

+

Set a record's attributes.  The fields in the NuRecordAttr struct +replace the fields in the record.  This can be used to change filetypes, +modification dates, access flags, and the file_sys_id field.

+

The changes become visible to NuContents calls only after NuFlush is called.

+

You can fill in values in the NuRecordAttr from a NuRecord struct with the NuRecordCopyAttr +call.

+

NuError NuUpdatePresizedThread(NuArchive* pArchive, NuThreadIdx +threadIdx, NuDataSource* pDataSource, long* pMaxLen)

+

Update the contents of a pre-sized thread.  This can only be used on +filename and comment threads.  Attempting to use it on other threads +results in a kNuErrNotPreSized return value.

+

"threadIdx" is the index of the thread to update, and "pDataSource" +is where the data comes from.  The "otherLen" field in "pDataSource" +is ignored, because this call cannot be used to resize an existing thread.  +(The only way to do that is to delete the thread and then create a new one.)

+

"pMaxLen" will hold the maximum size of the thread if the call +succeeds.  If the call fails because the existing thread is too small, kNuErrNotPreSized +is returned and "pMaxLen" will be valid.  (You can also get the +size by examining the thread's thCompThreadEOF field.)

+

This cannot be used on newly-added, deleted, or updated threads.

+

On successful completion, the library takes ownership of "pDataSource".  +The structure will be freed after a NuFlush call completes successfully or all +changes are aborted.  Until NuFlush or NuAbort completes, it is vital that +you don't free the underlying resource.  That is, don't close the FILE*, delete the file, or +free the buffer that the data source references.  If you don't want to keep +track of the resources used by FP and Buffer sources, you can specify "fcloseFunc" +or "freeFunc" functions to have them closed automatically.  See the explanation of +NuDataSource for details.

+

NuError NuDelete(NuArchive* pArchive)

+

Bulk delete.  This tries to delete every record in the archive, invoking +the SelectionFilter callback if one has been specified.

+

You cannot delete a record that is newly-added, has been modified, has +already been deleted, or has had threads added, deleted, or updated.  Such +records will be skipped over, so your selection filter simply won't see them.

+

Because deletion is a deferred write operation, none of the records will +actually be deleted until NuFlush is called.  If NuDelete was successful in +its attempt to delete every record, and no new records were added, the NuFlush +call will mark the archive as being brand new (this differs from v1.0, which +failed with kNuErrAllDeleted).  As a result, if you close the empty archive +without adding anything to it, the archive file will be removed.

+

NuError NuDeleteRecord(NuArchive* pArchive, NuRecordIdx recordIdx)

+

Delete a single record, specified by record index.

+

You cannot delete a record that is newly-added, has been modified, has +already been deleted, or has had threads added, deleted, or updated.

+

The record will be removed when NuFlush is called.

+

NuError NuDeleteThread(NuArchive* pArchive, NuThreadIdx threadIdx)

+

Delete a single thread, specified by thread index.  If you delete all of +the threads in a record, and don't add any new ones, the record will be removed.

+

You cannot delete a thread that is newly-added, deleted, or has been updated.

+

The thread will not be removed until NuFlush is +called.

+
+

General Interfaces

+

Archive Operations

+

NuError NuClose(NuArchive* pArchive)

+

Closes the +archive.  If the archive was opened read-write, any pending changes will be +flushed first.  If the flush attempt fails, NuClose will leave the archive open and return +with an error.

+

When the archive is closed, the temp file associated with a read/write +archive will be closed and removed.

+

All data structures associated with the archive are freed.  Attempting +to use "pArchive" further results in an error (or worse).

+

NuError NuAbort(NuArchive* pArchive)

+

Abort all pending changes.  NufxLib will throw out every pending +modification request, returning to the state it was in following the most recent +Open or Flush.

+

This does not close or manipulate any files, except for those pointed to by +data sources with "fcloseFunc" set.  For the most part it simply updates internal +data structures.

+

It's perfectly safe to call this if there are no pending changes.  The +call just returns without doing anything.

+

NuError NuGetMasterHeader(NuArchive* pArchive, const +NuMasterHeader** ppMasterHeader)

+

Get a pointer to the NuFX MasterHeader block.  One useful item here is +the number of records in the archive.

+

IMPORTANT: do not retain the pointer after calling NuFlush or NuAbort.

+

NuError NuGetExtraData(NuArchive* pArchive, void** ppData)
+NuError NuSetExtraData(NuArchive* pArchive, void* pData)

+

Store an arbitrary void* pointer in the NuArchive structure.  This can +be useful for accessing application data within a callback without resorting to +global variables.

+

NuError NuGetValue(NuArchive* pArchive, NuValueID ident, +NuValue* pValue)
+NuError NuSetValue(NuArchive* pArchive, NuValueID ident, NuValue +value)

+

Manipulate one of NufxLib's configurable values.  See the tables +for details.

+

NuError NuGetAttr(NuArchive* pArchive, NuAttrID ident, NuAttr* +pAttr)

+

Get an archive attribute, such as whether it's wrapped in a Binary II +header.  See the tables for details.

+

NuError NuDebugDumpArchive(NuArchive* pArchive)

+

Print debugging information to stdout.  The output contains a rather +verbose description of the archive.  This call is only functional if the +library was built with debugging enabled.  If the library was built without +assertions or debug messages, this call returns an error.

+

 

+

Data Sources

+

Sources and sinks provide a way for the application to add from and extract +to something other than a named file on disk.  There are three kinds of +sources and sinks:

+
    +
  1. File objects are named files on disk.  They are accessed by + filename.
    2. +
  2. FP objects are FILE pointers.  Pass in a pointer to any file + at any offset.
    3. +
  3. Buffer objects are pointers to memory.
    4. +
+

NuDataSource objects are used in conjunction with deferred write calls.  +They specify a location from which data is read.  All DataSource creation calls +take the following arguments:

+
    +
  • threadFormat - specifies the format of the source data.  + Usually this is kNuThreadFormatUncompressed.  If you are moving threads + directly from one archive to another, you may not want to expand and recompress them, + so you would indicate that the data source is already compressed.
    • +
  • otherLen - set to zero except for special situations.  For pre-sized threads, such as comments and filenames, + this defines the size of the full buffer.  For pre-compressed data, this + holds the threadEOF (i.e. size of the data when uncompressed).
    • +
  • ppDataSource - a pointer to the newly-allocated NuDataSource is + placed here if the call succeeds.
    • +
+

The remaining arguments are detailed next.

+

NuError NuCreateDataSourceForFile(NuThreadFormat threadFormat, unsigned long otherLen, const char* pathname, short isFromRsrcFork, +NuDataSource** ppDataSource)

+

Create a data source from a file on disk.  Because all write operations +are deferred, the file will not actually be opened until NuFlush is +called.  This means that if the file is unreadable or doesn't exist, the +data source create call will succeed, but the eventual NuFlush call will fail.

+

The entire contents of the file will be used.  The file is opened when +needed and closed when processing completes.

+

"pathname" is the name of the file to open.  If you use the +same pathname with more than one data source, each data source will open and +close the file.

+

"isFromRsrcFork" determines whether the data fork or resource fork +should be opened.  This only has meaning on systems like Mac OS and GS/OS, where the +"open" call determines which fork is opened.  For other systems, +always set it to "false".

+

NuError NuCreateDataSourceForFP(NuThreadFormat threadFormat, unsigned long otherLen, FILE* fp, long offset, long length, +NuCallback fcloseFunc, NuDataSource** ppDataSource)

+

Create a data source from a FILE*.  The FILE* must be seekable, i.e. you +can't use a stream like stdin.  Because all write operations are deferred, +any problems with the stream, such as an early EOF, will not be detected until +the NuFlush call is made.

+

"fp" is the stream to use.  It will be seeked immediately +before use, so it is permissible to use the same fp in more than one data +source.  If you are developing for a system that differentiates between +fopen(filename, "r") and fopen(filename, "rb"), use the +latter or you may get "unexpected EOF" failures.

+

"offset" is the starting offset in the file.  The file will be +seeked to this point right before it is used.

+

"length" is the number of bytes to use.

+

The "fcloseFunc" parameter points to a function that calls fclose() +on its argument.  It's bad practice (especially in the Win32 DLL world) to +allocate in the app and free in the library, so this provides a way to let the +library choose when to close the file, but let the application manage its own +heap.  If this argument is nil, the FILE* will not be closed when +processing on this data source completes.

+

IMPORTANT: if you use the same FILE* in more than one data source, do not +provide an fcloseFunc for any of them.  Deferred write operations are not +guaranteed to happen in any particular order, so if you set fcloseFunc the library +may close the file when it is still needed.

+

NuError NuCreateDataSourceForBuffer(NuThreadFormat threadFormat, unsigned long otherLen, const unsigned char* buffer, long offset, +long length, NuCallback freeFunc, NuDataSource** ppDataSource)

+

Create a data source from a memory buffer.  Invalid memory references +will not be detected until NuFlush is called.

+

"buffer" is a pointer to the memory you want to use.  It is +okay for "buffer" to be nil so long as "offset" and +"length" are zero.  This may be useful when creating an empty +comment thread.

+

"offset" is the offset from "buffer" at which the data +starts.

+

"length" is the number of bytes to use.

+

The "freeFunc" parameter points to a function that calls +"free", "delete", or "delete[]" on its argument.  +There's no way for nufxlib to know exactly how the memory was allocated (malloc/new/new[]/custom), +so the application needs to supply a function to clean it up.  If this +argument is nil, the buffer will not be freed when processing on this data +source completes..  (Side note: the "offset" parameter exists so that +you can use part of a buffer and then let the library free the whole thing afterward.)

+

IMPORTANT: if you use the same memory buffer in more than one data source, do +not provide a freeFunc for any of them.  Deferred write operations are not +guaranteed to happen in any particular order.

+

NuError NuFreeDataSource(NuDataSource* pDataSource)

+

Free a data source.  You should only do this if the data source was not +used in a successful deferred write call.

+

If "fcloseFunc" or "freeFunc" is set in the data source, the appropriate action will +be taken.  (NufxLib may actually make copies of DataSource objects with +ref-counting, so freeing your object may not cause an immediate fclose or free.)

+

void NuDataSourceSetRawCrc(NuDataSource* pDataSource)

+

When the data source contains already-compressed data, there's no way +for NufxLib to compute the CRC of the uncompressed data without expanding +it.  Version 3 records require a data CRC in the thread header.  This provides a way for the application to specify what value should +be in the "thThreadCrc" field.

+ +

 

+ +

Data Sinks

+

NuDataSink calls are used with the thread extraction function.  They +allow the application to specify where data is to be written to.  All +DataSink +creation calls take the following arguments:

+
    +
  • doExpand - should compressed data be expanded?  You + always want to set this to "true", unless you're moving compressed + data directly from one archive to another.
    • +
  • convertEOL - determines whether the automatic EOL conversion + routines are active when expanding the thread.  Possible values are + kNuConvertOff (never convert), kNuConvertOn (always convert), and + kNuConvertAuto (convert threads that look like ASCII text).  This value + overrides the kNuValueConvertExtractedEOL setting.  The EOL for + the current system can be set with the kNuValueEOL value.  (This is + ignored if "doExpand" is set to "false".)
    • +
  • ppDataSink - a pointer to a newly-allocated NuDataSink is placed + here
    • +
+

The remaining arguments are detailed next.

+

NuError NuCreateDataSinkForFile(short doExpand, NuValue convertEOL, +const char* pathname, unsigned char fssep, NuDataSink** ppDataSink)

+

Create a data sink for a named file on disk.  The file will be opened, +written to, and then closed.

+

Because of a peculiarity in NufxLib design, the OutputPathnameFilter callback +will be invoked during the extraction if one has been installed.  Since +your application supplied the filename, it most likely won't want to change it, +but this can still be useful in the case where the file exists and needs to be +renamed.  (This might even be useful, e.g. if your application insists on using +the record's filename directly when creating a data sink.)

+

"pathname" is the full pathname of the file to write to.

+

"fssep" is the filesystem separator used in the pathname.  +This is necessary so NufxLib can build any missing directory components.

+

Using the same pathname in more than one data sink will likely yield +disappointing results, as subsequent extractions will overwite the earlier ones.

+

NuError NuCreateDataSinkForFP(short doExpand, NuValue convertEOL, +FILE* fp, NuDataSink** ppDataSink)

+

Create a data sink from a FILE*.  The stream must be writeable, and must +be seeked to the desired offset before the extract call is made.

+

"fp" is the stream to use.

+

Using the same FILE* in more than one data sink isn't necessary: you can just +re-use the same data sink.  The stream is never +seeked, so subsequent extractions will append to the earlier ones.

+

NuError NuCreateDataSinkForBuffer(short doExpand, NuValue convertEOL, +unsigned char* buffer, unsigned long bufLen, NuDataSink** ppDataSink)

+

Use a memory buffer as a data sink.

+

"buffer" is a pointer to the memory buffer.

+

"bufLen" is the maximum amount of data that the memory buffer can +hold.

+

You can re-use a buffer data sink on multiple extractions.  The pointer +will be advanced, and bufLen decreased.  Exceeding the size of the buffer +causes the extraction to fail with a buffer overrun error.  (Thus, you can +extract more than one thread into the same buffer, but you can't extract one +thread into multiple buffers.)

+

NuError NuFreeDataSink(NuDataSink* pDataSink)

+

Free a NuDataSink.

+

NuError NuDataSinkGetOutCount(NuDataSink* pDataSink, unsigned long* +pOutCount)

+

Get the number of bytes that have been written to a data sink.  The +result will be placed into "pOutCount".  This can come in handy +if you've extracted a number of things into a memory buffer and aren't sure +exactly how much is in there (perhaps because of EOL conversions).

+

 

+

Callback Setters

+

These functions allow you to set callbacks on a per-archive basis.

+

Most NufxLib calls are illegal in a callback function (NufxLib is not +reentrant for a single NuArchive).  The only calls you are allowed to make +are NuGetExtraData, NuSetExtraData, NuGetValue, NuSetValue, and NuGetAttr.

+

The application must not keep copies of pointers passed to a callback.  +If you want to keep the information from (say) a NuRecord*, you will need to copy the +contents of the struct to local storage.

+

If something has a "const" pointer, don't write to it.  The +results of doing so are unpredictable (but most likely bad).

+

All callbacks are of type NuCallback, which is defined as:

+ + NuResult (*NuCallback)(NuArchive* pArchive, void* args); + +

The "set" functions return the previous callback, all of which +default to NULL.  If the "pArchive" argument is invalid, the +calls will fail and return kNuInvalidCallback.

+

  +

NuError NuSetSelectionFilter(NuArchive* pArchive, NuCallback +filterFunc)

+ +

The selection filter callback is used to select records and threads +during bulk operations. The argument passed into the callback is a "const +NuSelectionProposal*": + +

typedef struct NuSelectionProposal {
+	const NuRecord*     pRecord;
+	const NuThread*     pThread;
+} NuSelectionProposal;
+
+ +

These are pointers to the NuFX record and thread that we are about to +act upon. During an extract operation, "pThread" will point at the thread +we are about to extract. During a delete operation, "pThread" will point +at the first thread in the record we are about to delete. +

Valid return values from a selection filter: +

    +
  • kNuOK - accept this item. +
  • kNuSkip - skip this item. +
  • kNuAbort - abort current operation. +
+

If no selection filter is specified, then all records will be selected. +

  +

NuError NuSetOutputPathnameFilter(NuArchive* pArchive, NuCallback +filterFunc)

+ +

When extracting files, this callback allows you to change the name of +the file that will be opened on disk. It will be called once for every +thread we extract. The argument to the callback is a "NuPathnameProposal*": + +

typedef struct NuPathnameProposal {
+	const char*         pathname;
+	char                filenameSeparator;
+	const NuRecord*     pRecord;
+	const NuThread*     pThread;
+
+	const char*         newPathname;
+	unsigned char       newFilenameSeparator;
+	NuDataSink*         newDataSink;
+} NuPathnameProposal;
+
+ +

The fields are: +

    +
  • pathname - full pathname we're proposing to use. +
  • filenameSeparator - the character used to separate pathname +components, e.g. '/', '\', or ':'. If the separator isn't recognized +by the operating system, the application will need to change it. +
  • pRecord - pointer to the NuFX record being extracted. +
  • pThread - pointer to the NuFX thread being extracted. +
  • newPathname - if you want to change the pathname, set this to +a non-NULL value. +
  • newFilenameSeparator - if you want to change the filename +separator char, set this to a nonzero value. +
  • newDataSink - to extract the file to something other than a +filename, create a data sink and add the pointer here. +
+ +

If a record contains a data fork and a resource fork, +your filter will be called twice with the same pathname. (You can +examine pThread to see what kind of fork is being extracted.) If the +OS requires that extended files be initially created as such, then the +file will always be created as "extended" if the record indicates that +a resource fork is present. +

This mechanism can be used to implement a "rename file being extracted" +feature. If an error handler is defined, and it returns kNuRename when +NufxLib tries to overwrite an existing file, then the +pathname filter will be invoked again. + +

Valid return values from the output pathname filter: +

    +
  • kNuOK - accept this item. +
  • kNuSkip - skip this item. +
  • kNuAbort - abort the current operation. +
+ +

If no OutputPathnameFilter is set, the files will be opened with the +names that appear in the archive. + +

  +

NuError NuSetProgressUpdater(NuArchive* pArchive, NuCallback +updateFunc)

+ +

During add, extract, and test operations, NufxLib will send progress +update messages via the ProgressUpdater callback. The argument to the +callback is a "const NuProgressData*": + +

typedef struct NuProgressData {
+	NuOperation         operation;
+	NuProgressState     state;
+	short               percentComplete;
+
+	const char*         origPathname;
+	const char*         pathname;
+	const char*         filename;
+	const NuRecord*     pRecord;
+
+	unsigned long       uncompressedLength;
+	unsigned long       uncompressedProgress;
+
+	struct {
+		NuThreadFormat  threadFormat;
+	} compress;
+
+	struct {
+		unsigned long       totalCompressedLength;
+		unsigned long       totalUncompressedLength;
+
+		const NuThread*     pThread;
+		NuValue             convertEOL;
+	} expand;
+} NuProgressData;
+
+ +
    +
  • operation - the general class of operation are we performing. +
  • state - what state are we in. +
  • percentComplete - how far along are we, from 0 to 100. +
  • origPathmame - original pathname. When compressing, this is + the pathname for the file on disk; when expanding, this is the filename + as it appears in the archive. +
  • pathname - the full pathname after the pathname filter (if + any) has modified it. +
  • filename - "pathname" with everything up to the last fssep + removed. +
  • pRecord - pointer to the record we're compressing to or + expanding from. +
  • uncompressedLength - size of uncompressed data. +
  • uncompressedProgresss - #of uncompressed bytes we've read + or written so far. +
  • compress.threadFormat - type of compressing being applied. +
  • expand.totalCompressedLength - sum of compressed lengths of + all threads in this record. +
  • expand.totalUncompressedLength - sum of uncompressed lengths + of all threads in this record. +
  • expand.pThread - pointer to thread we're extracting. +
  • expand.convertEOL - EOL conversion mode used on this thread. + If it's "kConvertAuto", the first progress update after some data has + been processed will have either kConvertOn or kConvertOff. +
+ +

The possible values for a NuOperation value are: +

    +
  • kNuOpUnknown - unknown operation (the application should + never see this). +
  • kNuOpAdd - files are being added to an archive. +
  • kNuOpExtract - records or threads are being extracted from + an archive. +
  • kNuOpTest - records are being tested. +
  • kNuOpDelete - records or threads are being deleted. +
  • kNuOpContents - the archive contents are being examined. +
+ +

Deleting files and listing contents don't cause the progress update +callback to be called, so you'll never see "kNuOpDelete" or "kNuOpContents" +in a progress handler. + +The possible values for a NuProgressState value are: +

    +
  • kNuProgressPreparing - not started yet +
  • kNuProgressOpening - opening files +
  • kNuProgressCompressing - compressing data +
  • kNuProgressStoring - storing data (without compression) +
  • kNuProgressExpanding - expanding data +
  • kNuProgressCopying - copying uncompressed data (in or out) +
  • kNuProgressDone - all done, success +
  • kNuProgressSkipped - all done, we skipped this one +
  • kNuProgressAborted - all done, user cancelled the operation +
  • kNuProgressFailed - all done, failure +
+ +

Some values (say, kNuProgressCompressing) are only appropriate for +certain operations (kNuOpAdd). +

Valid return values from a progress updater are: +

    +
  • kNuOK - all is well. +
  • kNuAbort - abort operation. +
+

If no ProgressUpdater is defined, no progress update information will +be sent. + + + +

 

+

NuError NuSetErrorHandler(NuArchive* pArchive, NuCallback errorFunc)

+ +

The ErrorHandler callback deals with all exceptional conditions that +arise. The callback may define hard-coded policy or query the user for +directions. The argument to the callback is a "const NuErrorStatus*": + +

typedef struct NuErrorStatus {
+	NuOperation         operation;
+	NuError             err;
+	int                 sysErr;
+	const char*         message;
+	const NuRecord*     pRecord;
+	const char*         pathname;
+	const char*         origPathname;
+	char                filenameSeparator;
+
+	char                canAbort;
+	char                canRetry;
+	char                canIgnore;
+	char                canSkip;
+	char                canRename;
+	char                canOverwrite;
+} NuErrorStatus;
+
+ +
    +
  • operation - the general class of operation we are performing. +
  • err - NufxLib error code. +
  • sysErr - system error code, if applicable. +
  • message - (optional) message to user. +
  • pRecord - (optional) relevant record. +
  • pathname - (optional) name of file or record involved. +
  • origPathname - (optional) when adding, name of original file (from NuFileDetails). +
  • filenameSeparator - (optional) fssep in use at time. +
  • canAbort - callback may return kNuAbort. +
  • canRetry - callback may return kNuRetry. +
  • canIgnore - callback may return kNuIgnore. +
  • canSkip - callback may return kNuSkip. +
  • canRename - callback may return kNuRename. +
  • canOverwrite - callback may return kNuOverwrite. +
+ +

Some situations that may arise: +

+
operation == kNuOpExtract +
+
err == kNuErrFileExists +
We're extracting a file to the same pathname as an existing + file, and our overwrite policy is set to "maybe". +
err == kNuErrNotNewer +
We're extracting a file to the same pathname as an existing + file, and we're only allowed to overwrite older files. +
err == kNuErrDuplicateNotFound +
We're extracting a file in the "must overwrite" mode, but + the file doesn't exist. +
+
operation == kNuOpAdd +
+
err == kNuErrRecordExists +
We're adding a file whose "storage name" matches a file + already in the archive, and our overwrite policy is set to "maybe". +
err == kNuErrFileNotFound +
We tried to add a file, but when we went to open it the + file didn't exist. +
+
operation == kNuOpTest +
+
err == kNuErrBadMHCRC +
The master header CRC was bad. +
err == kNuErrBadRHCRC +
A record header CRC was bad. +
err == kNuErrBadThreadCRC +
A thread header CRC was bad. +
err == kNuErrBadDataCRC +
A thread in the data (e.g. LZW/1 CRC) was bad. +
+
+

The valid return values are defined by the NuErrorStatus structure. + +

If no ErrorHandler is defined, an appropriate default action (usually +kNuAbort) is taken. + + +

 

+

NuError NuSetErrorMessageHandler(NuArchive* pArchive, NuCallback messageFunc)
+NuError NuSetGlobalErrorMessageHandler(NuCallback messageFunc)

+

Specify a callback to receive text error messages.  These are typically +an error message followed by an explanation of the error code that the library +is about to return.  The callback takes an argument of type "const +NuErrorMessage*", which is defined as:

+ +
typedef struct NuErrorMessage {
+	const char*         message;
+	NuError             err;
+	short               isDebug;
+
+	const char*         file;
+	int                 line;
+	const char*         function;
+} NuErrorMessage;
+
+ +
    +
  • message - the message to display. +
  • err - relevant error code, if any. +
  • isDebug - set for debug-only messages. You should only get + these messages if the library is built with DEBUG_MSGS enabled. +
  • file - (if library built with DEBUG_MSGS) source file where + message originated. +
  • line - (if library built with DEBUG_MSGS) source line number. +
  • function - (if library built with DEBUG_MSGS, and compiler + supports __FUNCTION__) name of function where message originated. +
+

The return value is ignored. +

Some error messages aren't associated with an archive, generally because they +occur when an archive is being opened.  Since there's no way to associate +them with a single archive, the handler must be global to the entire +library.  The second form of this call allows you to specify where global +error messages should be sent.  The arguments to the callback are +identical, but "pArchive" will be nil.

+

If no callback is specified, the messages are sent to stderr.  If your +application doesn't have a stderr (perhaps it's a GUI application), be sure to +set both the ErrorMessag and GlobalErrorMessage handlers.

+
+

Helper Functions

+ +

Some of these are macros, some are functions.  None require that an +archive be open.

+ +

NuError NuGetVersion(long* pMajorVersion, long* pMinorVersion, const +char** ppBuildDate, const char** ppBuildFlags)

+

Get some information about NufxLib's version.  This sets the major and +minor version numbers, as well as setting strings with the build date and some +build flags.

+

Any or all of the arguments may be NULL, for values you aren't interested in.

+

The format of "ppBuildDate" is not defined [though it probably +should be].

+

The format of "ppBuildFlags" is a string of compiler flags +separated by white space (spaces or tabs).  It is expected to represent an +"interesting subset" of the flags sent to the compiler, such as the +level of optimization used.

+

const char* NuStrError(NuError err)

+

Return a pointer to a string describing a NufxLib error.  NufxLib errors +are "err" values less than zero.  "err" values greater +than zero are system errors that can be processed with strerror() or perror(), +and an "err" value of zero indicates success.

+ +

NuError NuTestFeature(NuFeature feature)

+ +

Test for support of an optional feature.  See the tables +for a list.  Returns kNuErrNone on success, kNuErrUnsupFeature if the +feature is known but not supported, or kNuErrUnknownFeature if the feature is +not recognized at all (probably because the version of NufxLib you're linked +with is older than what you compiled against).

+ +

unsigned long NuMakeThreadID(unsigned short class, unsigned short +kind)

+ +

Construct a NuThreadID, given a thread class and thread kind.

+ +

unsigned long NuGetThreadID(const NuThread* pThread)

+ +

Construct a NuThreadID, using the thread class and thread kind defined in a +NuThread.

+ +

unsigned short NuThreadIDGetClass(NuThreadID threadID)

+ +

Pull the thread class out of a NuThreadID.

+ +

unsigned short NuThreadIDGetKind(NuThreadID threadID)

+ +

Pull the thread kind out of a NuThreadID.

+ +

char NuGetSepFromSysInfo(unsigned short sysInfo)

+ +

Pull the filename separator character out of the file_sys_info word.

+ +

unsigned short NuSetSepInSysInfo(unsigned short sysInfo, char newSep)

+ +

Put the filename separator character into a file_sys_info word.

+ +

unsigned long NuRecordGetNumThreads(const NuRecord* pRecord)

+ +

Return the number of threads in a record.

+ +

const NuThread* NuGetThread(const NuRecord* pRecord, int idx)

+ +

Get the idx-th thread from pRecord.  If idx is less than zero or past +the end of the thread array, nil is returned.

+ +

void NuRecordCopyAttr(NuRecordAttr* pRecordAttr, const NuRecord* +pRecord)

+ +

Copy data from "pRecord" into "pRecordAttr".  Only +the fields that exist in a NuRecordAttr are copied.  This can be useful in +conjunction with the SetRecordAttr call.

+ +

NuError NuRecordCopyThreads(const NuRecord* pRecord, NuThread** +ppThreads)

+ +

Copy the thread array out of a record.  This is useful if you want to +keep your own copy of a thread array.

+ +

short NuIsPresizedThread(NuThreadID threadID);

+ +

Returns "true" if the threadID is considered pre-sized by +NufxLib.  Right now, only filenames and comments are given this treatment.

+ +

 

+ +
+

Tables

+ +

Configurable Values (NuValue)

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
kNuValueIgnoreCRCBoolean (false).  Don't verify header or data CRCs.  + This can provide a minor speed improvement, but allows certain kinds of + errors to go undetected.
kNuValueDataCompressionEnum (kNuCompressLZW2).  Threads that can be compressed + (i.e. data-class threads) will be compressed with the specified + compression.  Possible values are: +
    +
  • kNuCompressNone (no compression)
    • +
  • kNuCompressSQ (SQueeze)
    • +
  • kNuCompressLZW1 (ShrinkIt's dynamic LZW/1)
    • +
  • kNuCompressLZW2 (ShrinkIt's dynamic LZW/2)
    • +
  • kNuCompressLZC12 (12-bit LZW from "compress")
    • +
  • kNuCompressLZC16 (16-bit LZW from "compress")
    • +
  • kNuCompressDeflate [requires zlib]
    • +
+
kNuValueDiscardWrapperBoolean (false).  If changes are made to the archive + that cause a new copy to be reconstructed in the temp file, then when this + is set to "true" any BXY, BSE, or SEA wrapper will be stripped + off.  This also causes any "junk" at the start of the file + to be removed.
kNuValueEOLEnum (system-dependent).  End-of-line marker + appropriate for the current system.  If EOL conversion is enabled, + extracted files will be converted to this EOL value.  Valid values + are: +
    +
  • kNuEOLCR (carriage return, for ProDOS, GS/OS, Mac OS)
    • +
  • kNuEOLLF (line feed, for UNIX)
    • +
  • kNuEOLCRLF (CR+LF, for MS-DOS and Win32)
    • +
+
kNuValueConvertExtractedEOLEnum (kNuConvertOff).  This determines whether + "bulk" extractions do EOL conversions.  Possible values: +
    +
  • kNuConvertOff (don't try to convert)
    • +
  • kNuConvertOn (always convert)
    • +
  • kNuConvertAuto (convert if the input appears to be a text file)
    • +
+
kNuValueOnlyUpdateOlderBoolean (false).  If set, only overwrite existing + records and files if the item being added or extracted is newer than the + one being replaced.  Useful for an "update" or + "freshen" option.  The date used for comparison is the + modification date.
kNuValueAllowDuplicatesBoolean (false).  If set to "true", duplicate + records are allowed in the archive.  If "false", the + collision will be handled according to the kNuValueHandleExisting + setting.  Filename comparisons are case-insensitive.
kNuValueHandleExistingEnum (kNuMaybeOverwrite).  This determines how + duplicate filename collisions are handled.  Valid values: +
    +
  • kNuMaybeOverwrite (the ErrorHandler callback is invoked)
    • +
  • kNuNeverOverwrite (the file being added or extracted is skipped)
    • +
  • kNuAlwaysOverwrite (the existing file or record is deleted)
    • +
  • kNuMustOverwrite (fails if the file or record doesn't exist, useful + for a "freshen" option)
    • +
+

The case sensitivity when extracting is determined by the underlying + filesystem. +

kNuValueModifyOrigBoolean (false, unless the archive was just created by + NufxLib).  If this is "true", then an effort will be made + to handle all updates in the original archive, rather than reconstructing + the entire archive in a temp file.  Updates to pre-sized threads, + changes to record attributes, and additions of new files can all be made + to the original archive.  There is some risk of corruption if the + flush fails, so use this with caution.
kNuValueMimicSHKBoolean (false).  If set, attempt to mimic the behavior + of ShrinkIt as closely as possible.  See the ShrinkIt + Compatibility Mode section.
kNuValueMaskDatalessBoolean (false).  If set to "true", records + without data threads have "fake" threads created for them, so + that they appear as they would had they been created correctly.
kNuValueStripHighASCIIBoolean (false).  If set to "true", files + filled with high-ASCII characters will be stripped if and only if an EOL + conversion is performed. 
kNuValueJunkSkipMaxInteger (1024).  If the archive file doesn't start with + a recognized sequence, NufxLib will assume that some junk has been added + to the start of the file and will scan forward at most this many bytes in + an attempt to locate the real archive start.
kNuValueIgnoreLZW2LenBoolean (false).  If set to "true", the + length value embedded in LZW2 compressed chunks is ignored.  This is + useful for archives created with a specific broken application.  + (This is deprecated -- use HandleBadMac instead.)
kNuValueHandleBadMacBoolean (false).  Recognize and handle "bad + Mac" archives, which have a bad value ('?') for the filename + separator character, and write an LZW/2 length value in big-endian order.
+

 

+

Archive Attributes (NuAttr)

+ + + + + + + + + + + + + + + + + +
kNuAttrArchiveTypeReturns one of the following: +
    +
  • kNuArchiveNuFX (NuFX archive, e.g. ".SHK" or + ".SDK")
    • +
  • kNuArchiveBinaryII (Binary II, e.g. ".BNY" or ".BQY") + [not supported]
    • +
  • kNuArchiveNuFXInBNY (NuFX inside Binary II, e.g. ".BXY")
    • +
  • kNuArchiveNuFXSelfEx (self-extracting GSHK archive, e.g. + ".SEA")
    • +
  • kNuArchiveNuFXSelfExInBNY (self-ex inside Binary II, e.g. ".BSE")
    • +
+
kNuAttrNumRecordsReturns the number of records in the archive.  This + value does not reflect unflushed changes. +
kNuAttrHeaderOffsetReturns the offset of the NuFX header from the start of the + file.  This will be nonzero for archives with a Binary II or + self-extracting wrapper. +
kNuAttrJunkOffsetReturns the amount of junk found at the start of the + file.  A nonzero value here indicates that junk was found. +
+

 

+

Feature Tests (NuFeature)

+ + + + + + + + + + + + + + + + + + + + + +
kNuFeatureCompressSQTest for support of SQueeze compression +
kNuFeatureCompressLZWTest for support of ShrinkIt LZW/1 and LZW/2 compression +
kNuFeatureCompressLZCTest for support of 12- and 16-bit LZC +
kNuFeatureCompressDeflateTest for support of zlib "deflate" +
kNuFeatureCompressBzip2Test for support of libbz2 "bzip2" +
+

 

+ +
+

Additional Commentary

+

Replacing Existing Records and Files

+

+When using NuAddFile or NuAddRecord, there are three flags that affect +what happens when an existing record has the same name: +

+
AllowDuplicates (default false)
+
if set, adding a record with the same name as an + existing record is allowed. +
OnlyUpdateOlder (default false)
+
if set, we refuse to replace an existing record + unless its modification date is older. +
HandleExisting (default "maybe overwrite")
+
can be set to "maybe overwrite" (prompts the user), + "never overwrite" (returns an error), "always overwrite" (overwrites + the existing record), and "must overwrite" (causes an error if the + record *doesn't* exist). The "maybe overwrite" value is treated as + "never overwrite" if an error-handling callback isn't defined. +
+It's important to understand how these interact with each other, and what +they mean to both existing records and newly-added (pre-flush) records. +Two of them also have an effect when extracting files. +

+The AllowDuplicates flag determines whether or not we +think duplicate records are at all interesting. If an application sets it to +true, the floodgates are opened, and the two other flags are ignored. +

+The OnlyUpdateOlder flag is considered next. If it's +set to true, and an existing, identically named file in the archive appears +to be the same age or newer than the file being added, the record creation +attempt fails with an error (kNuErrNotNewer). +

+The HandleExisting flag comes into play if +we get past the first two. If a matching entry is found in the archive, +NufxLib either deletes it and allows the add, prompts the user for +instructions, or rejects it with an error. NuAddFile and NuAddRecord will +return with kNuErrRecordExists if they can't replace an existing record. +If "must overwrite" is set, and a matching record does not exist, then +kNuErrDuplicateNotFound is returned. +

+Both AddFile and AddRecord check for duplicates among existing and newly +added files. You aren't allowed to delete items that were just added, so +HandleExisting flag is ignored for files you have marked for addition +but haven't yet flushed. +

+AddFile has an additional behavior that takes precedence over all of +the flags: it will try to match up the individual forks of a file. +If it finds a file in the newly-added list with the +same name and a compatible data thread, the new file will be added to +the existing record. (A "compatible" data thread is the other half of a +forked file, e.g. the application added the data fork, and is now adding +the resource fork from the same file.) If the record was found but is not +compatible, the AllowDuplicates behavior is used to decide if another +"new" record with the same name should be added, or if an error should +be returned. +

+If this sort of treatment is undesirable, i.e. you want a data fork +and a resource fork with the same filename to be stored as two separate +records, then you should call AddRecord and AddThread. AddFile is meant +as a convenience for common operations. +

+It is possible for NuAddFile and NuAddRecord to partially complete. If +a record exists and is deleted, but the call later fails for some other +reason, the record will still be deleted.

Searching for existing records can +take time on a large archive.  Disabling AllowDuplicates will allow NufxLib +to avoid having to search through the lists of records to find matches. +

 

+When extracting files from an archive, the "OnlyUpdateOlder" and +"HandleExisting" flags are applied to the files on disk. This is done +much like the above. +

+To implement NuLib2's "update" feature, "OnlyUpdateOlder" needs to be +set to true. To implement the "freshen" feature, "OnlyUpdateOlder" +is set to true and "HandleExisting" is set to "must overwrite". +

+When extractions are done in bulk, the kNuErrDuplicateNotFound and +kNuErrNotNewer errors are passed to the application's error handler +function. The error handler is expected to return kNuSkip after perhaps +updating the progress status message, but is allowed to abort or require +NufxLib to overwrite the file anyway. If no error handler is defined, +the file is skipped silently. + +

 

+

ShrinkIt Compatibility Mode

+

One of the goals was to be as compatible with ShrinkIt as possible.  +ShrinkIt and GS/ShrinkIt occasionally do some strange things, so some of the +compatible behaviors are only activated when the "mimic ShrinkIt" flag +is set.

+

These behaviors are:

+
    +
  • When new files are added to an archive, the first record has a blank + comment added.  The size in GSHK is configurable, but defaults to 200 + bytes, so that's what NufxLib uses.  (If you've already provided a comment + for the first new record, an additional comment won't be added.)
    • +
  • All threads compressed with LZW have an extra zero byte added to the end.
    • +
  • When deciding if a 4K chunk was compressed with LZW/2, ShrinkIt compares + the size of the post-RLE output to the size of the LZW output.  It + doesn't take into account a two-byte header (the compressed data size) that is + prepended to the block.  This causes it on rare occasions to choose a + compressed block that is a byte or two larger than the pre-LZW data.
    • +
  • Files smaller than 512 bytes are never compressed.
    • +
  • All archives with SEA wrappers have an extra byte added to the end.  + (For BSE archives, where a Binary II wrapper is placed around the SEA + wrapper, I have chosen *not* to emulate GSHK's behavior in one particular + circumstance: if the SEAed archive is an exact multiple of 128 bytes, GSHK + still adds the extra byte, making the BSE file an odd size.  Binary II + files should always be a multiple of 128 bytes.)
    • +
+

Some GS/ShrinkIt behaviors are not fully emulated:

+
    +
  • The test on LZW/2 chunk size is inconsistent: sometimes the + comparison is "LZW < post-RLE", sometimes the comparison is + "LZW <= post-RLE".  My guess is that ShrinkIt is failing + to clear the carry flag before a comparison.  Whatever the case, it's + impossible to imitate correctly, so NufxLib always uses "LZW < post-RLE".
    • +
  • When adding a zero-byte data fork, + no data thread is created.  The record contains nothing but a filename + thread (and perhaps a comment thread).  This appears to be a bug.  + Similar behavior occurs for empty forked files.
    • +
+

Regarding the last item: a quick test with a handful of empty +files showed that GS/ShrinkIt v1.1 failed to extract the empty files it had just +archived.  P8 ShrinkIt v3.4 gets really confused on such archives, and insists that the +first entry is a zero-byte disk archive, while the other empty files are +actually four bytes long.  When asked to extract the files, it does +nothing.  When adding empty files, P8 ShrinkIt v3.4 does the correct thing, and creates an empty +data thread.

+

The default NufxLib behavior is to work around the bug.  When extracting files with a filename but no data or control threads, a zero-byte data file will be +created.  (In NufxLib v1.0 the default was to ignore such entries unless +the "mimic" flag was enabled.  This was changed in v1.1 to be +enabled at all times.  As of v1.2, an empty resource fork is also created +if the record's storage type indicates it's an extended file.)  If the +"MaskDataless" flag is enabled, fake data threads are created, and +applications won't even know there's a problem in the archive.

+

In general, with "mimic ShrinkIt" mode enabled, it should be +possible to extract files from a GS/ShrinkIt archive and re-add them to a new +archive, with little perceptible difference between the two.  Of course, it's +up to the application to ensure that all threads (including comments) are +retained, file dates aren't altered, and so on.  The only situations where +NufxLib cannot produce identical results are bugs (e.g. zero-length +data files always require more space) and option lists (which NufxLib does not currently support).

+

The bottom line: it is perfectly normal for NufxLib archives to be a few +bytes smaller than GS/ShrinkIt archives, even when "mimic ShrinkIt" +mode is enabled.  (An example: my 20MB boot partition compressed to about +14MB.  With "mimic" mode off, the file was 13K smaller, or about +0.1%.)

+ +

 

+

Compression Formats

+

+Of the various compression formats that NufxLib supports, only LZW/1 and LZW/2 are +widely supported. The latest versions of ShrinkIt and II Unshrink +try to unpack SQ compression but fail. Archives that use SQ, LZC12, +and LZC16 can only be unpacked by GS/ShrinkIt, NuLib, and NufxLib v1.1 and later. +

The "deflate" and "bzip2" algorithms are not supported by anything other than NufxLib +v1.1+ and CiderPress. +They are intended to be used with archives that will never be +unpacked on an Apple II. Disk images compressed with these algorithms are especially useful with +emulators that use NufxLib.

+ +

Some tests with deflate and bzip2 showed that, surprisingly, deflate is +nearly always better than bzip2 for Apple II files.  This is because +deflate appears to do slightly better on machine code and small (< 32K) text +files.  Since most Apple II files and disk images fall into these +categories, there is little advantage to using bzip2.  Because deflate uses +less memory and is faster, and libbz2 isn't nearly as ubiquitous as libz, I've +chosen to disable bzip2 by default.

+ +

You can use the NuFeatureTest call to test for the presence of any of the +compression algorithms.  This makes it possible to build a library or DLL +without LZW in it.

+ +
+

Porting

+

NufxLib v1.0 was developed under Solaris 2.5 and Red Hat Linux 6.0, and was ported +to Win32 shortly before the alpha release.  Porting to other UNIX-like +platforms should be straightforward, with most differences contained in the +"autoconf" configuration system.  For example, the BeOS/PPC port +was largely a matter of getting the compiler settings right.

+

Mac OS and GS/OS have the ability to store file types and resource forks +natively.  Support for this is not currently part of NufxLib.  A +data-fork-only port, akin to what is used on UNIX and Win32, should be +straightforward though.  (In fact, Mac OS X "just worked".)

+

Nothing in NufxLib explicitly requires 32-bit ints, but I'm sure it has been +assumed somewhere.

+
+

Design Notes

+

The decision to pass FILE* structures instead of file descriptors was +somewhat arbitrary.  The library uses buffered I/O internally, so it was +convenient to have them passed in, rather than having an fd passed in and rely +on the existence of an fdopen() call.  On the other hand, if an application +is built with a different version of the stdio library (in which the structure of +a FILE* are different), linking with NufxLib might not work.  Given that NufxLib +is distributed as BSD-licensed source code, I don't see this as +being a major problem, since you can always rebuild NufxLib with the altered +stdio lib.  (This caused me some grief under Windows, because the non-debug +multithreaded DLL version of libc apparently does something wonky with FILE* and +fwrite.  If the Win32 DLL and Win32 app aren't linked against the same libc, +fwrite() will crash.  Other versions of libc, e.g. debug multithreaded and +debug single-threaded, interact just fine with each other.)  My conclusion +after fighting with Win32 is that it would have been better to pass file +descriptors or a "NuFILE*" with read/write/seek operations that reside +wholly within the NufxLib library.

+

The decision to pass data sources and sinks around as structures rather than as +function pointers was born of a desire to reduce complexity.  Setting up a data +source or sink requires making a function call with a lot of arguments, but once +that's done you can forget all about it -- the code will happily close your +files and free your memory when you're done with it.  A functional +interface would require passing in read, write, close, and seek functions, which +gives the application more flexibility but essentially requires the application +to implement its own version of the data source and sink structures.  Since +NufxLib is intended for manipulating archives, not compressing streams of data, +the added flexibility did not justify the cost.  (I'm becoming less certain +of that as time goes by.  If I had it all to do again, I probably would use +the functional interface for all file accesses.)

+

It might have been useful to allow read/write/seek hooks for the archive +itself.  The current architecture prevents you from processing an archive +that has been loaded into memory, unless you have memory-based FILE* streams in your +libc.  This became annoying during the development of CiderPress, because I +wanted to handle archives within a wrapper, such as ".shk.gz".

+

Returning a pointer to an allocated NuArchive structure worked pretty well +until I wanted to set a parameter that affected the way open works (the +junk-skipping feature).  Creating the structure in a separate call before +the "open" would +have been better.

+

It might, for portability reasons, have been better to require a "create +file" callback.  This would offload most of the system-dependent stuff +in FileIO.c onto the application.  I chose not to do this because I felt it +moved too much of the work out of NufxLib and into the hands of the +developer.  Requiring the application to deal with the "OnlyUpdateOlder" +and "HandleExisting" flags seemed excessive, and if there really are +wide variants in the way files are created and modification dates are tested, +then we might as well solve the problem once and for all in the library instead +of requiring every application to solve it for themselves.  (I could, of +course, provide sample code for several different platforms, but sample code +tends to suffer from bit rot.)

+

There is no real support for GS/OS option lists.  The only place you'd +ever want to add these is on a IIgs, and I find it unlikely that NufxLib will +edge out GS/ShrinkIt as the preferred archiver.  (Besides, I question their +value even on a IIgs.)  NufxLib will very carefully preserve them when modifying +a record, but there's no way to add, delete, or modify them directly.

+ +

The use of RecordIdx and ThreadIdx, rather than record filename and thread +offset number, was chosen for a number of reasons.  The most important was +that they are unambiguous.  Consider that two records may have the same +filename, that one record may have two filenames, and that a record may have no +filename at all, and the need for RecordIdx becomes painfully clear.  I +could've avoided ThreadIdx, by using a combination of RecordIdx and thread +number, but after a few additions and deletions there is a clear advantage to +having a unique identifier for every thread.  Besides, it allowed me to +design calls like NuExtractThread so that they only had to take one identifier +as an argument, reducing the amount of stuff an application needs to keep track +of, as well as the amount of error checking that has to be done in the library.

+ +

Allowing threads to be copied without expanding and recompressing them is +neat, but if I 'd know how cluttered the interface would become I probably +wouldn't have supported the feature.  The NuDataSource calls are confusing +enough as it is with the pre-sized thread stuff.

+ +

The "bulk" NuAdd interface can be cumbersome.  When extracting +you can skip the bulk approach and handle filename conflicts yourself, but when +adding you don't have a good alternative if you're adding lots of files.  +You would have to follow every NuAdd with a NuFlush, which has some performance +problems because (unless you configure the safety options off) NuFlush will +write all data to the temp file and rotate it.

+ +

I chose not to implement EOL conversions when adding files.  It's too +painful to do this in the library.  It would be easier for the application +to write an EOL-converted file into a temp file, then use the file add call on +that.  (The "storage name" is set independently from the source +file name, so there's no problem with temp file names showing up in the +archive.)  One approach that could be used within NufxLib would be to +"pre-flight" the file by doing an EOL conversion pass to determine the +final file length, then feed that length into the compression functions.  +The "NuStraw" interface would do the conversion transparently to the +compression routines.

+ +

The compression functions might have been better written with a zlib-like +API.  This would have made it easier to extract the code and use it in +other projects.  The only disadvantage of doing so is that it adds a little +extra buffer copying overhead.

+ +

Some attention should have been paid to internationalization.

+ +

Some perhaps useful calls that weren't implemented:

+ +
    +
  • NuWhatIsThisFile(char* filename).  Returns an enum indicating what we + think is in the file, i.e. SHK, BXY, BSE, BQY, BNY, SEA, ZIP, ZOO, ARC, TAR, + .gz, .Z, etc.  Useful for an application that supports more than one + kind of archive.  Not sure if it belongs in NufxLib.  (Right now + the "open archive" calls will return a "hey, this is Binary + II" error code; this mechanism could be extended.)
    • +
  • NuRearrangeArchive(...args...).  There is no way to sort an archive + short of reconstructing it manually.  An interface for swapping the + positions of two records could be provided.  This would allow sorting + an archive.
    • +
  • NuExtractThreadChunk(len).  NufxLib allows you to extract an entire + thread to a buffer or a file, but doesn't let you extract small pieces and + handle them yourself.  It's an all-or-nothing grab.  Since we're + dealing with Apple II files, which tend to be relatively small, this + shouldn't cause problems.  It can be inconvenient though to have to + grab large chunks of memory to apply a filter before writing data.  It + might make sense to define a DataSink with a callback function (instead of a + buffer or FP) that is expected to write the data.
    • +
+
+

History

+

A brief history of NufxLib releases.  See "ChangeLog.txt" in +the sources for more detail.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
VersionDateComments
v0.0mid-1998Work begins
v0.12000/01/17First version viewed by test volunteers.
v0.52000/02/09Alpha test version.
v0.62000/03/05Beta test version.
v1.02000/05/18Initial release.
v1.0.12000/05/22Added workaround for badly-formed archives.
v1.12002/10/20Many new features, notably support for several compression + formats.
v2.02003/03/18Support for Win32 DLL features.
v2.0.12003/10/16Added junk-skipping and a workaround for bad option lists; + Mac OS X stuff from sheppy.
v2.0.22004/03/10Handle zeroed MasterEOF, and correctly set permissions on + "locked" files.
v2.0.32004/10/11Fixed some obscure bugs that CiderPress was hitting.
v2.1.02005/09/17Added kNuValueIgnoreLZW2Len.
v2.1.12006/02/18Fix two minor bugs.
v2.2.02007/02/19Switched to BSD license.  Identify "bad Mac" + archives automatically.
v2.2.22014/10/30Updated build files, especially for Win32.
+  +
+

Acknowledgements

+

I'd like to thank Eric Shepherd for participating in some ping-pong e-mail +sessions while I tried to get autoconf, BeOS, and some crufty versions of "make" +figured out for v1.0.

+
+

This document is Copyright © 2000-2007 by Andy +McFadden.  All Rights Reserved.

+

The latest version can be found on the NuLib web site at +http://www.nulib.com/.

+
+ + diff --git a/nulib2-manual.htm b/nulib2-manual.htm new file mode 100644 index 0000000..70df151 --- /dev/null +++ b/nulib2-manual.htm @@ -0,0 +1,699 @@ + + + + + + +NuLib2 Manual + + + +
+ +

NuLib2 Manual
+[ Home ] [ NuLib Downloads ] [ NuLib Library ] [ NuLib2 Manual ] [ NufxLib API ] [ Bugs & Features ]

+
+ +
+ +
 
+
NuLib2 v2.1.1 Manual - By Andy McFadden - Last revised 2006/02/18
+ +

Table of Contents

+ + + +

 

+ +

Introduction

+ +

NuLib2 is a command-line archive utility, along the lines of "PKZIP".  +It allows you to perform many common operations on NuFX archives, such as those +created by the Apple II "ShrinkIt" utility, as well as Binary II +archives.  Files with extensions +"SHK", "SDK", "BXY", "BSE", "SEA", +"BNY", and "BQY" are handled.

+ +

You can add, delete, extract, test, and list files in a NuFX archive.  +Compressed disk images can be extracted to files, and vice-versa, making it a +handy utility to have when using an Apple II emulator.

+ +

NuLib2 is the successor to NuLib, which did many of the same things.  +NuLib2 is not meant to set a new standard or fight for supremacy on the PC +desktop; rather, it is intended to help people working with Apple IIs and Apple II +emulators.  All compression algorithms specified in the NuFX specification +are fully supported.

+ +

NuLib2 has a number of features not found in NuLib:

+ +
    +
  • Supports filetype preservation (on systems with "long" filenames).
    • +
  • Supports resource forks.
    • +
  • Archive integrity test actually does something useful.
    • +
  • ShrinkIt-style table of contents listing formats filenames correctly.
    • +
  • Support for adding and viewing comments.
    • +
  • Allows streaming input (i.e. read from stdin) for most + "read-only" operations.
    • +
  • Can operate on large archives while using very little memory (with + streaming mode).
    • +
  • Transparently handles .BXY, .SEA, and .BSE wrappers.
    • +
  • Automatically recognizes and handles Binary II archives.
    • +
  • EOL conversions (e.g. CRLF to CR) are more automatic.
    • +
  • Deals with some Y2K issues.
    • +
  • Comes with an auto-configuration script for UNIX-like platforms.
    • +
  • Creates "version 3" records, and uses LZW/2.  Most of + GS/ShrinkIt's quirks are emulated, so the archives created by NuLib2 are + nearly identical to those created by GSHK.
    • +
  • Optionally supports gzip-style "deflate" compression and bzip2 + BWT (as an + extension to the NuFX standard).
    • +
  • Source code is cleaner and command-line options are simpler.
    • +
+ +

All features of the original NuLib are supported, except for a couple of +really obscure ones.  NuLib2 is built on top of NufxLib, a NuFX archive manipulation library.

+ +

 

+ +

Command Overview

+ +

Commands are specified like this:

+ +
+

nulib2 -command[modifiers] archive [filename-list]

+ +
+ +

There are seven commands: add files, list files (two variations), extract +files (two variations), delete files, and verify archive integrity.  There +are ten modifiers, discussed in later sections.

+ +

If you run nulib2 without any arguments, you will be presented with an +identification banner and a command summary.  The identification banner describes the current version of the NuLib2 application, and the version of +NufxLib upon which it was built.  The latest version of NuLib2 can always +be found at http://www.nulib.com/.  If +you want a more detailed command summary, use:

+ +
+

nulib2 -h

+ +
+ +

If you wish to specify modifiers, you may lump them together with the +command, like this:

+ +
+ nulib2 -xel archive.shk + +
+

or specify each independently, like this:

+ +
+ nulib2 -x -e -l archive.shk + +
+

If you want to give your archive a name that starts with a hyphen, you will +have to specify it as a full or partial path, e.g. "./-archive.shk".  +Otherwise, NuLib2 will think you are trying to specify more modifiers.  One +exception to this is that you can specify stdin as the archive for list and +extract operations:

+ +
+

nulib2 -v - < archive.shk

+ +
+ +

Some commands require a list of filenames.  These must be listed +after the archive name, e.g.:

+ +
+ +

nulib2 -x archive.shk foo ack:splat bar

+ +
+ +

The names given will be compared with records in a case-insensitive fashion, so +asking NuLib2 to extract "foo" or "FOO" would match on "foo", +"FOO", and "fOo".

+ +

The commands are explained next.

+ +

 

+ +

Listing Archive Contents (-v, -t)

+ +

The contents of NuFX archives are listed in a format similar to what ProDOS 8 +ShrinkIt 2.x displays.  If you use the command:

+ +
+ +nulib2 v archive.shk + +
+

you will see output similar to this:

+
 archive.shk     Created:22-Aug-90 15:33   Mod:22-Aug-90 15:33     Recs:    4
+
+ Name                        Type Auxtyp Archived         Fmat Size Un-Length
+-----------------------------------------------------------------------------
+ BUG.REPORTS                 TXT  $0003  22-Aug-90 15:33  lz2   32%      3089
+ FINDER.DATA                 FND  $0000  22-Aug-90 15:33  unc  100%       458
++HELLO                       BAS  $0801  22-Aug-90 15:33  lz2   86%       605
+ GIF.SYS16                   S16+ $0000  22-Aug-90 15:33  lz2   31%     39165
+-----------------------------------------------------------------------------
+ Uncomp: 4152  Comp: 1988  %of orig: 47%
+ +For each file, the display includes the filename, ProDOS file type, ProDOS +"aux" type, the date and time when the file was added to the archive, +the compression format, a compression ratio percentage, and the length of the +file before it was compressed.  The first line has some information about +the archive, and the last line has a summary of file sizes and compression +performance. +

A "+" in the leftmost column, in front of the filename, indicates +that the file is "locked".  NuLib2 considers a file to be locked +when it has the ProDOS write, rename, and delete permissions disabled, but still +has read permission enabled.  All other files are considered to be +unlocked.

+

A "+" is shown next to the file type if the file is +"extended", meaning it has a resource fork.  A "-" next +to the file type indicates that it's an empty file stored improperly due to a +bug in GSHK.

+

The listing for a disk image is similar:

+
 prodisk.shk     Created:21-Apr-90 15:11   Mod:21-Apr-90 15:11     Recs:    1
+
+ Name                        Type Auxtyp Archived         Fmat Size Un-Length
+-----------------------------------------------------------------------------
+ TEST                        Disk 140k   21-Apr-90 15:11  lz1   37%    143360
+-----------------------------------------------------------------------------
+ Uncomp: 143360  Comp: 53051  %of orig: 37%
+The file type is displayed as "Disk", the "aux" type is the +size of the disk image in KBytes, and -- if the image is for a ProDOS disk -- +the filename is the disk volume name. +

The "Fmat" column will be one of "lz1", "lz2", +or "unc", indicating ShrinkIt LZW/1, LZW/2, or uncompressed.  +Files added to archives by NuLib2 always use LZW/2 or, if compression fails, are +stored uncompressed.  LZW/1 is used by P8 ShrinkIt and the original NuLib.

+

The filename field will show the full name, including all subdirectories +leading up to the file.  If there isn't enough room to display the entire +filename, it will be truncated on the left, replaced with two dots.  The +following example has five files from the "gno:user:man:man2" +directory, one of which was too long to fit in the display:

+
 Name                        Type Auxtyp Archived         Fmat Size Un-Length
+-----------------------------------------------------------------------------
+ gno:usr:man:man2:sigblock.2 GWP  $8010  01-Dec-97 00:07  lz2   58%      2667
+ gno:usr:man:man2:signal.2   GWP  $8010  01-Dec-97 00:07  lz2   56%      7036
+ gno:usr:man:man2:sigpause.2 GWP  $8010  01-Dec-97 00:07  lz2   61%      2358
+ ..usr:man:man2:sigsetmask.2 GWP  $8010  01-Dec-97 00:07  lz2   56%      3046
+ gno:usr:man:man2:wait.2     GWP  $8010  01-Dec-97 00:07  lz2   51%      6577
+-----------------------------------------------------------------------------
+You can also list the contents of an archive like this: +
+

nulib2 t archive.shk

+
+

The 't' command generates a simple list of filenames:

+ +
BUG.REPORTS
+FINDER.DATA
+HELLO
+GIF.SYS16
+The filenames are never truncated or embellished, which makes this command +useful when you're searching for a specific file. +

 

+ +

Creating Archives and Adding Files (-a)

+ +

You can add files to a new or existing archive with the "-a" +command.  If the archive you specify does not exist, a new one will be +created.

+ +

All files will be added to the end of the archive.  If the name of the +file being added matches the name of a file already present in the archive, you +will be allowed to replace the existing file or skip adding the new file:

+ +
+

Replace BASIC.System? [y]es, [n]o, [A]ll, [N]one:

+ +
+

If you respond with "y", the existing file will be deleted +when the archive is updated.  If you say "n", the new file will +be skipped.  Entering "A" or "N" will cause NuLib2 to +automatically enter "y" or "n" respectively on any future +conflicts.  (You may also hit 'q' here to abort the operation and quit.)

+ +

Files in subdirectories will be added with whatever path separator is +appropriate for the current system.  On a UNIX-like system that would be +'/', while under Win32 it's '\'.  GS/OS follows the Mac OS convention and uses ':'.

+ +

There are a number of modifiers that can be used with this command.

+ +
+
-u
+
Update files.  If a matching file is found in the archive, it will + only be replaced if the file on disk is newer.  Files that don't + already exist in the archive will be added.  The + "Replace...?" dialog will not appear.
+
-f
+
Freshen files.  If a matching file is found in the archive, it will + only be replaced if the file on disk is newer.  Files that don't + already exist in the archive will not be added.  The + "Replace...?" dialog will not appear.
+
-r
+
Recursively descend into subdirectories.  The standard behavior for + NuLib2 is to ignore directories.  When this flag is set, it will add + all of the files in a specified subdirectory.  (Note: empty directories + will not be added.)
+
-j
+
Junk paths.  If you add a file called "foo/bar/myfile", + when this flag is set it will be stored simply as "myfile".
+
-0
+
Don't compress.  Files will be stored without compression.  + (Note that's "dash zero", not the letter 'O'.)
+
-z
+
Use "deflate" compression, equivalent to "gzip + -9".  Both NufxLib and NuLib2 must be built with zlib + support, or the flag will be disabled.  Note that "deflated" + files can only be opened by applications based on NufxLib, so don't use this + for files that will be opened on an Apple II.  If you specify "-zz", + compression equivalent to "bzip2 -8" will be used instead.  + (Support for bzip2 compression is disabled by default, so it may not be + available.)
+
-c
+
Add one-line comments.  NuLib2 will prompt you for a comment on every + new file.  Hitting return ends the comment.  (Hint: on most UNIX + systems, you can use Ctrl-V Ctrl-M to insert a carriage return.  NuLib2 + automatically converts carriage returns in comments to whatever is + appropriate for the current system, so you can "sneak" multi-line + comments in this way.)  Maximum length is 200 characters.
+
-k
+
Store files as disk images.  All files will be processed as ProDOS-ordered + disk images.  See Disk Images for more information.
+
-e
+
Preserve ProDOS file types.  Files with preservation information will + be added with their file type and aux type intact, and resource forks and + disk images will be processed correctly.  For this to work, the files + must have been extracted with the "-e" flag set.  If you + specify "-ee", NuLib2 will attempt to guess the file types of + common files by their extensions (e.g. "file.txt" would be stored + as type $04 (TXT)).  See the ProDOS + Attribute Preservation document for more information.
+
+ +

While adding files, NuLib2 displays the name of the file as it will appear in +the archive, along with a completion percentage:

+
DONE compressing  BASIC.System
+DONE storing      BOOT.GSOS
+DONE compressing  BOOTU3
+DONE storing      COMM/FINDER.DATA
+DONE compressing  COMM/ProTERM/PT3.SYSTEM
+DONE compressing  COMM/ProTERM/PT3
+DONE compressing  COMM/ProTERM/PT3.GLOBAL
+DONE storing      COMM/ProTERM/PT3.WELCOME
+ 15% compressing  COMM/ProTERM/PT3.CODE0
+

Files stored without compression will say "stored", while files +compressed with LZW/2 will say "compressing".  NuLib2 tries to +mimic GS/ShrinkIt as closely as possible, so files shorter than 512 bytes are +never compressed, and files that don't get smaller are stored uncompressed.

+ +

 

+ +

Extracting Files (-x, -p)

+ +

Files can be extracted with the "-x" and "-p" +commands.  You may specify a list of files to extract, or specify none and +extract all files.

+ +

A simple example that extracts all files from an archive:

+ +
+ +

nulib2 x archive.shk

+ +
+ +

If you wanted to extract a file called "fubar" and a file called +"ack:splat", you would use:

+ +
+ +

nulib2 x archive.shk fubar ack:splat

+ +
+ +

The directory hierarchy is always preserved unless "-j" is +set.  In the above example, the files would be extracted as "fubar" +in the current directory and "splat" in a subdirectory called "ack" +(assuming that the file was archived on a system where ':' separates directory +names).

+ +

Using the "-r" flag (described below), you could extract all of the +files in the "ack" subdirectory, like this:

+ +
+ +

nulib2 xr archive.shk ack:

+ +
+ +

The "-r" flag does a prefix string match, meaning it just compares +the first part of the name in the archive against the name you specify.  So +if you had entered:

+ +
+

nulib2 xr archive.shk ack

+ +
+

above, it would have extracted "acknowledge" and "acksent" +as well as "ack:foo" and "ack:bar".  Specifying the +filesystem separation character (usually '/' or ':') allows you to grab just the +directory you want.

+ +

If you try to extract a file with the same name as an existing file, you will +be prompted for instructions:

+ +
+ +

Replace BASIC.System? [y]es, [n]o, [A]ll, [N]one, [r]ename:

+ +
+ +

You can choose (from left to right) to overwrite the existing file, skip the +extraction of this file, overwrite all existing files, never overwrite an +existing file, or rename the current file.  If you elect to rename, you +will be prompted for a new name for the file.  (You may also hit 'q' here +to quit immediately.)

+ +

The modifiers usable with the "-x" command are:

+ +
+
-u
+
Update files.  If a matching file is found on disk, it will + only be replaced if the file in the archive is newer.  Files that don't + already exist on disk will be extracted.  The + "Replace...?" dialog will not appear.
+
-f
+
Freshen files.  If a matching file is found on disk, it will + only be replaced if the file in the archive is newer.  Files that don't + already exist on disk will not be extracted.  The + "Replace...?" dialog will not appear.
+
-r
+
Recursively describe the set of files to extract.  This allows you to + extract an entire subdirectory tree from the archive.  (Perhaps someday + NuLib2 will support extraction by regular expression, and this modifier can + go away.)
+
-j
+
Junk paths.  The pathname is stripped off of the file.  If you + extract "foo" and "ack:splat", you will end up with a + file called "foo" and a file called "splat" in the + current directory.
+
-l
+
Convert EOL.  When set, NuLib2 tries to identify which files in the + archive are text files and which have binary data.  It then converts + the text file end-of-line markers to whatever is appropriate on the current system.  If you + specify "-ll", NuLib2 will convert all files except disk + images.  High-ASCII files, such as DOS text files and Merlin 8 source + files, are automatically stripped.
+
-c
+
Display comments while extracting.  If a comment was stored with the + record, it will be displayed on the screen.  End-of-line markers (e.g. + CR, LF, or CRLF) used in the comment are automatically converted.
+
-s
+
Stomp existing files.  When set, NuLib2 will overwrite existing files + without prompting you for confirmation.
+
-e
+
Preserve ProDOS file types.  Extracted files will have the ProDOS + file type and aux type appended to the name (e.g. "foo.txt#040000").  + Resource forks and disk images will be labeled.  If you specify "-ee", + NuLib2 will append the file's extension to the end of the file, so that + systems like Win32 that rely on filename extensions can recognize the file + type (e.g. "foo.txt#040000.txt").  See the ProDOS + Attribute Preservation document for more information. 
+
+ +

While extracting files, NuLib2 displays progress information:

+ +
DONE expanding   BASIC.System
+DONE extracting  BOOT.GSOS
+DONE expanding   BOOTU3
+DONE extracting  COMM/FINDER.DATA
+DONE expanding   COMM/ProTERM/PT3.SYSTEM
+DONE expanding   COMM/ProTERM/PT3
+DONE expanding   COMM/ProTERM/PT3.GLOBAL
+DONE extracting  COMM/ProTERM/PT3.WELCOME
+ 90% expanding   COMM/ProTERM/PT3.CODE0
+ +The filenames shown are the names as they are being written to disk. For example, +if you extracted files with "-e" (preserve types) and "-l" +(convert end-of-line character) set:  +
+

+nulib2 -xle archive.shk +

+
+

You would see something like:

+
DONE expanding   BASIC.System#ff2000
+DONE extracting  BOOT.GSOS#fc0801
+DONE expanding   BOOTU3#060800
+DONE extracting  COMM/FINDER.DATA#c90000
+DONE expanding   COMM/ProTERM/PT3.SYSTEM#ff2000
+DONE expanding   COMM/ProTERM/PT3#ff2000
+DONE expanding + COMM/ProTERM/PT3.GLOBAL#040000
+DONE extracting+ COMM/ProTERM/PT3.WELCOME#040000
+ 90% expanding   COMM/ProTERM/PT3.CODE0#060000
+The "+" sign indicates that an EOL conversion was performed on the +file. +

 

+ +

The "-p" command extracts the files to a pipe.  The contents +of the extracted files are written to stdout.  The normal status messages are suppressed.

+ +

Only the "-r" and "-l" modifiers can be used with +"-p".

+ +

This command is useful for examining the contents of individual files.  +For example, the command:

+ +
+ nulib2 -pl archive.shk document.txt | more + +
+

will dump the contents of "document.txt", with text automatically +converted, and pipe the results into "more".  If you specify more +than one file, they will be sent to the output one after the other.

+ +

 

+ +

Verifying Archive Integrity (-i)

+ +

If you want to verify that an archive is undamaged, use this command.  +NuLib2 does the same processing that it does when extracting files from the +archive, but no output is produced.  Every CRC (Cyclic Redundancy Check - a +checksum computed when the data was added to the archive) is tested.

+ +

The progress information shown when testing an archive looks like this:

+ +
DONE verifying  BASIC.System
+DONE verifying  BOOT.GSOS
+DONE verifying  BOOTU3
+DONE verifying  COMM/FINDER.DATA
+DONE verifying  COMM/ProTERM/PT3.SYSTEM
+DONE verifying  COMM/ProTERM/PT3
+DONE verifying  COMM/ProTERM/PT3.GLOBAL
+DONE verifying  COMM/ProTERM/PT3.WELCOME
+ 90% verifying  COMM/ProTERM/PT3.CODE0
+

If a problem is found, you will see a message like this one:

+ +
  Found a bad CRC in BASIC.System
+  Archive may be damaged, continue anyway?  [y]es, [n]o:
+ +

If you choose 'y', NuLib2 will pick up where it left off, and continue +processing the archive.  If you choose 'n', NuLib2 stops immediately.

+ +

If you don't specify a list of files:

+ +
+

nulib2 -i archive.shk

+ +
+

then NuLib2 will test every file in the archive.  If you use a list:

+ +
+

nulib2 -i archive.shk foo ack:splat

+ +
+

it will only test the files you told it to.

+ +

You may use the "-r" modifier when specifying files to test, in the +same fashion as when extracting files.

+ +

 

+ +

Deleting Files (-d)

+ +

This command deletes entire records from an archive.  A list of +filenames must be specified. If you manage to delete all of the files, +the archive itself will be removed. + +

An example of this command:

+ +
+ +nulib2 -d archive.shk foo ack:splat + +
+ +

A progress report will be displayed during processing, e.g.

+ +
Deleting foo
+Deleting bar
+Deleting ack/splat
+ +

The "-r" modifier may be used to select entire +subdirectories.  See the notes in the section on +extraction, above.

+ +

 

+ +

Disk Images

+ +

NuLib2 doesn't try to write floppy disk images onto a floppy the way ShrinkIt +does.  +Instead, it extracts disk images as ProDOS-ordered files, suitable for use with +your favorite Apple II emulator.

+ +

When adding disk images, NuLib2 assumes that the file is in ProDOS block +order.  If you want the archive to work correctly with ShrinkIt on an Apple +II, don't add DOS-ordered disk image files.

+ +

The NufxLib library comes with a sample program called "imgconv" that can +convert between 2IMG (".2mg") images and NuFX (".shk") +archives.  It is able to convert DOS-ordered .2mg files to ProDOS order +before adding them to the NuFX archive.

+ +

NuLib2 will only accept files that are a multiple of 512 bytes.  If you +insist on trying to add odd-sized files as disk images, NuLib2 will print a warning +and add it as an ordinary file.

+ +

 

+ +

Binary II Archives

+ +

NuLib2 supports Binary II (.BNY, .BQY) archives in a more or less transparent +fashion.  For all operations except adding and deleting files you can +specify a .BNY file in place of a .SHK and get more or less the same results.

+ +

A few modifier flags aren't currently supported for Binary II archives.  +You can't specify EOL conversion with "-l", freshen or update with +"-f" or "-u", or extract comments (there are no such things) +with "-c".  You can, however, specify filenames using +"-r" for partial matches, use "-e" and "-ee" to +extend filenames, and use "-j" to truncate paths.

+ +

The default behavior is to refuse to overwrite existing files.  NuLib2 +currently just stops if it encounters a file that already exists.  You can +use the "-s" flag to tell NuLib2 to overwrite any existing +files.  It will not, however, overwrite directories with files or files +with directories.

+ +

Files that were compressed with "Squeeze" compression (via BLU +v2.27 or SQ3) will be automatically un-squeezed.  If the filenames end in +".QQ", the extension will be stripped.  Programs like ShrinkIt +and BLU use the filename embedded within the SQ format as the output name when +extracting, but NuLib2 ignores that because it tends to leave you with a +filename you weren't expecting.

+ +

In some cases you will need to explicitly tell NuLib2 that a file is a Binary +II archive with the "-b" flag.  If you want to strip the Binary +II header off of a .BXY file, you will need to use "nulib2 -xb +file.BXY".  Otherwise, NuLib2 defaults to extracting from the +NuFX archive embedded in the BXY file.  You will also need to use the +"-b" modifier for a streaming archive (e.g. "nulib2 -xb - +< file.BQY"), because the stream can't be rewound after the test +for NuFX fails.

+ +

 

+ +

Miscellaneous

+ +

Files that were somehow archived without a filename will be referred to as "UNKNOWN".  Rumor has it some older versions of ShrinkIt omitted +the file name for DOS 3.3 disk images.

+ +

On Win32 and UNIX-like systems, the ProDOS access permissions are not preserved +precisely.  ProDOS has read, write, rename, and delete permission, as well +as "backup needed" and "invisible" flags.  If NuLib2 +believes the file is locked, it will disable write permission on the file.  +When adding files, "locked" files will have read and "backup +needed" enabled and the other flags disabled, while "unlocked" +flags will have all the flags except "invisible" enabled.  In +other words, the basic concepts of "locked" and "unlocked" +are preserved, but the full set of flags is not.

+

The NuFX specification forbids filenames that start with the filename +separator character, i.e. you can't put "/foo/bar" in an +archive.  If you specify a full pathname, the leading '/' will be dropped.

+

On Win32 and UNIX-like systems, a leading "./" in the filename is redundant +and will be stripped.  If ".." is used as a path component, the +pathname will be reduced to just the filename.

+

When making changes to an archive, a new archive is constructed in a +temporary file ("nulibtmpXXXXX") in the current directory.  When +everything completes successfully, the temp file is renamed over the original +archive.  The exception to this is that newly-created archives are written +in place.  If NuLib2 crashes or is killed with signal while modifying an archive, the temp file may +be left lying around.

+

Filenames with invalid characters will be converted to something palatable +for the current system (e.g. "/" is set to "_" on UNIX-like +systems).  If file type preservation is enabled, the character will be +preserved exactly (e.g. '/' becomes "%2f").

+

NuFX archives store three dates with every file: creation, modification, and when +it was archived.  On systems that don't have creation dates, the +modification date will be substituted.

+

There are certain filenames you can't use on a Windows "FAT" +filesystem, such as "AUX" and "PRN".  +Neither Win98 nor Linux's vfat driver will allow it.  Standard utilities +like WinZip fail with a mysterious error message.  As a workaround, the Win32 version of NuLib2 will consistently +prefix all MS-DOS device entries with '_', so "AUX" and "aux.foo.txt" +will be extracted +as "_AUX" and "_aux.foo.txt".  If filetype preservation is enabled, the name used +will be "%00AUX" (the %00 is removed when the file is added with -e, +thus preserving the original name).  This handling does not apply to +versions built for other systems, so attempting +to extract a file named "AUX" onto a FAT filesystem under Linux will cause NuLib2 to +fail.

+

No attempt is made to extract and preserve comments, even in +"preserve" mode.  This is probably a bug.

+

Archive files that start with junk -- such as a vestigal MacBinary header or +HTTP headers -- appear occasionally on FTP sites.  NuLib2 will search +through the first 1024 bytes of the file to find the actual archive start.

+

Silly benchmark of the day: creating a 14MB archive containing the contents +of my hard drive took about 40 minutes on an accelerated IIgs.  NuLib2 +accomplished the same feat in about six seconds on a 500MHz Pentium-III running +Linux.

+

 

+ +

Acknowledgments

+

NuLib2 would not have been possible without all the lessons learned from +NuLib.  Andy Nicholas, Kent Dickey, Frank Petroski, Robert B. Hess, Bruce +Kahn, and Devin Reade contributed code to NuLib's development.

+

My original plan for preserving ProDOS file types was, in retrospect, mighty +screwed up.  My thanks to Bill North for setting me straight.

+Eric Shepherd spent a bunch of time messing around with early versions of NuLib2 +on BeOS, and helped me get all the configuration stuff in order.

Devin Reade +built it on several different platforms, and made a repository for binary +distributions.

+
+

This document is Copyright © 2000-2006 by Andy +McFadden.  All Rights Reserved.

+

The latest version can be found on the NuLib web site at +http://www.nulib.com/.

+
+ +