fixed #14: add check for nonstandard dos33 sector size and reset if found

* used when restoring segment parser from json

LICENSE

@@ -1,339 +1,373 @@
-		    GNU GENERAL PUBLIC LICENSE
-		       Version 2, June 1991
+Mozilla Public License Version 2.0
+==================================
 
- Copyright (C) 1989, 1991 Free Software Foundation, Inc.,
- 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
- Everyone is permitted to copy and distribute verbatim copies
- of this license document, but changing it is not allowed.
+1. Definitions
+--------------
 
-			    Preamble
+1.1. "Contributor"
+    means each individual or legal entity that creates, contributes to
+    the creation of, or owns Covered Software.
 
-  The licenses for most software are designed to take away your
-freedom to share and change it.  By contrast, the GNU General Public
-License is intended to guarantee your freedom to share and change free
-software--to make sure the software is free for all its users.  This
-General Public License applies to most of the Free Software
-Foundation's software and to any other program whose authors commit to
-using it.  (Some other Free Software Foundation software is covered by
-the GNU Lesser General Public License instead.)  You can apply it to
-your programs, too.
+1.2. "Contributor Version"
+    means the combination of the Contributions of others (if any) used
+    by a Contributor and that particular Contributor's Contribution.
 
-  When we speak of free software, we are referring to freedom, not
-price.  Our General Public Licenses are designed to make sure that you
-have the freedom to distribute copies of free software (and charge for
-this service if you wish), that you receive source code or can get it
-if you want it, that you can change the software or use pieces of it
-in new free programs; and that you know you can do these things.
+1.3. "Contribution"
+    means Covered Software of a particular Contributor.
 
-  To protect your rights, we need to make restrictions that forbid
-anyone to deny you these rights or to ask you to surrender the rights.
-These restrictions translate to certain responsibilities for you if you
-distribute copies of the software, or if you modify it.
+1.4. "Covered Software"
+    means Source Code Form to which the initial Contributor has attached
+    the notice in Exhibit A, the Executable Form of such Source Code
+    Form, and Modifications of such Source Code Form, in each case
+    including portions thereof.
 
-  For example, if you distribute copies of such a program, whether
-gratis or for a fee, you must give the recipients all the rights that
-you have.  You must make sure that they, too, receive or can get the
-source code.  And you must show them these terms so they know their
-rights.
+1.5. "Incompatible With Secondary Licenses"
+    means
 
-  We protect your rights with two steps: (1) copyright the software, and
-(2) offer you this license which gives you legal permission to copy,
-distribute and/or modify the software.
+    (a) that the initial Contributor has attached the notice described
+        in Exhibit B to the Covered Software; or
 
-  Also, for each author's protection and ours, we want to make certain
-that everyone understands that there is no warranty for this free
-software.  If the software is modified by someone else and passed on, we
-want its recipients to know that what they have is not the original, so
-that any problems introduced by others will not reflect on the original
-authors' reputations.
+    (b) that the Covered Software was made available under the terms of
+        version 1.1 or earlier of the License, but not also under the
+        terms of a Secondary License.
 
-  Finally, any free program is threatened constantly by software
-patents.  We wish to avoid the danger that redistributors of a free
-program will individually obtain patent licenses, in effect making the
-program proprietary.  To prevent this, we have made it clear that any
-patent must be licensed for everyone's free use or not licensed at all.
+1.6. "Executable Form"
+    means any form of the work other than Source Code Form.
 
-  The precise terms and conditions for copying, distribution and
-modification follow.
+1.7. "Larger Work"
+    means a work that combines Covered Software with other material, in 
+    a separate file or files, that is not Covered Software.
 
-		    GNU GENERAL PUBLIC LICENSE
-   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
+1.8. "License"
+    means this document.
 
-  0. This License applies to any program or other work which contains
-a notice placed by the copyright holder saying it may be distributed
-under the terms of this General Public License.  The "Program", below,
-refers to any such program or work, and a "work based on the Program"
-means either the Program or any derivative work under copyright law:
-that is to say, a work containing the Program or a portion of it,
-either verbatim or with modifications and/or translated into another
-language.  (Hereinafter, translation is included without limitation in
-the term "modification".)  Each licensee is addressed as "you".
+1.9. "Licensable"
+    means having the right to grant, to the maximum extent possible,
+    whether at the time of the initial grant or subsequently, any and
+    all of the rights conveyed by this License.
 
-Activities other than copying, distribution and modification are not
-covered by this License; they are outside its scope.  The act of
-running the Program is not restricted, and the output from the Program
-is covered only if its contents constitute a work based on the
-Program (independent of having been made by running the Program).
-Whether that is true depends on what the Program does.
+1.10. "Modifications"
+    means any of the following:
 
-  1. You may copy and distribute verbatim copies of the Program's
-source code as you receive it, in any medium, provided that you
-conspicuously and appropriately publish on each copy an appropriate
-copyright notice and disclaimer of warranty; keep intact all the
-notices that refer to this License and to the absence of any warranty;
-and give any other recipients of the Program a copy of this License
-along with the Program.
+    (a) any file in Source Code Form that results from an addition to,
+        deletion from, or modification of the contents of Covered
+        Software; or
 
-You may charge a fee for the physical act of transferring a copy, and
-you may at your option offer warranty protection in exchange for a fee.
+    (b) any new file in Source Code Form that contains any Covered
+        Software.
 
-  2. You may modify your copy or copies of the Program or any portion
-of it, thus forming a work based on the Program, and copy and
-distribute such modifications or work under the terms of Section 1
-above, provided that you also meet all of these conditions:
+1.11. "Patent Claims" of a Contributor
+    means any patent claim(s), including without limitation, method,
+    process, and apparatus claims, in any patent Licensable by such
+    Contributor that would be infringed, but for the grant of the
+    License, by the making, using, selling, offering for sale, having
+    made, import, or transfer of either its Contributions or its
+    Contributor Version.
 
-    a) You must cause the modified files to carry prominent notices
-    stating that you changed the files and the date of any change.
+1.12. "Secondary License"
+    means either the GNU General Public License, Version 2.0, the GNU
+    Lesser General Public License, Version 2.1, the GNU Affero General
+    Public License, Version 3.0, or any later versions of those
+    licenses.
 
-    b) You must cause any work that you distribute or publish, that in
-    whole or in part contains or is derived from the Program or any
-    part thereof, to be licensed as a whole at no charge to all third
-    parties under the terms of this License.
+1.13. "Source Code Form"
+    means the form of the work preferred for making modifications.
 
-    c) If the modified program normally reads commands interactively
-    when run, you must cause it, when started running for such
-    interactive use in the most ordinary way, to print or display an
-    announcement including an appropriate copyright notice and a
-    notice that there is no warranty (or else, saying that you provide
-    a warranty) and that users may redistribute the program under
-    these conditions, and telling the user how to view a copy of this
-    License.  (Exception: if the Program itself is interactive but
-    does not normally print such an announcement, your work based on
-    the Program is not required to print an announcement.)
+1.14. "You" (or "Your")
+    means an individual or a legal entity exercising rights under this
+    License. For legal entities, "You" includes any entity that
+    controls, is controlled by, or is under common control with You. For
+    purposes of this definition, "control" means (a) the power, direct
+    or indirect, to cause the direction or management of such entity,
+    whether by contract or otherwise, or (b) ownership of more than
+    fifty percent (50%) of the outstanding shares or beneficial
+    ownership of such entity.
 
-These requirements apply to the modified work as a whole.  If
-identifiable sections of that work are not derived from the Program,
-and can be reasonably considered independent and separate works in
-themselves, then this License, and its terms, do not apply to those
-sections when you distribute them as separate works.  But when you
-distribute the same sections as part of a whole which is a work based
-on the Program, the distribution of the whole must be on the terms of
-this License, whose permissions for other licensees extend to the
-entire whole, and thus to each and every part regardless of who wrote it.
+2. License Grants and Conditions
+--------------------------------
 
-Thus, it is not the intent of this section to claim rights or contest
-your rights to work written entirely by you; rather, the intent is to
-exercise the right to control the distribution of derivative or
-collective works based on the Program.
+2.1. Grants
 
-In addition, mere aggregation of another work not based on the Program
-with the Program (or with a work based on the Program) on a volume of
-a storage or distribution medium does not bring the other work under
-the scope of this License.
+Each Contributor hereby grants You a world-wide, royalty-free,
+non-exclusive license:
 
-  3. You may copy and distribute the Program (or a work based on it,
-under Section 2) in object code or executable form under the terms of
-Sections 1 and 2 above provided that you also do one of the following:
+(a) under intellectual property rights (other than patent or trademark)
+    Licensable by such Contributor to use, reproduce, make available,
+    modify, display, perform, distribute, and otherwise exploit its
+    Contributions, either on an unmodified basis, with Modifications, or
+    as part of a Larger Work; and
 
-    a) Accompany it with the complete corresponding machine-readable
-    source code, which must be distributed under the terms of Sections
-    1 and 2 above on a medium customarily used for software interchange; or,
+(b) under Patent Claims of such Contributor to make, use, sell, offer
+    for sale, have made, import, and otherwise transfer either its
+    Contributions or its Contributor Version.
 
-    b) Accompany it with a written offer, valid for at least three
-    years, to give any third party, for a charge no more than your
-    cost of physically performing source distribution, a complete
-    machine-readable copy of the corresponding source code, to be
-    distributed under the terms of Sections 1 and 2 above on a medium
-    customarily used for software interchange; or,
+2.2. Effective Date
 
-    c) Accompany it with the information you received as to the offer
-    to distribute corresponding source code.  (This alternative is
-    allowed only for noncommercial distribution and only if you
-    received the program in object code or executable form with such
-    an offer, in accord with Subsection b above.)
+The licenses granted in Section 2.1 with respect to any Contribution
+become effective for each Contribution on the date the Contributor first
+distributes such Contribution.
 
-The source code for a work means the preferred form of the work for
-making modifications to it.  For an executable work, complete source
-code means all the source code for all modules it contains, plus any
-associated interface definition files, plus the scripts used to
-control compilation and installation of the executable.  However, as a
-special exception, the source code distributed need not include
-anything that is normally distributed (in either source or binary
-form) with the major components (compiler, kernel, and so on) of the
-operating system on which the executable runs, unless that component
-itself accompanies the executable.
+2.3. Limitations on Grant Scope
 
-If distribution of executable or object code is made by offering
-access to copy from a designated place, then offering equivalent
-access to copy the source code from the same place counts as
-distribution of the source code, even though third parties are not
-compelled to copy the source along with the object code.
+The licenses granted in this Section 2 are the only rights granted under
+this License. No additional rights or licenses will be implied from the
+distribution or licensing of Covered Software under this License.
+Notwithstanding Section 2.1(b) above, no patent license is granted by a
+Contributor:
 
-  4. You may not copy, modify, sublicense, or distribute the Program
-except as expressly provided under this License.  Any attempt
-otherwise to copy, modify, sublicense or distribute the Program is
-void, and will automatically terminate your rights under this License.
-However, parties who have received copies, or rights, from you under
-this License will not have their licenses terminated so long as such
-parties remain in full compliance.
+(a) for any code that a Contributor has removed from Covered Software;
+    or
 
-  5. You are not required to accept this License, since you have not
-signed it.  However, nothing else grants you permission to modify or
-distribute the Program or its derivative works.  These actions are
-prohibited by law if you do not accept this License.  Therefore, by
-modifying or distributing the Program (or any work based on the
-Program), you indicate your acceptance of this License to do so, and
-all its terms and conditions for copying, distributing or modifying
-the Program or works based on it.
+(b) for infringements caused by: (i) Your and any other third party's
+    modifications of Covered Software, or (ii) the combination of its
+    Contributions with other software (except as part of its Contributor
+    Version); or
 
-  6. Each time you redistribute the Program (or any work based on the
-Program), the recipient automatically receives a license from the
-original licensor to copy, distribute or modify the Program subject to
-these terms and conditions.  You may not impose any further
-restrictions on the recipients' exercise of the rights granted herein.
-You are not responsible for enforcing compliance by third parties to
-this License.
+(c) under Patent Claims infringed by Covered Software in the absence of
+    its Contributions.
 
-  7. If, as a consequence of a court judgment or allegation of patent
-infringement or for any other reason (not limited to patent issues),
-conditions are imposed on you (whether by court order, agreement or
-otherwise) that contradict the conditions of this License, they do not
-excuse you from the conditions of this License.  If you cannot
-distribute so as to satisfy simultaneously your obligations under this
-License and any other pertinent obligations, then as a consequence you
-may not distribute the Program at all.  For example, if a patent
-license would not permit royalty-free redistribution of the Program by
-all those who receive copies directly or indirectly through you, then
-the only way you could satisfy both it and this License would be to
-refrain entirely from distribution of the Program.
+This License does not grant any rights in the trademarks, service marks,
+or logos of any Contributor (except as may be necessary to comply with
+the notice requirements in Section 3.4).
 
-If any portion of this section is held invalid or unenforceable under
-any particular circumstance, the balance of the section is intended to
-apply and the section as a whole is intended to apply in other
-circumstances.
+2.4. Subsequent Licenses
 
-It is not the purpose of this section to induce you to infringe any
-patents or other property right claims or to contest validity of any
-such claims; this section has the sole purpose of protecting the
-integrity of the free software distribution system, which is
-implemented by public license practices.  Many people have made
-generous contributions to the wide range of software distributed
-through that system in reliance on consistent application of that
-system; it is up to the author/donor to decide if he or she is willing
-to distribute software through any other system and a licensee cannot
-impose that choice.
+No Contributor makes additional grants as a result of Your choice to
+distribute the Covered Software under a subsequent version of this
+License (see Section 10.2) or under the terms of a Secondary License (if
+permitted under the terms of Section 3.3).
 
-This section is intended to make thoroughly clear what is believed to
-be a consequence of the rest of this License.
+2.5. Representation
 
-  8. If the distribution and/or use of the Program is restricted in
-certain countries either by patents or by copyrighted interfaces, the
-original copyright holder who places the Program under this License
-may add an explicit geographical distribution limitation excluding
-those countries, so that distribution is permitted only in or among
-countries not thus excluded.  In such case, this License incorporates
-the limitation as if written in the body of this License.
+Each Contributor represents that the Contributor believes its
+Contributions are its original creation(s) or it has sufficient rights
+to grant the rights to its Contributions conveyed by this License.
 
-  9. The Free Software Foundation may publish revised and/or new versions
-of the General Public License from time to time.  Such new versions will
-be similar in spirit to the present version, but may differ in detail to
-address new problems or concerns.
+2.6. Fair Use
 
-Each version is given a distinguishing version number.  If the Program
-specifies a version number of this License which applies to it and "any
-later version", you have the option of following the terms and conditions
-either of that version or of any later version published by the Free
-Software Foundation.  If the Program does not specify a version number of
-this License, you may choose any version ever published by the Free Software
-Foundation.
+This License is not intended to limit any rights You have under
+applicable copyright doctrines of fair use, fair dealing, or other
+equivalents.
 
-  10. If you wish to incorporate parts of the Program into other free
-programs whose distribution conditions are different, write to the author
-to ask for permission.  For software which is copyrighted by the Free
-Software Foundation, write to the Free Software Foundation; we sometimes
-make exceptions for this.  Our decision will be guided by the two goals
-of preserving the free status of all derivatives of our free software and
-of promoting the sharing and reuse of software generally.
+2.7. Conditions
 
-			    NO WARRANTY
+Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted
+in Section 2.1.
 
-  11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
-FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW.  EXCEPT WHEN
-OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
-PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
-OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
-MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.  THE ENTIRE RISK AS
-TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU.  SHOULD THE
-PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
-REPAIR OR CORRECTION.
+3. Responsibilities
+-------------------
 
-  12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
-WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
-REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
-INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
-OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
-TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
-YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
-PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
-POSSIBILITY OF SUCH DAMAGES.
+3.1. Distribution of Source Form
 
-		     END OF TERMS AND CONDITIONS
+All distribution of Covered Software in Source Code Form, including any
+Modifications that You create or to which You contribute, must be under
+the terms of this License. You must inform recipients that the Source
+Code Form of the Covered Software is governed by the terms of this
+License, and how they can obtain a copy of this License. You may not
+attempt to alter or restrict the recipients' rights in the Source Code
+Form.
 
-	    How to Apply These Terms to Your New Programs
+3.2. Distribution of Executable Form
 
-  If you develop a new program, and you want it to be of the greatest
-possible use to the public, the best way to achieve this is to make it
-free software which everyone can redistribute and change under these terms.
+If You distribute Covered Software in Executable Form then:
 
-  To do so, attach the following notices to the program.  It is safest
-to attach them to the start of each source file to most effectively
-convey the exclusion of warranty; and each file should have at least
-the "copyright" line and a pointer to where the full notice is found.
+(a) such Covered Software must also be made available in Source Code
+    Form, as described in Section 3.1, and You must inform recipients of
+    the Executable Form how they can obtain a copy of such Source Code
+    Form by reasonable means in a timely manner, at a charge no more
+    than the cost of distribution to the recipient; and
 
-    <one line to give the program's name and a brief idea of what it does.>
-    Copyright (C) <year>  <name of author>
+(b) You may distribute such Executable Form under the terms of this
+    License, or sublicense it under different terms, provided that the
+    license for the Executable Form does not attempt to limit or alter
+    the recipients' rights in the Source Code Form under this License.
 
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; either version 2 of the License, or
-    (at your option) any later version.
+3.3. Distribution of a Larger Work
 
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
+You may create and distribute a Larger Work under terms of Your choice,
+provided that You also comply with the requirements of this License for
+the Covered Software. If the Larger Work is a combination of Covered
+Software with a work governed by one or more Secondary Licenses, and the
+Covered Software is not Incompatible With Secondary Licenses, this
+License permits You to additionally distribute such Covered Software
+under the terms of such Secondary License(s), so that the recipient of
+the Larger Work may, at their option, further distribute the Covered
+Software under the terms of either this License or such Secondary
+License(s).
 
-    You should have received a copy of the GNU General Public License along
-    with this program; if not, write to the Free Software Foundation, Inc.,
-    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+3.4. Notices
 
-Also add information on how to contact you by electronic and paper mail.
+You may not remove or alter the substance of any license notices
+(including copyright notices, patent notices, disclaimers of warranty,
+or limitations of liability) contained within the Source Code Form of
+the Covered Software, except that You may alter any license notices to
+the extent required to remedy known factual inaccuracies.
 
-If the program is interactive, make it output a short notice like this
-when it starts in an interactive mode:
+3.5. Application of Additional Terms
 
-    Gnomovision version 69, Copyright (C) year name of author
-    Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
-    This is free software, and you are welcome to redistribute it
-    under certain conditions; type `show c' for details.
+You may choose to offer, and to charge a fee for, warranty, support,
+indemnity or liability obligations to one or more recipients of Covered
+Software. However, You may do so only on Your own behalf, and not on
+behalf of any Contributor. You must make it absolutely clear that any
+such warranty, support, indemnity, or liability obligation is offered by
+You alone, and You hereby agree to indemnify every Contributor for any
+liability incurred by such Contributor as a result of warranty, support,
+indemnity or liability terms You offer. You may include additional
+disclaimers of warranty and limitations of liability specific to any
+jurisdiction.
 
-The hypothetical commands `show w' and `show c' should show the appropriate
-parts of the General Public License.  Of course, the commands you use may
-be called something other than `show w' and `show c'; they could even be
-mouse-clicks or menu items--whatever suits your program.
+4. Inability to Comply Due to Statute or Regulation
+---------------------------------------------------
 
-You should also get your employer (if you work as a programmer) or your
-school, if any, to sign a "copyright disclaimer" for the program, if
-necessary.  Here is a sample; alter the names:
+If it is impossible for You to comply with any of the terms of this
+License with respect to some or all of the Covered Software due to
+statute, judicial order, or regulation then You must: (a) comply with
+the terms of this License to the maximum extent possible; and (b)
+describe the limitations and the code they affect. Such description must
+be placed in a text file included with all distributions of the Covered
+Software under this License. Except to the extent prohibited by statute
+or regulation, such description must be sufficiently detailed for a
+recipient of ordinary skill to be able to understand it.
 
-  Yoyodyne, Inc., hereby disclaims all copyright interest in the program
-  `Gnomovision' (which makes passes at compilers) written by James Hacker.
+5. Termination
+--------------
 
-  <signature of Ty Coon>, 1 April 1989
-  Ty Coon, President of Vice
+5.1. The rights granted under this License will terminate automatically
+if You fail to comply with any of its terms. However, if You become
+compliant, then the rights granted under this License from a particular
+Contributor are reinstated (a) provisionally, unless and until such
+Contributor explicitly and finally terminates Your grants, and (b) on an
+ongoing basis, if such Contributor fails to notify You of the
+non-compliance by some reasonable means prior to 60 days after You have
+come back into compliance. Moreover, Your grants from a particular
+Contributor are reinstated on an ongoing basis if such Contributor
+notifies You of the non-compliance by some reasonable means, this is the
+first time You have received notice of non-compliance with this License
+from such Contributor, and You become compliant prior to 30 days after
+Your receipt of the notice.
 
-This General Public License does not permit incorporating your program into
-proprietary programs.  If your program is a subroutine library, you may
-consider it more useful to permit linking proprietary applications with the
-library.  If this is what you want to do, use the GNU Lesser General
-Public License instead of this License.
+5.2. If You initiate litigation against any entity by asserting a patent
+infringement claim (excluding declaratory judgment actions,
+counter-claims, and cross-claims) alleging that a Contributor Version
+directly or indirectly infringes any patent, then the rights granted to
+You by any and all Contributors for the Covered Software under Section
+2.1 of this License shall terminate.
+
+5.3. In the event of termination under Sections 5.1 or 5.2 above, all
+end user license agreements (excluding distributors and resellers) which
+have been validly granted by You or Your distributors under this License
+prior to termination shall survive termination.
+
+************************************************************************
+*                                                                      *
+*  6. Disclaimer of Warranty                                           *
+*  -------------------------                                           *
+*                                                                      *
+*  Covered Software is provided under this License on an "as is"       *
+*  basis, without warranty of any kind, either expressed, implied, or  *
+*  statutory, including, without limitation, warranties that the       *
+*  Covered Software is free of defects, merchantable, fit for a        *
+*  particular purpose or non-infringing. The entire risk as to the     *
+*  quality and performance of the Covered Software is with You.        *
+*  Should any Covered Software prove defective in any respect, You     *
+*  (not any Contributor) assume the cost of any necessary servicing,   *
+*  repair, or correction. This disclaimer of warranty constitutes an   *
+*  essential part of this License. No use of any Covered Software is   *
+*  authorized under this License except under this disclaimer.         *
+*                                                                      *
+************************************************************************
+
+************************************************************************
+*                                                                      *
+*  7. Limitation of Liability                                          *
+*  --------------------------                                          *
+*                                                                      *
+*  Under no circumstances and under no legal theory, whether tort      *
+*  (including negligence), contract, or otherwise, shall any           *
+*  Contributor, or anyone who distributes Covered Software as          *
+*  permitted above, be liable to You for any direct, indirect,         *
+*  special, incidental, or consequential damages of any character      *
+*  including, without limitation, damages for lost profits, loss of    *
+*  goodwill, work stoppage, computer failure or malfunction, or any    *
+*  and all other commercial damages or losses, even if such party      *
+*  shall have been informed of the possibility of such damages. This   *
+*  limitation of liability shall not apply to liability for death or   *
+*  personal injury resulting from such party's negligence to the       *
+*  extent applicable law prohibits such limitation. Some               *
+*  jurisdictions do not allow the exclusion or limitation of           *
+*  incidental or consequential damages, so this exclusion and          *
+*  limitation may not apply to You.                                    *
+*                                                                      *
+************************************************************************
+
+8. Litigation
+-------------
+
+Any litigation relating to this License may be brought only in the
+courts of a jurisdiction where the defendant maintains its principal
+place of business and such litigation shall be governed by laws of that
+jurisdiction, without reference to its conflict-of-law provisions.
+Nothing in this Section shall prevent a party's ability to bring
+cross-claims or counter-claims.
+
+9. Miscellaneous
+----------------
+
+This License represents the complete agreement concerning the subject
+matter hereof. If any provision of this License is held to be
+unenforceable, such provision shall be reformed only to the extent
+necessary to make it enforceable. Any law or regulation which provides
+that the language of a contract shall be construed against the drafter
+shall not be used to construe this License against a Contributor.
+
+10. Versions of the License
+---------------------------
+
+10.1. New Versions
+
+Mozilla Foundation is the license steward. Except as provided in Section
+10.3, no one other than the license steward has the right to modify or
+publish new versions of this License. Each version will be given a
+distinguishing version number.
+
+10.2. Effect of New Versions
+
+You may distribute the Covered Software under the terms of the version
+of the License under which You originally received the Covered Software,
+or under the terms of any subsequent version published by the license
+steward.
+
+10.3. Modified Versions
+
+If you create software not governed by this License, and you want to
+create a new license for such software, you may create and use a
+modified version of this License if you rename the license and remove
+any references to the name of the license steward (except to note that
+such modified license differs from this License).
+
+10.4. Distributing Source Code Form that is Incompatible With Secondary
+Licenses
+
+If You choose to distribute Source Code Form that is Incompatible With
+Secondary Licenses under the terms of this version of the License, the
+notice described in Exhibit B of this License must be attached.
+
+Exhibit A - Source Code Form License Notice
+-------------------------------------------
+
+  This Source Code Form is subject to the terms of the Mozilla Public
+  License, v. 2.0. If a copy of the MPL was not distributed with this
+  file, You can obtain one at https://mozilla.org/MPL/2.0/.
+
+If it is not possible or desirable to put the notice in a particular
+file, then You may include the notice in a location (such as a LICENSE
+file in a relevant directory) where a recipient would be likely to look
+for such a notice.
+
+You may add additional accurate notices of copyright ownership.
+
+Exhibit B - "Incompatible With Secondary Licenses" Notice
+---------------------------------------------------------
+
+  This Source Code Form is "Incompatible With Secondary Licenses", as
+  defined by the Mozilla Public License, v. 2.0.

MANIFEST.in

@@ -1,3 +1,9 @@
 include LICENSE
 include README.rst
 recursive-include scripts *
+include atrcopy/templates/*
+include test_data/*_test?.atr
+include test_data/*_test?.atr.*
+include test_data/*.xex
+include test_data/rebuild.sh
+include test_data/create_binary.py

README.rst

@@ -1,20 +1,487 @@
 atrcopy
 =======
 
-Utilities to list files on and extract files from Atari 8-bit emulator disk
-images.  Eventually, I hope to add support for these images to pyfilesystem.
+Python command line utility to manage file systems on Atari 8-bit and Apple ][
+disk images.
+
+.. contents:: **Contents**
 
 Prerequisites
--------------
+=============
 
-Starting with atrcopy 2.0, numpy is required.
+Python
+------
 
-The standard python install tool, pip, does not seem to be able to handle the
-automatic installation of numpy, so to install atrcopy, use::
+Starting with ``atrcopy`` 7.0, Python 3.6 is **required**. Python 2 support has
+been dropped. Python 3.7 and beyond will be supported when they are released,
+but 3.6 will probably remain the minimum version. From what I know now of
+future Python versions, I don't plan on requiring any language features beyond
+3.6.
+
+Supported Python versions:
+
+* Python 3.6 (and later)
+
+If you need Python 2 support, ``atrcopy`` 6.5 and earlier supports Python 2.7,
+which you can install with ``pip install "atrcopy<7.0"``
+
+Dependencies
+------------
+
+* numpy
+
+It will be automatically installed when installing ``atrcopy`` with ``pip`` as
+described below.
+
+For development, pytest is used to run the test suite, but this is not required
+for normal installation of ``atrcopy``.
+
+Installation
+============
+
+``atrcopy`` is available in the `PyPI <https://pypi.python.org/pypi/atrcopy/>`_
+and installable using ``pip``::
 
-    pip install numpy
     pip install atrcopy
 
+Linux and macOS note: if numpy needs to be installed on your system, it may be
+compiled from source which can take several minutes.
+
+Features
+========
+
+* list contents of disk images
+* copy files to and from disk images
+* delete files from disk images
+* create new disk images
+* concatenate binary data together into a file on the disk image
+* compile assembly source into binary files if `pyatasm <https://pypi.python.org/pypi/pyatasm>`_ is installed
+
+**Note:** The command line argument structure was changed starting with
+``atrcopy`` 4.0 -- it is now based on subcommands, much like ``git`` uses ``git
+pull``, ``git clone``, ``git branch``, etc. Upgrading from a version prior to
+4.0 will require modification of scripts that use ``atrcopy`` 3.x-style command
+line arguments.
+
+
+Supported Formats
+=================
+
+Supported Disk Image Types
+--------------------------
+
+* ``XFD``: XFormer images, basically raw disk dumps
+* ``ATR``: Nick Kennedy's disk image format; includes 16 byte header
+* ``DSK``: Apple ][ DOS 3.3 disk image; raw sector dump
+
+Supported File System Formats
+-----------------------------
+
++----------------+-------------+---------+-------+-------------------+
+| File System    | Platform    | Read    | Write | Status            |
++================+=============+=========+=======+===================+
+| DOS 2 (90K)    | Atari 8-bit | Yes     | Yes   | Fully supported   |
++----------------+-------------+---------+-------+-------------------+
+| DOS 2 (180K)   | Atari 8-bit | Yes     | Yes   | Fully supported   |
++----------------+-------------+---------+-------+-------------------+
+| DOS 2.5 (130K) | Atari 8-bit | Yes     | Yes   | Fully supported   |
++----------------+-------------+---------+-------+-------------------+
+| DOS 3 (130K)   | Atari 8-bit | No      | No    | Unimplemented     |
++----------------+-------------+---------+-------+-------------------+
+| SpartaDOS      | Atari 8-bit | No      | No    | Under development |
++----------------+-------------+---------+-------+-------------------+
+| MyDOS          | Atari 8-bit | Partial | No    | Under development |
++----------------+-------------+---------+-------+-------------------+
+| DOS 3.3        | Apple ][    | Yes     | Yes   | Fully supported   |
++----------------+-------------+---------+-------+-------------------+
+| ProDOS 8       | Apple ][    | No      | No    | Unimplemented     |
++----------------+-------------+---------+-------+-------------------+
+
+
+Other Supported Formats
+-----------------------
+
++----------+----------------------------------+---------+-------+-----------------+
+| Format   | Platform/description             | Read    | Write | Status          |
++==========+==================================+=========+=======+=================+
+| ``.xex`` | Atari 8-bit executable files     | Yes     | Yes   | Fully supported |
++----------+----------------------------------+---------+-------+-----------------+
+| KBoot    | Atari 8-bit ``xex`` in boot disk | Yes     | Yes   | Fully supported |
++----------+----------------------------------+---------+-------+-----------------+
+| ``.car`` | Atari 8-bit cartridge images     | Yes     | No    | Read only       |
++----------+----------------------------------+---------+-------+-----------------+
+| BSAVE    | Apple ][ ``BSAVE`` data          | Yes     | Yes   | Fully supported |
++----------+----------------------------------+---------+-------+-----------------+
+| ``.zip`` | MAME ROM zipfiles                | Partial | No    | Experimental    |
++----------+----------------------------------+---------+-------+-----------------+
+
+**Note:** Atari ROM cartridges are supported in both both plain binary and
+atari800 ``.car`` format
+
+
+Supported Compression/Container Formats
+---------------------------------------
+
+Starting with ``atrcopy`` 8.0, compressed disk images are supported
+transparently, so any type of disk image compressed with one of the supported
+container formats can be used directly, without first decompressing it before
+running ``atrcopy``.
+
++--------------------+----------+------+-------+------------------------------+
+| Container          | File Ext | Read | Write | Status                       |
++====================+==========+======+=======+==============================+
+| gzip               | .gz      | Yes  | No    | Read only                    |
++--------------------+----------+------+-------+------------------------------+
+| bzip2              | .bz2     | Yes  | No    | Read only                    |
++--------------------+----------+------+-------+------------------------------+
+| lzma               | .xz      | Yes  | No    | Read only                    |
++--------------------+----------+------+-------+------------------------------+
+| Disk Communicator  | .dcm     | No   | No    | Recognized but unimplemented |
++--------------------+----------+------+-------+------------------------------+
+
+
+Usage
+=====
+
+::
+
+    atrcopy DISK_IMAGE <global options> COMMAND <command options>
+
+where the available commands include:
+
+* ``list``: list files on the disk image. This is the default if no command is specified
+* ``create``: create a new disk image
+* ``add``: add files to a disk image
+* ``extract``: copy files from the disk image to the local file system
+* ``assemble``: create a binary file from ATasm source, optionally including segments containing raw binary data
+* ``boot``: create a boot disk using various binary data as input
+* ``delete``: delete files from the disk image
+* ``vtoc``: show and manipulate the VTOC for images that support it
+
+Except when using the ``--help`` option, the ``DISK_IMAGE`` is always required
+which points to the path on your local file system of the disk image.
+``COMMAND`` is one of the commands listed above, and the commands may be
+abbreviated as shown here::
+
+    $ atrcopy --help
+    usage: atrcopy DISK_IMAGE [-h] [-v] [--dry-run] COMMAND ...
+
+    Manipulate files on several types of 8-bit computer disk images. Type 'atrcopy
+    DISK_IMAGE COMMAND --help' for list of options available for each command.
+
+    positional arguments:
+      COMMAND
+        list (t,ls,dir,catalog)
+                            List files on the disk image. This is the default if
+                            no command is specified
+        crc                 List files on the disk image and the CRC32 value in
+                            format suitable for parsing
+        extract (x)         Copy files from the disk image to the local filesystem
+        add (a)             Add files to the disk image
+        create (c)          Create a new disk image
+        assemble (s,asm)    Create a new binary file in the disk image
+        boot (b)            Create a bootable disk image
+        delete (rm,del)     Delete files from the disk image
+        vtoc (v)            Show a formatted display of sectors free in the disk
+                            image
+        segments            Show the list of parsed segments in the disk image
+
+    optional arguments:
+      -h, --help            show this help message and exit
+      -v, --verbose
+      --dry-run             don't perform operation, just show what would have
+                            happened
+
+
+Help for available options for each command is available without specifying a
+disk image, using a command line like::
+
+    atrcopy COMMAND --help
+
+so for example, the help for assembling a binary file is::
+
+    $ atrcopy asm --help
+    usage: atrcopy DISK_IMAGE assemble [-h] [-f] [-s [ASM [ASM ...]]]
+                                       [-d [DATA [DATA ...]]] [-r RUN_ADDR] -o
+                                       OUTPUT
+
+    optional arguments:
+      -h, --help            show this help message and exit
+      -f, --force           allow file overwrites in the disk image
+      -s [ASM [ASM ...]], --asm [ASM [ASM ...]]
+                            source file(s) to assemble using pyatasm
+      -d [DATA [DATA ...]], -b [DATA [DATA ...]], --data [DATA [DATA ...]]
+                            binary data file(s) to add to assembly, specify as
+                            file@addr. Only a portion of the file may be included;
+                            specify the subset using standard python slice
+                            notation: file[subset]@addr
+      -r RUN_ADDR, --run-addr RUN_ADDR, --brun RUN_ADDR
+                            run address of binary file if not the first byte of
+                            the first segment
+      -o OUTPUT, --output OUTPUT
+                            output file name in disk image
+
+
+
+Examples
+========
+
+List all files on a disk image::
+
+    $ atrcopy DOS_25.ATR 
+    DOS_25.ATR: ATR Disk Image (size=133120 (1040x128B), crc=0 flags=0 unused=0) Atari DOS Format: 1010 usable sectors (739 free), 6 files
+    File #0  (.2.u.*) 004 DOS     SYS  037
+    File #1  (.2.u.*) 041 DUP     SYS  042
+    File #2  (.2.u.*) 083 RAMDISK COM  009
+    File #3  (.2.u.*) 092 SETUP   COM  070
+    File #4  (.2.u.*) 162 COPY32  COM  056
+    File #5  (.2.u.*) 218 DISKFIX COM  057
+
+Extract a file::
+
+    $ atrcopy DOS_25.ATR extract SETUP.COM
+    DOS_25.ATR: ATR Disk Image (size=133120 (1040x128B), crc=0 flags=0 unused=0) Atari DOS Format: 1010 usable sectors (739 free), 6 files
+    extracting SETUP.COM -> SETUP.COM
+
+Extract all files::
+
+    $ atrcopy DOS_25.ATR extract --all
+    DOS_25.ATR: ATR Disk Image (size=133120 (1040x128B), crc=0 flags=0 unused=0) Atari DOS Format: 1010 usable sectors (739 free), 6 files
+    extracting File #0  (.2.u.*) 004 DOS     SYS  037 -> DOS.SYS
+    extracting File #1  (.2.u.*) 041 DUP     SYS  042 -> DUP.SYS
+    extracting File #2  (.2.u.*) 083 RAMDISK COM  009 -> RAMDISK.COM
+    extracting File #3  (.2.u.*) 092 SETUP   COM  070 -> SETUP.COM
+    extracting File #4  (.2.u.*) 162 COPY32  COM  056 -> COPY32.COM
+    extracting File #5  (.2.u.*) 218 DISKFIX COM  057 -> DISKFIX.COM
+
+Extract all, using the abbreviated command and converting to lower case on the
+host file system::
+
+    $ atrcopy DOS_25.ATR x --all -l
+    DOS_25.ATR: ATR Disk Image (size=133120 (1040x128B), crc=0 flags=0 unused=0) Atari DOS Format: 1010 usable sectors (739 free), 6 files
+    extracting File #0  (.2.u.*) 004 DOS     SYS  037 -> dos.sys
+    extracting File #1  (.2.u.*) 041 DUP     SYS  042 -> dup.sys
+    extracting File #2  (.2.u.*) 083 RAMDISK COM  009 -> ramdisk.com
+    extracting File #3  (.2.u.*) 092 SETUP   COM  070 -> setup.com
+    extracting File #4  (.2.u.*) 162 COPY32  COM  056 -> copy32.com
+    extracting File #5  (.2.u.*) 218 DISKFIX COM  057 -> diskfix.com
+
+Creating Disk Images
+--------------------
+
+Several template disk images are included in the distribution, and these can be
+used to create blank disk images that subsequent uses of ``atrcopy`` can
+reference.
+
+The available disk images can be viewed using ``atrcopy create --help``::
+
+    $ atrcopy create --help
+    usage: atrcopy DISK_IMAGE create [-h] [-f] TEMPLATE
+
+    positional arguments:
+      TEMPLATE     template to use to create new disk image; see below for list of
+                   available built-in templates
+
+    optional arguments:
+      -h, --help   show this help message and exit
+      -f, --force  replace disk image file if it exists
+
+    available templates:
+      dos2dd          Atari 8-bit DOS 2 double density (180K), empty VTOC
+      dos2ed          Atari 8-bit DOS 2 enhanced density (130K), empty VTOC
+      dos2ed+2.5      Atari 8-bit DOS 2 enhanced density (130K) DOS 2.5 system disk
+      dos2sd          Atari 8-bit DOS 2 single density (90K), empty VTOC
+      dos2sd+2.0s     Atari 8-bit DOS 2 single density (90K) DOS 2.0S system disk
+      dos33           Apple ][ DOS 3.3 (140K) standard RWTS, empty VTOC
+      dos33autobrun   Apple ][ DOS 3.3 (140K) disk image for binary program
+                      development: HELLO sets fullscreen HGR and calls BRUN on
+                      user-supplied AUTOBRUN binary file
+
+To create a new image, use::
+
+    $ atrcopy game.dsk create dos33autobrun
+
+which will create a new file called ``game.dsk`` based on the ``dos33autobrun``
+image.
+
+``dos33autobrun`` is a special image that can be used to create autoloading
+binary programs. It contains an Applesoft Basic file called ``HELLO`` which
+will autoload on boot. It sets the graphics mode to fullscreen hi-res graphics
+(the first screen at $2000) and executes a ``BRUN`` command to start a binary
+file named ``AUTOBRUN``. ``AUTOBRUN`` doesn't exist in the image, it's for you
+to supply.
+
+
+Creating a Custom Boot Disk
+---------------------------
+
+Blocks of binary data can be combined into a boot disk in either ATR format for
+Atari or DSK format for Apple::
+
+    $ atrcopy boot --help
+    usage: atrcopy DISK_IMAGE boot [-h] [-f] [-s [ASM [ASM ...]]]
+                                   [-d [DATA [DATA ...]]] [-b [OBJ [OBJ ...]]]
+                                   [-r RUN_ADDR]
+
+    optional arguments:
+      -h, --help            show this help message and exit
+      -f, --force           allow file overwrites in the disk image
+      -s [ASM [ASM ...]], --asm [ASM [ASM ...]]
+                            source file(s) to assemble using pyatasm
+      -d [DATA [DATA ...]], --data [DATA [DATA ...]]
+                            binary data file(s) to add to assembly, specify as
+                            file@addr. Only a portion of the file may be included;
+                            specify the subset using standard python slice
+                            notation: file[subset]@addr
+      -b [OBJ [OBJ ...]], --obj [OBJ [OBJ ...]], --bload [OBJ [OBJ ...]]
+                            binary file(s) to add to assembly, either executables
+                            or labeled memory dumps (e.g. BSAVE on Apple ][),
+                            parsing each file's binary segments to add to the
+                            resulting disk image at the load address for each
+                            segment
+      -r RUN_ADDR, --run-addr RUN_ADDR, --brun RUN_ADDR
+                            run address of binary file if not the first byte of
+                            the first segment
+
+One of ``-s``, ``-d``, or ``-b`` must be speficied to provide the source for
+the boot disk. The ``-b`` argument can take an Atari binary in XEX format, and
+will properly handle multiple segments within that file. If no starting address
+is supplied (or, if using an XEX, to override the start address normally
+contained within the XEX), use the ``-r`` option. Otherwise, the run address
+will point to the first byte of the first binary segment.
+
+
+Creating Programs on the Disk Image
+-----------------------------------
+
+The simple assembler included in ``atrcopy`` can create binary programs by
+connecting binary data together in a single file and specifying a start address
+so it can be executed by the system's binary run command.
+
+It is also possible to assemble text files that use the MAC/65 syntax, because
+support for `pyatasm <https://pypi.python.org/pypi/pyatasm>`_ is built-in (but
+optional). MAC/65 is a macro assembler originally designed for the Atari 8-bit
+machines but since it produces 6502 code it can be used to compile for any
+machine that uses the 6502: Apple, Commodore, etc.
+
+Creating Atari 8-bit Executables
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Atari 8-bit object files include a small header and an arbitrary number of
+segments. Each segment defines a contiguous block of data with a start and end
+address. If the file has multiple segments, they will be processed in the order
+they appear in the file, not by segment start address.
+
+This example creates a new ``xex`` on a disk that combines the segments of an
+already existing executable with some new assembly code.
+
+After creating the test image with::
+
+    $ atrcopy test.atr create dos2sd
+    using dos2sd template:
+      Atari 8-bit DOS 2 single density (90K), empty VTOC
+    created test.atr: ATR Disk Image (size=92160 (720x128B), crc=0 flags=0 unused=0) Atari DOS Format: 707 usable sectors (707 free), 0 files
+
+this command compiles the file ``test_header.s`` and prefixes it to the
+existing executable::
+
+    $ atrcopy test.atr asm -s test_header.s -b air_defense_v18.xex -o test.xex -f
+    test.atr: ATR Disk Image (size=92160 (720x128B), crc=0 flags=0 unused=0) Atari DOS Format: 707 usable sectors (707 free), 0 files
+    fname: test_header.s
+    Pass 1: Success. (0 warnings)
+    Pass 2: 
+    adding 0600 - 0653, size=0053 ($53 bytes @ 0600) from test_header.s assembly
+    adding 02e2 - 02e4, size=0002 ($2 bytes @ 02e2) from test_header.s assembly
+    adding  $02e0-$02e2 ($0002 @ $0006) from air_defense_v18.xex
+    adding  $6000-$6bd4 ($0bd4 @ $000c) from air_defense_v18.xex
+    total file size: $c3d (3133) bytes
+    copying test.xex to test.atr
+
+
+Creating DOS 3.3 Binaries
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For this example, the goal is to produce a single binary file that combines a
+hi-res image ``title.bin`` loaded at 2000 hex (the first hi-res screen) and
+code at 6000 hex from the binary file ``game``, with a start address of 6000
+hex.
+
+The binary file ``game`` was assembled using the assembler from the 
+`cc65 <https://github.com/cc65/cc65>`_ project, using the command::
+
+    cl65 -t apple2 --cpu 6502 --start-addr 0x6000 -o game game.s
+
+Because the Apple ][ binary format is limited to a single contiguous block of
+data with a start address of the first byte of data loaded, ``atrcopy`` will
+fill the gaps between any segments that aren't contiguous with zeros. If the
+start address is not the first byte of the first specified segment, a small
+segment will be included at the beginning that jumps to the specified ``brun``
+address (shown here as the segment from 1ffd - 2000). Note the gap between 4000
+and 6000 hex will be filled with zeros::
+
+    $ atrcopy game.dsk create dos33autobrun
+    using dos33autobrun template:
+      Apple ][ DOS 3.3 (140K) disk image for binary program development: HELLO sets
+      fullscreen HGR and calls BRUN on user-supplied AUTOBRUN binary file
+    created game.dsk: DOS 3.3 Disk Image (size=143360 (560x256b)
+    File #0  ( A) 002 HELLO                          003 001
+
+    $ atrcopy game.dsk asm -d title.bin@2000 -b game --brun 6000 -f -o AUTOBRUN
+    game.dsk: DOS 3.3 Disk Image (size=143360 (560x256b)
+    adding BSAVE data $6000-$6ef3 ($0ef3 @ $0004) from game
+    setting data for $1ffd - $2000 at index $0004
+    setting data for $2000 - $4000 at index $0007
+    setting data for $6000 - $6ef3 at index $4007
+    total file size: $4efa (20218) bytes
+    copying AUTOBRUN to game.dsk
+
+
+Example on macOS
+----------------
+
+macOS supplies python with the operating system so you shouldn't need to
+install a framework version from python.org.
+
+To prevent overwriting important system files, it's best to create a working
+folder: a new empty folder somewhere and do all your testing in that folder.
+For this example, create a folder called ``atrtest`` in your ``Documents``
+folder.  Put a few disk images in this directory to use for testing.
+
+Since this is a command line program, you must get to a command line prompt.
+Start a Terminal by double clicking on Terminal.app in the
+``Applications/Utilities`` folder in the Finder.  When Terminal opens, it will
+put you in your home folder automatically.  Go to the ``atrtest`` folder by
+typing::
+
+    cd Documents/atrtest
+
+You can see the ATR images you placed in this directory by using the
+command::
+
+    ls -l
+
+For example, you might see::
+
+    mac:~/Documents/atrtest $ ls -l
+    -rw-r--r-- 1 rob  staff  92176 May 18 21:57 GAMES1.ATR
+
+Now, run the program by typing ``atrcopy GAMES1.ATR`` and you should
+see the contents of the ``ATR`` image in the familiar Atari DOS format::
+
+    mac:~/Documents/atrtest $ atrcopy GAMES1.ATR
+    GAMES1.ATR: ATR Disk Image (size=92160 (720x128B), crc=0 flags=0 unused=0) Atari DOS Format: 707 usable sectors (17 free), 9 files
+    File #0  (.2.u.*) 004 DOS     SYS  039
+    File #1  (.2.u.*) 043 MINER2       138
+    File #2  (.2.u.*) 085 DEFENDER     132
+    File #3  (.2.u.*) 217 CENTIPEDE    045
+    File #4  (.2.u.*) 262 GALAXIAN     066
+    File #5  (.2.u.*) 328 AUTORUN SYS  005
+    File #6  (.2.u.*) 439 DIGDUG       133
+    File #7  (.2.u.*) 531 ANTEATER     066
+    File #8  (.2.u.*) 647 ASTEROIDS    066
+
+See other examples as above.
+
 
 References
 ==========
@@ -23,142 +490,34 @@ References
 * http://atari.kensclassics.org/dos.htm
 * http://www.crowcastle.net/preston/atari/
 * http://www.atarimax.com/jindroush.atari.org/afmtatr.html
+* https://archive.org/details/Beneath_Apple_DOS_OCR
 
-
-Supported Disk Image Formats
-============================
-
-* ``XFD``: XFormer images, basically raw disk dumps
-* ``ATR``: Nick Kennedy's disk image format; includes 16 byte header
-
-Supported Filesystem Formats
-----------------------------
-
-* XEX format: Atari executable files
-* Atari DOS in single, enhanced, and double density
-* KBoot format: a single executable file packaged up into a bootable disk image
-
-
-Example Usage
-=============
-
-To extract all non SYS files while converting to lower case, use::
-
-    $ python atrcopy.py /tmp/GAMES1.ATR -x -l -n
-    GAMES1.ATR
-    File #0 : *DOS     SYS  039 : skipping system file dos.sys
-    File #1 : *MINER2       138 : copying to miner2
-    File #2 : *DEFENDER     132 : copying to defender
-    File #3 : *CENTIPEDE    045 : copying to centiped.e
-    File #4 : *GALAXIAN     066 : copying to galaxian
-    File #5 : *AUTORUN SYS  005 : skipping system file autorun.sys
-    File #6 : *DIGDUG       133 : copying to digdug
-    File #7 : *ANTEATER     066 : copying to anteater
-    File #8 : *ASTEROIDS    066 : copying to asteroid.s
-
-
-Example on Mac OS X
--------------------
-
-OS X supplies python with the operating system so you shouldn't need to install
-a framework version from python.org.
-
-To prevent overwriting important system files, it's best to create a working
-folder: a new empty folder somewhere and do all your testing in that folder.
-For this example, create a folder called ``atrtest`` in your ``Documents``
-folder.  Put a few disk images in this directory to use for testing.
-
-Download or copy the file ``atrcopy.py`` and put it the ``Documents/atrtest``
-folder.
-
-Since this is a command line programe, you must start a Terminal by double
-clicking on Terminal.app in the ``Applications/Utilities`` folder in
-the Finder.  When Terminal opens, it will put you in your home folder
-automatically.  Go to the ``atrtest`` folder by typing::
-
-    cd Documents/atrtest
-
-You should see the file ``atrcopy.py`` as well as the other ATR images you
-placed in this directory by using the command::
-
-    ls -l
-
-For example, you might see::
-
-    mac:~/Documents/atrtest $ ls -l
-    -rw-r--r-- 1 rob  staff  92176 May 18 21:57 GAMES1.ATR
-    -rwxr-xr-x 1 rob  staff   8154 May 18 22:36 atrcopy.py
-
-Now, run the program by typing ``python atrcopy.py YOURFILE.ATR`` and you should
-see the contents of the ``ATR`` image in the familiar Atari DOS format::
-
-    mac:~/Documents/atrtest $ python atrcopy.py GAMES1.ATR
-    GAMES1.ATR
-    File #0 : *DOS     SYS  039 
-    File #1 : *MINER2       138 
-    File #2 : *DEFENDER     132 
-    File #3 : *CENTIPEDE    045 
-    File #4 : *GALAXIAN     066 
-    File #5 : *AUTORUN SYS  005 
-    File #6 : *DIGDUG       133 
-    File #7 : *ANTEATER     066 
-    File #8 : *ASTEROIDS    066 
-
-Without any additional arguments, it will not extract files.  To actually pull
-the files out of the ``ATR`` image, you need to specify the ``-x`` command line
-argument::
-
-    mac:~/Documents/atrtest $ python atrcopy.py -x GAMES1.ATR
-    GAMES1.ATR
-    File #0 : *DOS     SYS  039 : copying to DOS.SYS
-    File #1 : *MINER2       138 : copying to MINER2
-    File #2 : *DEFENDER     132 : copying to DEFENDER
-    File #3 : *CENTIPEDE    045 : copying to CENTIPED.E
-    File #4 : *GALAXIAN     066 : copying to GALAXIAN
-    File #5 : *AUTORUN SYS  005 : copying to AUTORUN.SYS
-    File #6 : *DIGDUG       133 : copying to DIGDUG
-    File #7 : *ANTEATER     066 : copying to ANTEATER
-    File #8 : *ASTEROIDS    066 : copying to ASTEROID.S
-
-There are other flags, like the ``-l`` flag to covert to lower case, and the
-``--xex`` flag to add the `.XEX` extension to the filename, and ``-n`` to skip
-DOS files.  So a full example might be::
-
-    mac:~/Documents/atrtest $ python atrcopy.py -n -l -x --xex GAMES1.ATR
-    GAMES1.ATR
-    File #0 : *DOS     SYS  039 : skipping system file dos.sys
-    File #1 : *MINER2       138 : copying to miner2.xex
-    File #2 : *DEFENDER     132 : copying to defender.xex
-    File #3 : *CENTIPEDE    045 : copying to centipede.xex
-    File #4 : *GALAXIAN     066 : copying to galaxian.xex
-    File #5 : *AUTORUN SYS  005 : skipping system file autorun.sys
-    File #6 : *DIGDUG       133 : copying to digdug.xex
-    File #7 : *ANTEATER     066 : copying to anteater.xex
-    File #8 : *ASTEROIDS    066 : copying to asteroids.xex
-
-
-Command Line Arguments
+Related Atari Projects
 ----------------------
 
-The available command line arguments are summarized using the standard ``--
-help`` argument::
+* `franny <http://atari8.sourceforge.net/franny.html>`_: (C, macOS/linux) Command line program to manage Atari DOS 2 and SpartaDOS II image and file systems
+* `dir2atr <http://www.horus.com/~hias/atari/>`_: (Win) Suite of command line programs to manage Atari disk images and DOS 2/MyDOS file systems
+* `atadim <http://raster.infos.cz/atari/forpc/atadim.htm>`_: (Win) Graphical program to manage Atari disk images and DOS 2/MyDOS file systems
 
-    $ python atrcopy.py --help
-    usage: atrcopy.py [-h] [-v] [-l] [--dry-run] [-n] [-x] [--xex] ATR [ATR ...]
+Related Apple Projects
+----------------------
 
-    Extract images off ATR or XFD format disks
+Turns out there are a ton of Apple ][ disk image viewers and editors! I was pointed to the list from the `diskii project <https://github.com/zellyn/diskii>`_, so I've included most of that list here.
 
-    positional arguments:
-      ATR            a disk image file [or a list of them]
-
-    optional arguments:
-      -h, --help     show this help message and exit
-      -v, --verbose
-      -l, --lower    convert filenames to lower case
-      --dry-run      don't extract, just show what would have been extracted
-      -n, --no-sys   only extract things that look like games (no DOS or .SYS
-                     files)
-      -x, --extract  extract files
-      --xex          add .xex extension
-      -f, --force    force operation on disk images that have bad directory
-                     entries or look like boot disks
+* `a2disk <https://github.com/jtauber/a2disk>`_ (Python 3) DOS 3.3 reader and Applesoft BASIC detokenizer
+* `cppo <https://github.com/RasppleII/a2server/blob/master/scripts/tools/cppo>`_ (Python) a script from the `a2server <http://ivanx.com/a2server/>`_ project to read DOS 3.3 and ProDOS disk images
+* `Driv3rs <https://github.com/thecompu/Driv3rs>`_ (Python) Apple III SOS DSK image utility
+* `c2d <https://github.com/datajerk/c2d>`_: (C, Win/macOS/linux) Command line program to create bootable Apple disk images (no file system)
+* `Apple Commander <http://applecommander.sourceforge.net/>`_: (Java) Command line program to manage Apple disk images and file systems
+* `Cider Press <http://a2ciderpress.com/>`_: (Win) Graphical program to manage Apple disk images and file systems
+* `diskii <https://github.com/zellyn/diskii>`_: (Go) Command line tool, under development
+* `Cadius <http://brutaldeluxe.fr/products/crossdevtools/cadius/index.html>`_ (Win) Brutal Deluxe's commandline tools
+* `dsktool <https://github.com/cybernesto/dsktool.rb>`_ (Ruby)
+* `Apple II Disk Tools <https://github.com/cmosher01/Apple-II-Disk-Tools>`_ (C)
+* `libA2 <https://github.com/madsen/perl-libA2>`_ (Perl)
+* `AppleSAWS <https://github.com/markdavidlong/AppleSAWS>`_ (Qt, Win/macOS/linux) very cool looking GUI
+* `DiskBrowser <https://github.com/dmolony/DiskBrowser>`_ (Java) GUI tool that even displays Wizardry levels and VisiCalc files!
+* `dos33fsprogs <https://github.com/deater/dos33fsprogs>`_ (C)
+* `apple2-disk-util <https://github.com/slotek/apple2-disk-util>`_ (Ruby)
+* `dsk2nib <https://github.com/slotek/dsk2nib>`_ (C)
+* `standard-delivery <https://github.com/peterferrie/standard-delivery>`_ (6502 assembly) Apple II single-sector fast boot-loader

atrcopy/__init__.py

@@ -1,21 +1,29 @@
-__version__ = "3.0.0"
+import os
+import sys
+import zlib
+import json
 
 import logging
+log = logging.getLogger(__name__)
+
+from ._version import __version__
 
 try:
     import numpy as np
 except ImportError:
     raise RuntimeError("atrcopy %s requires numpy" % __version__)
 
-from errors import *
-from ataridos import AtariDosDiskImage, AtariDosFile, get_xex
-from diskimages import AtrHeader, BootDiskImage, add_atr_header
-from kboot import KBootImage, add_xexboot_header
-from segments import SegmentData, SegmentSaver, DefaultSegment, EmptySegment, ObjSegment, RawSectorsSegment, user_bit_mask, match_bit_mask, comment_bit_mask, data_bit_mask, selected_bit_mask, diff_bit_mask, not_user_bit_mask, interleave_segments
-from spartados import SpartaDosDiskImage
-from cartridge import A8CartHeader, AtariCartImage
-from parsers import SegmentParser, DefaultSegmentParser, guess_parser_for_mime, guess_parser_for_system, iter_parsers, iter_known_segment_parsers, mime_parse_order
-from utils import to_numpy
+from . import errors
+from .ataridos import AtrHeader, AtariDosDiskImage, BootDiskImage, AtariDosFile, XexContainerSegment, get_xex, add_atr_header
+from .dos33 import Dos33DiskImage
+from .kboot import KBootImage, add_xexboot_header
+from .segments import SegmentData, SegmentSaver, DefaultSegment, EmptySegment, ObjSegment, RawSectorsSegment, SegmentedFileSegment, user_bit_mask, match_bit_mask, comment_bit_mask, data_style, selected_bit_mask, diff_bit_mask, not_user_bit_mask, interleave_segments, SegmentList, get_style_mask, get_style_bits
+from .spartados import SpartaDosDiskImage
+from .cartridge import A8CartHeader, AtariCartImage, RomImage
+from .parsers import SegmentParser, DefaultSegmentParser, guess_parser_by_size, guess_parser_for_mime, guess_parser_for_system, guess_container, iter_parsers, iter_known_segment_parsers, mime_parse_order, parsers_for_filename
+from .magic import guess_detail_for_mime
+from .utils import to_numpy, text_to_int
+from .dummy import LocalFilesystem
 
 
 def process(image, dirent, options):
@@ -32,68 +40,598 @@ def process(image, dirent, options):
             outfilename = "%s%s.XEX" % (dirent.filename, dirent.ext)
     if options.lower:
         outfilename = outfilename.lower()
-    
+
     if options.dry_run:
         action = "DRY_RUN: %s" % action
         skip = True
     if options.extract:
-        print "%s: %s %s" % (dirent, action, outfilename)
+        print("%s: %s %s" % (dirent, action, outfilename))
         if not skip:
             bytes = image.get_file(dirent)
             with open(outfilename, "wb") as fh:
                 fh.write(bytes)
     else:
-        print dirent
+        print(dirent)
+
+
+def find_diskimage_from_data(data, verbose=False):
+    data = to_numpy(data)
+    parser = None
+    container = guess_container(data, verbose)
+    if container is not None:
+        data = container.unpacked
+    rawdata = SegmentData(data)
+    mime, parser = guess_parser_by_size(rawdata)
+    if parser is None:
+        for mime in mime_parse_order:
+            if verbose:
+                print("Trying MIME type %s" % mime)
+            parser = guess_parser_for_mime(mime, rawdata, verbose)
+            if parser is None:
+                continue
+            if verbose:
+                print("Found parser %s" % parser.menu_name)
+            mime2 = guess_detail_for_mime(mime, rawdata, parser)
+            if mime != mime2:
+                mime = mime2
+                if verbose:
+                    print("Magic signature match: %s" % mime)
+            break
+    if parser is None:
+        raise errors.UnsupportedDiskImage("Unknown disk image type")
+    return parser, mime
+
+
+def find_diskimage(filename, verbose=False):
+    if filename == ".":
+        parser = LocalFilesystem()
+        mime = ""
+    else:
+        with open(filename, "rb") as fh:
+            if verbose:
+                print("Loading file %s" % filename)
+            data = to_numpy(fh.read())
+        parser, mime = find_diskimage_from_data(data, verbose)
+    parser.image.filename = filename
+    parser.image.ext = ""
+    return parser, mime
+
+
+def extract_files(image, files):
+    if options.all:
+        files = image.files
+    for name in files:
+        try:
+            dirent = image.find_dirent(name)
+        except errors.FileNotFound:
+            print("%s not in %s" % (name, image))
+            continue
+        output = dirent.filename
+        if options.lower:
+            output = output.lower()
+        if options.dir:
+            if not os.path.exists(options.dir):
+                os.makedirs(options.dir)
+            output = os.path.join(options.dir, output)
+        if not options.dry_run:
+            data = image.get_file(dirent)
+            if os.path.exists(output) and not options.force:
+                print("skipping %s, file exists. Use -f to overwrite" % output)
+                continue
+            print("extracting %s -> %s" % (name, output))
+            if options.text:
+                data = data.replace(b'\x7f', b'\t')
+                data = data.replace(b'\x9b', b'\n')
+            with open(output, "wb") as fh:
+                fh.write(data)
+        else:
+            print("extracting %s -> %s" % (name, output))
+
+
+def save_file(image, name, filetype, data):
+    try:
+        dirent = image.find_dirent(name)
+        if options.force:
+            image.delete_file(name)
+        else:
+            print("skipping %s, use -f to overwrite" % (name))
+            return False
+    except errors.FileNotFound:
+        pass
+    print("copying %s to %s" % (name, image.filename))
+    if not options.dry_run:
+        image.write_file(name, filetype, data)
+        return True
+    return False
+
+
+def add_files(image, files):
+    filetype = options.filetype
+    if not filetype:
+        filetype = image.default_filetype
+    changed = False
+    for name in files:
+        with open(name, "rb") as fh:
+            data = fh.read()
+        name = os.path.basename(name)
+        changed = save_file(image, name, filetype, data)
+    if changed:
+        image.save()
+
+
+def remove_files(image, files):
+    changed = False
+    for name in files:
+        try:
+            dirent = image.find_dirent(name)
+        except errors.FileNotFound:
+            print("%s not in %s" % (name, image))
+            continue
+        print("removing %s from %s" % (name, image))
+        if not options.dry_run:
+            image.delete_file(name)
+            changed = True
+    if changed:
+        image.save()
+
+
+def list_files(image, files, show_crc=False, show_metadata=False):
+    files = set(files)
+    for dirent in image.files:
+        if not files or dirent.filename in files:
+            if show_crc:
+                data = image.get_file(dirent)
+                crc = zlib.crc32(data) & 0xffffffff  # correct for some platforms that return signed int
+                extra = "  %08x" % crc
+            else:
+                extra = ""
+            print("%s%s" % (dirent, extra))
+            if show_metadata:
+                print(dirent.extra_metadata(image))
+
+
+def crc_files(image, files):
+    files = set(files)
+    for dirent in image.files:
+        if not files or dirent.filename in files:
+            data = image.get_file(dirent)
+            crc = zlib.crc32(data) & 0xffffffff  # correct for some platforms that return signed int
+            print("%s: %08x" % (dirent.filename, crc))
+
+
+def assemble_segments(source_files, data_files, obj_files, run_addr=""):
+    if source_files:
+        try:
+            import pyatasm
+        except ImportError:
+            raise errors.AtrError("Please install pyatasm to compile code.")
+    changed = False
+    segments = SegmentList()
+    for name in source_files:
+        try:
+            asm = pyatasm.Assemble(name)
+        except SyntaxError as e:
+            raise errors.AtrError("Assembly error: %s" % e.msg)
+        log.debug("Assembled %s into:" % name)
+        for first, last, object_code in asm.segments:
+            s = segments.add_segment(object_code, first)
+            log.debug("  %s" % s.name)
+            print("adding %s from %s assembly" % (s, name))
+    for name in data_files:
+        if "@" not in name:
+            raise errors.AtrError("Data files must include a load address specified with the @ char")
+        name, addr = name.rsplit("@", 1)
+        first = text_to_int(addr)
+        log.debug("Adding data file %s at $%04x" % (name, first))
+        subset = slice(0, sys.maxsize)
+        if "[" in name and "]" in name:
+            name, slicetext = name.rsplit("[", 1)
+            if ":" in slicetext:
+                start, end = slicetext.split(":", 1)
+                try:
+                    start = int(start)
+                except:
+                    start = 0
+                if end.endswith("]"):
+                    end = end[:-1]
+                try:
+                    end = int(end)
+                except:
+                    end = None
+                subset = slice(start, end)
+        with open(name, 'rb') as fh:
+            data = fh.read()[subset]
+            s = segments.add_segment(data, first)
+            log.debug("read data for %s" % s.name)
+    for name in obj_files:
+        try:
+            parser, _ = find_diskimage(name, options.verbose)
+        except errors.AtrError as e:
+            print(f"skipping {name}: {e}")
+        else:
+            for s in parser.segments:
+                if hasattr(s, 'run_address'):
+                    if not run_addr:
+                        run_addr = s.run_address()
+                    else:
+                        print(f"already have run address {run_addr}; skipping {s.run_address()}")
+                elif s.origin > 0:
+                    print(f"adding {s} from {name}")
+                    segments.add_segment(s.data, s.origin)
+    if options.verbose:
+        for s in segments:
+            print("%s - %04x)" % (str(s)[:-1], s.origin + len(s)))
+    if run_addr:
+        try:
+            run_addr = text_to_int(run_addr)
+        except (AttributeError, ValueError):
+            # not text, try as integer
+            try:
+                run_addr = int(run_addr)
+            except ValueError:
+                run_addr = None
+
+    return segments, run_addr
+
+def assemble(image, source_files, data_files, obj_files, run_addr=""):
+    segments, run_addr = assemble_segments(source_files, data_files, obj_files, run_addr)
+    file_data, filetype = image.create_executable_file_image(options.output, segments, run_addr)
+    print("total file size: $%x (%d) bytes" % (len(file_data), len(file_data)))
+    changed = save_file(image, options.output, filetype, file_data)
+    if changed:
+        image.save()
+
+
+def boot_image(image_name, source_files, data_files, obj_files, run_addr=""):
+    try:
+        image_cls = parsers_for_filename(image_name)[0]
+    except errors.InvalidDiskImage as e:
+        print("%s: %s" % (image_name, e))
+        return None
+    segments, run_addr = assemble_segments(source_files, data_files, obj_files, run_addr)
+    if segments:
+        image = image_cls.create_boot_image(segments, run_addr)
+        print("saving boot disk %s" % (image_name))
+        image.save(image_name)
+    else:
+        print("No segments to save to boot disk")
+
+
+def shred_image(image, value=0):
+    print("shredding: free sectors from %s filled with %d" % (image, value))
+    if not options.dry_run:
+        image.shred()
+        image.save()
+
+
+def get_template_path(rel_path="templates"):
+    path = __file__
+
+    template_path = os.path.normpath(os.path.join(os.path.dirname(path), rel_path))
+    frozen = getattr(sys, 'frozen', False)
+    if frozen:
+        if frozen == True:
+            # pyinstaller sets frozen=True and uses sys._MEIPASS
+            root = sys._MEIPASS
+            template_path = os.path.normpath(os.path.join(root, template_path))
+        elif frozen  == 'macosx_app':
+            #print "FROZEN!!! %s" % frozen
+            root = os.environ['RESOURCEPATH']
+            if ".zip/" in template_path:
+                zippath, template_path = template_path.split(".zip/")
+            template_path = os.path.normpath(os.path.join(root, template_path))
+        else:
+            print("App packager %s not yet supported for image paths!!!")
+    return template_path
+
+
+def get_template_images(partial=""):
+    import glob
+
+    path = get_template_path()
+    files = glob.glob(os.path.join(path, "*"))
+    templates = {}
+    for path in files:
+        name = os.path.basename(path)
+        if name.endswith(".inf"):
+            continue
+        if partial not in name:
+            continue
+        try:
+            with open(path + ".inf", "r") as fh:
+                s = fh.read()
+                try:
+                    j = json.loads(s)
+                except ValueError:
+                    continue
+                j['name'] = name
+                j['path'] = path
+                templates[name] = j
+        except IOError:
+            continue
+    return templates
+
+
+def get_template_info():
+    import textwrap
+    fmt = "  %-14s  %s"
+
+    templates = get_template_images()
+
+    lines = []
+    lines.append("available templates:")
+    for name in sorted(templates.keys()):
+        d = textwrap.wrap(templates[name]["description"], 80 - 1 - 14 - 2 - 2)
+        lines.append(fmt % (os.path.basename(name), d[0]))
+        lines.extend([fmt % ("", line) for line in d[1:]])
+    return os.linesep.join(lines) + os.linesep
+
+
+def get_template_data(template):
+    possibilities = get_template_images(template)
+    if not possibilities:
+        raise errors.InvalidDiskImage("Unknown template disk image %s" % template)
+    if len(possibilities) > 1:
+        raise errors.InvalidDiskImage("Name %s is ambiguous (%d matches: %s)" % (template, len(possibilities), ", ".join(sorted(possibilities.keys()))))
+    name, inf = possibilities.popitem()
+    path = inf['path']
+    try:
+        with open(path, "rb") as fh:
+            data = fh.read()
+    except IOError:
+        raise errors.InvalidDiskImage("Failed reading template file %s" % path)
+    return data, inf
+
+
+def create_image(template, name):
+    import textwrap
+
+    try:
+        data, inf = get_template_data(template)
+    except errors.InvalidDiskImage as e:
+        info = get_template_info()
+        print("Error: %s\n\n%s" % (e, info))
+        return
+    print("Using template %s:\n  %s" % (inf['name'], "\n  ".join(textwrap.wrap(inf["description"], 77))))
+    if not options.dry_run:
+        if os.path.exists(name) and not options.force:
+            print("skipping %s, use -f to overwrite" % (name))
+        else:
+            with open(name, "wb") as fh:
+                fh.write(data)
+            parser, _ = find_diskimage(name, options.verbose)
+            print("created %s: %s" % (name, str(parser.image)))
+            list_files(parser.image, [])
+    else:
+        print("creating %s" % name)
+
 
 def run():
-    import sys
     import argparse
-    
-    parser = argparse.ArgumentParser(description="Extract images off ATR format disks")
+
+    global options
+
+    # Subparser command aliasing from: https://gist.github.com/sampsyo/471779
+    # released into the public domain by its author
+    class AliasedSubParsersAction(argparse._SubParsersAction):
+        class _AliasedPseudoAction(argparse.Action):
+            def __init__(self, name, aliases, help):
+                dest = name
+                if aliases:
+                    dest += ' (%s)' % ','.join(aliases)
+                sup = super(AliasedSubParsersAction._AliasedPseudoAction, self)
+                sup.__init__(option_strings=[], dest=dest, help=help) 
+
+        def add_parser(self, name, **kwargs):
+            if 'aliases' in kwargs:
+                aliases = kwargs['aliases']
+                del kwargs['aliases']
+            else:
+                aliases = []
+
+            parser = super(AliasedSubParsersAction, self).add_parser(name, **kwargs)
+
+            # Make the aliases work.
+            for alias in aliases:
+                self._name_parser_map[alias] = parser
+            # Make the help text reflect them, first removing old help entry.
+            if 'help' in kwargs:
+                help = kwargs.pop('help')
+                self._choices_actions.pop()
+                pseudo_action = self._AliasedPseudoAction(name, aliases, help)
+                self._choices_actions.append(pseudo_action)
+
+            return parser
+
+    command_aliases = {
+        "list": ["t", "ls", "dir", "catalog"],
+        "crc": [],
+        "extract": ["x"],
+        "add": ["a"],
+        "create": ["c"],
+        "boot": ["b"],
+        "assemble": ["s", "asm"],
+        "delete": ["rm", "del"],
+        "vtoc": ["v"],
+        "segments": [],
+    }
+    # reverse aliases does the inverse mapping of command aliases, including
+    # the identity mapping of "command" to "command"
+    reverse_aliases = {z: k for k, v in command_aliases.items() for z in (v + [k])}
+
+    skip_diskimage_summary = set(["crc"])
+
+    usage = "%(prog)s [-h] [-v] [--dry-run] DISK_IMAGE [...]"
+    subparser_usage = "%(prog)s [-h] [-v] [--dry-run] DISK_IMAGE"
+
+    parser = argparse.ArgumentParser(prog="atrcopy DISK_IMAGE", description="Manipulate files on several types of 8-bit computer disk images. Type '%(prog)s COMMAND --help' for list of options available for each command.")
+    parser.register('action', 'parsers', AliasedSubParsersAction)
     parser.add_argument("-v", "--verbose", default=0, action="count")
-    parser.add_argument("-d", "--debug", action="store_true", default=False, help="debug the currently under-development parser")
-    parser.add_argument("-l", "--lower", action="store_true", default=False, help="convert filenames to lower case")
-    parser.add_argument("--dry-run", action="store_true", default=False, help="don't extract, just show what would have been extracted")
-    parser.add_argument("-n", "--no-sys", action="store_true", default=False, help="only extract things that look like games (no DOS or .SYS files)")
-    parser.add_argument("-x", "--extract", action="store_true", default=False, help="extract files")
-    parser.add_argument("--xex", action="store_true", default=False, help="add .xex extension")
-    parser.add_argument("-f", "--force", action="store_true", default=False, help="force operation on disk images that have bad directory entries or look like boot disks")
-    parser.add_argument("files", metavar="ATR", nargs="+", help="an ATR image file [or a list of them]")
-    parser.add_argument("-s", "--segments", action="store_true", default=False, help="display segments")
-    options, extra_args = parser.parse_known_args()
+    parser.add_argument("--dry-run", action="store_true", default=False, help="don't perform operation, just show what would have happened")
+
+    subparsers = parser.add_subparsers(dest='command', help='', metavar="COMMAND")
+
+    command = "list"
+    list_parser = subparsers.add_parser(command, help="List files on the disk image. This is the default if no command is specified", aliases=command_aliases[command])
+    list_parser.add_argument("-g", "--segments", action="store_true", default=False, help="display segments")
+    list_parser.add_argument("-m", "--metadata", action="store_true", default=False, help="show extra metadata for named files")
+    list_parser.add_argument("-c", "--crc", action="store_true", default=False, help="compute CRC32 for each file")
+    list_parser.add_argument("files", metavar="FILENAME", nargs="*", help="an optional list of files to display")
+
+    command = "crc"
+    crc_parser = subparsers.add_parser(command, help="List files on the disk image and the CRC32 value in format suitable for parsing", aliases=command_aliases[command])
+    crc_parser.add_argument("files", metavar="FILENAME", nargs="*", help="an optional list of files to display")
+
+    command = "extract"
+    extract_parser = subparsers.add_parser(command, help="Copy files from the disk image to the local filesystem", aliases=command_aliases[command])
+    extract_parser.add_argument("-a", "--all", action="store_true", default=False, help="operate on all files on disk image")
+    extract_parser.add_argument("-l", "--lower", action="store_true", default=False, help="convert extracted filenames to lower case")
+    #extract_parser.add_argument("-n", "--no-sys", action="store_true", default=False, help="only extract things that look like games (no DOS or .SYS files)")
+    extract_parser.add_argument("-e", "--ext", action="store", nargs=1, default=False, help="add the specified extension")
+    extract_parser.add_argument("-d", "--dir", action="store", default=False, help="extract to the specified directory")
+    extract_parser.add_argument("-t", "--text", action="store_true", default=False, help="convert text files to unix-style text files")
+    extract_parser.add_argument("-f", "--force", action="store_true", default=False, help="allow file overwrites on local filesystem")
+    extract_parser.add_argument("files", metavar="FILENAME", nargs="*", help="if not using the -a/--all option, a file (or list of files) to extract from the disk image.")
+
+    command = "add"
+    add_parser = subparsers.add_parser(command, help="Add files to the disk image", aliases=command_aliases[command])
+    add_parser.add_argument("-f", "--force", action="store_true", default=False, help="allow file overwrites in the disk image")
+    add_parser.add_argument("-t", "--filetype", action="store", default="", help="file type metadata for writing to disk images that require it (e.g. DOS 3.3)")
+    add_parser.add_argument("files", metavar="FILENAME", nargs="+", help="a file (or list of files) to copy to the disk image")
+
+    command = "create"
+    create_parser = subparsers.add_parser(command, help="Create a new disk image", aliases=command_aliases[command], epilog="<generated on demand to list available templates>", formatter_class=argparse.RawDescriptionHelpFormatter)
+    create_parser.add_argument("-f", "--force", action="store_true", default=False, help="replace disk image file if it exists")
+    create_parser.add_argument("template", metavar="TEMPLATE", nargs=1, help="template to use to create new disk image; see below for list of available built-in templates")
+
+    command = "assemble"
+    assembly_parser = subparsers.add_parser(command, help="Create a new binary file in the disk image", aliases=command_aliases[command])
+    assembly_parser.add_argument("-f", "--force", action="store_true", default=False, help="allow file overwrites in the disk image")
+    assembly_parser.add_argument("-s", "--asm", nargs="*", action="append", help="source file(s) to assemble using pyatasm")
+    assembly_parser.add_argument("-d","--data", nargs="*", action="append", help="binary data file(s) to add to assembly, specify as file@addr. Only a portion of the file may be included; specify the subset using standard python slice notation: file[subset]@addr")
+    assembly_parser.add_argument("-b", "--obj", "--bload", nargs="*", action="append", help="binary file(s) to add to assembly, either executables or labeled memory dumps (e.g. BSAVE on Apple ][), parsing each file's binary segments to add to the resulting disk image at the load address for each segment")
+    assembly_parser.add_argument("-r", "--run-addr", "--brun", action="store", default="", help="run address of binary file if not the first byte of the first segment")
+    assembly_parser.add_argument("-o", "--output", action="store", default="", required=True, help="output file name in disk image")
+
+    command = "boot"
+    boot_parser = subparsers.add_parser(command, help="Create a bootable disk image", aliases=command_aliases[command])
+    boot_parser.add_argument("-f", "--force", action="store_true", default=False, help="allow file overwrites in the disk image")
+    boot_parser.add_argument("-s", "--asm", nargs="*", action="append", help="source file(s) to assemble using pyatasm")
+    boot_parser.add_argument("-d","--data", nargs="*", action="append", help="binary data file(s) to add to assembly, specify as file@addr. Only a portion of the file may be included; specify the subset using standard python slice notation: file[subset]@addr")
+    boot_parser.add_argument("-b", "--obj", "--bload", nargs="*", action="append", help="binary file(s) to add to assembly, either executables or labeled memory dumps (e.g. BSAVE on Apple ][), parsing each file's binary segments to add to the resulting disk image at the load address for each segment")
+    boot_parser.add_argument("-r", "--run-addr", "--brun", action="store", default="", help="run address of binary file if not the first byte of the first segment")
+
+    command = "delete"
+    delete_parser = subparsers.add_parser(command, help="Delete files from the disk image", aliases=command_aliases[command])
+    delete_parser.add_argument("-f", "--force", action="store_true", default=False, help="remove the file even if it is write protected ('locked' in Atari DOS 2 terms), if write-protect is supported disk image")
+    delete_parser.add_argument("files", metavar="FILENAME", nargs="+", help="a file (or list of files) to remove from the disk image")
+
+    command = "vtoc"
+    vtoc_parser = subparsers.add_parser(command, help="Show a formatted display of sectors free in the disk image", aliases=command_aliases[command])
+    vtoc_parser.add_argument("-e", "--clear-empty", action="store_true", default=False, help="fill empty sectors with 0")
+
+    command = "segments"
+    vtoc_parser = subparsers.add_parser(command, help="Show the list of parsed segments in the disk image", aliases=command_aliases[command])
+
+
+    # argparse doesn't seem to allow an argument fixed to item 1, so have to
+    # hack with the arg list to get arg #1 to be the disk image. Because of
+    # this hack, we have to perform an additional hack to figure out what the
+    # --help option applies to if it's in the argument list.
+    args = list(sys.argv[1:])
+    if len(args) > 0:
+        found_help = -1
+        first_non_dash = 0
+        num_non_dash = 0
+        non_dash = []
+        for i, arg in enumerate(args):
+            if arg.startswith("-"):
+                if i == 0:
+                    first_non_dash = -1
+                if arg =="-h" or arg == "--help":
+                    found_help = i
+            else:
+                num_non_dash += 1
+                non_dash.append(arg)
+                if first_non_dash < 0:
+                    first_non_dash = i
+        if found_help >= 0 or first_non_dash < 0:
+            if found_help == 0 or first_non_dash < 0:
+                # put dummy argument so help for entire script will be shown
+                args = ["--help"]
+            elif non_dash[0] in reverse_aliases:
+                # if the first argument without a leading dash looks like a
+                # command instead of a disk image, show help for that command
+                args = [non_dash[0], "--help"]
+            elif len(non_dash) > 0 and non_dash[1] in reverse_aliases:
+                # if the first argument without a leading dash looks like a
+                # command instead of a disk image, show help for that command
+                args = [non_dash[1], "--help"]
+            else:
+                # show script help
+                args = ["--help"]
+            if reverse_aliases.get(args[0], None) == "create":
+                create_parser.epilog = get_template_info()
+        else:
+            # Allow global options to come before or after disk image name
+            disk_image_name = args[first_non_dash]
+            args[first_non_dash:first_non_dash + 1] = []
+            if num_non_dash == 1:
+                # If there is only a disk image but no command specified,
+                # use the default
+                args.append('list')
+    else:
+        disk_image_name = None
+        parser.print_help()
+        sys.exit(1)
+
+    # print "parsing: %s" % str(args)
+    options = parser.parse_args(args)
+    # print options
+    command = reverse_aliases[options.command]
 
     # Turn off debug messages by default
+    logging.basicConfig(level=logging.WARNING)
     log = logging.getLogger("atrcopy")
     if options.verbose:
         log.setLevel(logging.DEBUG)
     else:
         log.setLevel(logging.INFO)
-    
-    for filename in options.files:
-        with open(filename, "rb") as fh:
-            if options.verbose:
-                print "Loading file %s" % filename
-            rawdata = SegmentData(fh.read())
-            parser = None
-            for mime in mime_parse_order:
-                if options.verbose:
-                    print "Trying MIME type %s" % mime
-                parser = guess_parser_for_mime(mime, rawdata)
-                if parser is None:
-                    continue
-                if options.verbose:
-                    print "Found parser %s" % parser.menu_name
-                print "%s: %s" % (filename, parser.image)
-                if options.segments:
-                    print "\n".join([str(a) for a in parser.segments])
-                elif parser.image.files or options.force:
-                    for dirent in parser.image.files:
-                        try:
-                            process(parser.image, dirent, options)
-                        except FileNumberMismatchError164:
-                            print "Error 164: %s" % str(dirent)
-                        except ByteNotInFile166:
-                            print "Invalid sector for: %s" % str(dirent)
-                break
-            if parser is None:
-                print "%s: Unknown file type" % filename
+
+    if command == "create":
+        create_image(options.template[0], disk_image_name)
+    elif command == "boot":
+        asm = options.asm[0] if options.asm else []
+        data = options.data[0] if options.data else []
+        obj = options.obj[0] if options.obj else []
+        boot_image(disk_image_name, asm, data, obj, options.run_addr)
+    else:
+        try:
+            parser, mime = find_diskimage(disk_image_name, options.verbose)
+        except (errors.UnsupportedContainer, errors.UnsupportedDiskImage, IOError) as e:
+            print(f"{disk_image_name}: {e}")
+        else:
+            if command not in skip_diskimage_summary:
+                print(f"{disk_image_name}: {parser.image}{' (%s}' % mime if mime and options.verbose else ''}")
+            if command == "vtoc":
+                vtoc = parser.image.get_vtoc_object()
+                print(vtoc)
+                if options.clear_empty:
+                    shred_image(parser.image)
+            elif command == "list":
+                list_files(parser.image, options.files, options.crc, options.metadata)
+            elif command == "crc":
+                crc_files(parser.image, options.files)
+            elif command == "add":
+                add_files(parser.image, options.files)
+            elif command == "delete":
+                remove_files(parser.image, options.files)
+            elif command == "extract":
+                extract_files(parser.image, options.files)
+            elif command == "assemble":
+                asm = options.asm[0] if options.asm else []
+                data = options.data[0] if options.data else []
+                obj = options.obj[0] if options.obj else []
+                assemble(parser.image, asm, data, obj, options.run_addr)
+            elif command == "segments":
+                print("\n".join([str(a) for a in parser.segments]))

atrcopy/_metadata.py

@@ -0,0 +1,4 @@
+__author__ = "Rob McMullen"
+__author_email__ = "feedback@playermissile.com"
+__url__ = "https://github.com/robmcmullen/atrcopy"
+__bug_report_url__ = "https://github.com/robmcmullen/atrcopy/issues"

atrcopy/_version.py

@@ -0,0 +1 @@
+__version__ = "10.1"

atrcopy/ataridos.py

@@ -1,15 +1,73 @@
 import numpy as np
 
-from errors import *
-from diskimages import DiskImageBase
-from segments import EmptySegment, ObjSegment, RawSectorsSegment, DefaultSegment, SegmentSaver
-from utils import to_numpy
+from . import errors
+from .diskimages import DiskImageBase, BaseHeader, Bootable
+from .segments import SegmentData, EmptySegment, ObjSegment, RawSectorsSegment, DefaultSegment, SegmentedFileSegment, SegmentSaver, get_style_bits
+from .utils import *
+from .executables import get_xex
 
 import logging
 log = logging.getLogger(__name__)
+try:  # Expensive debugging
+    _xd = _expensive_debugging
+except NameError:
+    _xd = False
 
 
-class AtariDosDirent(object):
+class AtariDosWriteableSector(WriteableSector):
+    @property
+    def next_sector_num(self):
+        return self._next_sector_num
+
+    @next_sector_num.setter
+    def next_sector_num(self, value):
+        self._next_sector_num = value
+        index = self.sector_size - 3
+        hi, lo = divmod(value, 256)
+        self.data[index + 0] = (self.file_num << 2) | (hi & 0x03)
+        self.data[index + 1] = lo
+        self.data[index + 2] = self.used
+        if _xd: log.debug("sector metadata for %d: %s" % (self._sector_num, self.data[index:index + 3]))
+        # file number will be added later when known.
+
+
+class AtariDosVTOC(VTOC):
+    def parse_segments(self, segments):
+        self.vtoc1 = segments[0].data
+        bits = np.unpackbits(self.vtoc1[0x0a:0x64])
+        self.sector_map[0:720] = bits
+        if _xd: log.debug("vtoc before:\n%s" % str(self))
+
+    def calc_bitmap(self):
+        if _xd: log.debug("vtoc after:\n%s" % str(self))
+        packed = np.packbits(self.sector_map[0:720])
+        self.vtoc1[0x0a:0x64] = packed
+        s = WriteableSector(self.sector_size, self.vtoc1)
+        s.sector_num = 360
+        self.sectors.append(s)
+
+
+class AtariDosDirectory(Directory):
+    @property
+    def dirent_class(self):
+        return AtariDosDirent
+
+    def encode_empty(self):
+        return np.zeros([16], dtype=np.uint8)
+
+    def encode_dirent(self, dirent):
+        data = dirent.encode_dirent()
+        if _xd: log.debug("encoded dirent: %s" % data)
+        return data
+
+    def set_sector_numbers(self, image):
+        num = 361
+        for sector in self.sectors:
+            sector.sector_num = num
+            num += 1
+
+
+class AtariDosDirent(Dirent):
     # ATR Dirent structure described at http://atari.kensclassics.org/dos.htm
     format = np.dtype([
         ('FLAG', 'u1'),
@@ -20,7 +78,7 @@ class AtariDosDirent(object):
         ])
 
     def __init__(self, image, file_num=0, bytes=None):
-        self.file_num = file_num
+        Dirent.__init__(self, file_num)
         self.flag = 0
         self.opened_output = False
         self.dos_2 = False
@@ -31,18 +89,26 @@ class AtariDosDirent(object):
         self.deleted = False
         self.num_sectors = 0
         self.starting_sector = 0
-        self.filename = ""
-        self.ext = ""
+        self.basename = b''
+        self.ext = b''
         self.is_sane = True
         self.current_sector = 0
         self.current_read = 0
         self.sectors_seen = None
         self.parse_raw_dirent(image, bytes)
-    
+
     def __str__(self):
-        flags = self.summary()
-        return "File #%-2d (%s) %03d %-8s%-3s  %03d" % (self.file_num, flags, self.starting_sector, self.filename, self.ext, self.num_sectors)
-    
+        return "File #%-2d (%s) %03d %-8s%-3s  %03d" % (self.file_num, self.summary, self.starting_sector, self.basename.decode("latin1"), self.ext.decode("latin1"), self.num_sectors)
+
+    def __eq__(self, other):
+        return self.__class__ == other.__class__ and self.filename == other.filename and self.starting_sector == other.starting_sector and self.num_sectors == other.num_sectors
+
+    @property
+    def filename(self):
+        ext = (b'.' + self.ext) if self.ext else b''
+        return (self.basename + ext).decode('latin1')
+
+    @property
     def summary(self):
         output = "o" if self.opened_output else "."
         dos2 = "2" if self.dos_2 else "."
@@ -52,7 +118,7 @@ class AtariDosDirent(object):
         locked = "*" if self.locked else " "
         flags = "%s%s%s%s%s%s" % (output, dos2, mydos, in_use, deleted, locked)
         return flags
-    
+
     @property
     def verbose_info(self):
         flags = []
@@ -63,11 +129,14 @@ class AtariDosDirent(object):
         if self.deleted: flags.append("DEL")
         if self.locked: flags.append("LOCK")
         return "flags=[%s]" % ", ".join(flags)
-    
-    def parse_raw_dirent(self, image, bytes):
-        if bytes is None:
+
+    def extra_metadata(self, image):
+        return self.verbose_info
+
+    def parse_raw_dirent(self, image, data):
+        if data is None:
             return
-        values = bytes.view(dtype=self.format)[0]  
+        values = data.view(dtype=self.format)[0]
         flag = values[0]
         self.flag = flag
         self.opened_output = (flag&0x01) > 0
@@ -79,10 +148,34 @@ class AtariDosDirent(object):
         self.deleted = (flag&0x80) > 0
         self.num_sectors = int(values[1])
         self.starting_sector = int(values[2])
-        self.filename = str(values[3]).rstrip()
-        self.ext = str(values[4]).rstrip()
+        self.basename = bytes(values[3]).rstrip()
+        self.ext = bytes(values[4]).rstrip()
         self.is_sane = self.sanity_check(image)
-    
+
+    def encode_dirent(self):
+        data = np.zeros([self.format.itemsize], dtype=np.uint8)
+        values = data.view(dtype=self.format)[0]
+        flag = (1 * int(self.opened_output)) | (2 * int(self.dos_2)) | (4 * int(self.mydos)) | (0x10 * int(self.is_dir)) | (0x20 * int(self.locked)) | (0x40 * int(self.in_use)) | (0x80 * int(self.deleted))
+        values[0] = flag
+        values[1] = self.num_sectors
+        values[2] = self.starting_sector
+        values[3] = self.basename
+        values[4] = self.ext
+        return data
+
+    def mark_deleted(self):
+        self.deleted = True
+        self.in_use = False
+
+    def update_sector_info(self, sector_list):
+        self.num_sectors = sector_list.num_sectors
+        self.starting_sector = sector_list.first_sector
+
+    def add_metadata_sectors(self, vtoc, sector_list, header):
+        # no extra sectors are needed for an Atari DOS file; the links to the
+        # next sector is contained in the sector.
+        pass
+
     def sanity_check(self, image):
         if not self.in_use:
             return True
@@ -91,14 +184,25 @@ class AtariDosDirent(object):
         if self.num_sectors < 0 or self.num_sectors > image.header.max_sectors:
             return False
         return True
-    
+
+    def get_sectors_in_vtoc(self, image):
+        sector_list = BaseSectorList(image.header)
+        self.start_read(image)
+        while True:
+            sector = WriteableSector(image.header.sector_size, None, self.current_sector)
+            sector_list.append(sector)
+            _, last, _, _ = self.read_sector(image)
+            if last:
+                break
+        return sector_list
+
     def start_read(self, image):
         if not self.is_sane:
-            raise InvalidDirent("Invalid directory entry '%s'" % str(self))
+            raise errors.InvalidDirent("Invalid directory entry '%s'" % str(self))
         self.current_sector = self.starting_sector
         self.current_read = self.num_sectors
         self.sectors_seen = set()
-    
+
     def read_sector(self, image):
         raw, pos, size = image.get_raw_bytes(self.current_sector)
         bytes, num_data_bytes = self.process_raw_sector(image, raw)
@@ -107,18 +211,28 @@ class AtariDosDirent(object):
     def process_raw_sector(self, image, raw):
         file_num = raw[-3] >> 2
         if file_num != self.file_num:
-            raise FileNumberMismatchError164()
+            raise errors.FileNumberMismatchError164("Expecting file %d, found %d" % (self.file_num, file_num))
         self.sectors_seen.add(self.current_sector)
         next_sector = ((raw[-3] & 0x3) << 8) + raw[-2]
         if next_sector in self.sectors_seen:
-            raise InvalidFile("Bad sector pointer data: attempting to reread sector %d" % next_sector)
+            raise errors.InvalidFile("Bad sector pointer data: attempting to reread sector %d" % next_sector)
         self.current_sector = next_sector
         num_bytes = raw[-1]
         return raw[0:num_bytes], num_bytes
-    
-    def get_filename(self):
-        ext = ("." + self.ext) if self.ext else ""
-        return self.filename + ext
+
+    def set_values(self, filename, filetype, index):
+        if type(filename) is not bytes:
+            filename = filename.encode("latin1")
+        if b'.' in filename:
+            filename, ext = filename.split(b'.', 1)
+        else:
+            ext = b'   '
+        self.basename = b'%-8s' % filename[0:8]
+        self.ext = ext
+        self.file_num = index
+        self.dos_2 = True
+        self.in_use = True
+        if _xd: log.debug("set_values: %s" % self)
 
 
 class MydosDirent(AtariDosDirent):
@@ -135,22 +249,37 @@ class XexSegmentSaver(SegmentSaver):
     export_extensions = [".xex"]
 
 
+class XexContainerSegment(DefaultSegment):
+    can_resize_default = True
+
+
 class XexSegment(ObjSegment):
     savers = [SegmentSaver, XexSegmentSaver]
 
 
-class AtariDosFile(object):
+class RunAddressSegment(ObjSegment):
+    # FIXME: defining run_address as a property doesn't work for some reason.
+    # @property
+    # def run_address(self):
+    #     return self.rawdata[0:2].view(dtype="<u2")[0]
+    def run_address(self):
+        return self.rawdata[0:2].data.view(dtype="<u2")[0]
+
+
+
+class AtariDosFile(Bootable):
     """Parse a binary chunk into segments according to the Atari DOS object
     file format.
     
     Ref: http://www.atarimax.com/jindroush.atari.org/afmtexe.html
     """
+
     def __init__(self, rawdata):
         self.rawdata = rawdata
         self.size = len(rawdata)
         self.segments = []
         self.files = []
-    
+
     def __str__(self):
         return "\n".join(str(s) for s in self.segments) + "\n"
 
@@ -159,13 +288,15 @@ class AtariDosFile(object):
 
     def relaxed_check(self):
         pass
-    
+
     def parse_segments(self):
         r = self.rawdata
         b = r.get_data()
+        s = r.get_style()
         pos = 0
+        style_pos = 0
         first = True
-        log.debug("Initial parsing: size=%d" % self.size)
+        if _xd: log.debug("Initial parsing: size=%d" % self.size)
         while pos < self.size:
             if pos + 1 < self.size:
                 header, = b[pos:pos+2].view(dtype='<u2')
@@ -179,50 +310,215 @@ class AtariDosFile(object):
                 first = False
                 continue
             elif first:
-                raise InvalidBinaryFile
-            log.debug("header parsing: header=0x%x" % header)
+                raise errors.InvalidBinaryFile("Object file doesn't start with 0xffff")
+            if _xd: log.debug("header parsing: header=0x%x" % header)
             if len(b[pos:pos + 4]) < 4:
                 self.segments.append(ObjSegment(r[pos:pos + 4], 0, 0, 0, len(b[pos:pos + 4]), "Short Segment Header"))
                 break
             start, end = b[pos:pos + 4].view(dtype='<u2')
+            s[style_pos:pos + 4] = get_style_bits(data=True)
             if end < start:
-                raise InvalidBinaryFile
+                raise errors.InvalidBinaryFile("Nonsensical start and end addresses")
             count = end - start + 1
             found = len(b[pos + 4:pos + 4 + count])
             if found < count:
                 self.segments.append(ObjSegment(r[pos + 4:pos + 4 + count], pos, pos + 4, start, end, "Incomplete Data"))
                 break
-            self.segments.append(ObjSegment(r[pos + 4:pos + 4 + count], pos, pos + 4, start, end))
+            if start == 0x2e0:
+                segment_cls = RunAddressSegment
+            else:
+                segment_cls = ObjSegment
+            self.segments.append(segment_cls(r[pos + 4:pos + 4 + count], pos, pos + 4, start, end))
             pos += 4 + count
+            style_pos = pos
+
+
+class AtrHeader(BaseHeader):
+    sector_class = AtariDosWriteableSector
+
+    # ATR Format described in http://www.atarimax.com/jindroush.atari.org/afmtatr.html
+    format = np.dtype([
+        ('wMagic', '<u2'),
+        ('wPars', '<u2'),
+        ('wSecSize', '<u2'),
+        ('btParsHigh', 'u1'),
+        ('dwCRC','<u4'),
+        ('unused','<u4'),
+        ('btFlags','u1'),
+        ])
+    file_format = "ATR"
+
+    def __init__(self, bytes=None, sector_size=128, initial_sectors=3, create=False):
+        BaseHeader.__init__(self, sector_size, initial_sectors, 360, 1)
+        if create:
+            self.header_offset = 16
+            self.check_size(0)
+        if bytes is None:
+            return
+
+        if len(bytes) == 16:
+            values = bytes.view(dtype=self.format)[0]
+            if values[0] != 0x296:
+                raise errors.InvalidAtrHeader("no ATR header magic value")
+            self.image_size = (int(values[3]) * 256 * 256 + int(values[1])) * 16
+            self.sector_size = int(values[2])
+            self.crc = int(values[4])
+            self.unused = int(values[5])
+            self.flags = int(values[6])
+            self.header_offset = 16
+        else:
+            raise errors.InvalidAtrHeader("incorrect AHC header size of %d" % len(bytes))
+
+    def __str__(self):
+        return "%s Disk Image (size=%d (%dx%dB), crc=%d flags=%d unused=%d)" % (self.file_format, self.image_size, self.max_sectors, self.sector_size, self.crc, self.flags, self.unused)
+
+    def encode(self, raw):
+        values = raw.view(dtype=self.format)[0]
+        values[0] = 0x296
+        paragraphs = self.image_size // 16
+        parshigh, pars = divmod(paragraphs, 256*256)
+        values[1] = pars
+        values[2] = self.sector_size
+        values[3] = parshigh
+        values[4] = self.crc
+        values[5] = self.unused
+        values[6] = self.flags
+        return raw
+
+    def check_size(self, size):
+        if size == 92160 or size == 92176:
+            self.image_size = 92160
+            self.sector_size = 128
+            self.initial_sector_size = 0
+            self.num_initial_sectors = 0
+        elif size == 184320 or size == 184336:
+            self.image_size = 184320
+            self.sector_size = 256
+            self.initial_sector_size = 0
+            self.num_initial_sectors = 0
+        elif size == 183936 or size == 183952:
+            self.image_size = 183936
+            self.sector_size = 256
+            self.initial_sector_size = 128
+            self.num_initial_sectors = 3
+        else:
+            self.image_size = size
+        self.first_vtoc = 360
+        self.num_vtoc = 1
+        self.first_directory = 361
+        self.num_directory = 8
+        self.tracks_per_disk = 40
+        self.sectors_per_track = 18
+        self.payload_bytes = self.sector_size - 3
+        initial_bytes = self.initial_sector_size * self.num_initial_sectors
+        self.max_sectors = ((self.image_size - initial_bytes) // self.sector_size) + self.num_initial_sectors
+
+    def get_pos(self, sector):
+        if not self.sector_is_valid(sector):
+            raise errors.ByteNotInFile166("Sector %d out of range" % sector)
+        if sector <= self.num_initial_sectors:
+            pos = self.num_initial_sectors * (sector - 1)
+            size = self.initial_sector_size
+        else:
+            pos = self.num_initial_sectors * self.initial_sector_size + (sector - 1 - self.num_initial_sectors) * self.sector_size
+            size = self.sector_size
+        pos += self.header_offset
+        return pos, size
+
+    def strict_check(self, image):
+        size = len(image)
+        if self.header_offset == 16 or size in [92176, 133136, 184336, 183952]:
+            return
+        raise errors.InvalidDiskImage("Uncommon size of ATR file")
+
+
+class XfdHeader(AtrHeader):
+    file_format = "XFD"
+
+    def __str__(self):
+        return "%s Disk Image (size=%d (%dx%dB)" % (self.file_format, self.image_size, self.max_sectors, self.sector_size)
+
+    def __len__(self):
+        return 0
+
+    def to_array(self):
+        raw = np.zeros([0], dtype=np.uint8)
+        return raw
+
+    def strict_check(self, image):
+        size = len(image)
+        if size in [92160, 133120, 183936, 184320]:
+            return
+        raise errors.InvalidDiskImage("Uncommon size of XFD file")
 
 
 class AtariDosDiskImage(DiskImageBase):
+    default_executable_extension = "XEX"
+
     def __init__(self, *args, **kwargs):
         self.first_vtoc = 360
         self.num_vtoc = 1
         self.vtoc2 = 0
         self.first_data_after_vtoc = 369
         DiskImageBase.__init__(self, *args, **kwargs)
-    
+
+    @property
+    def writeable_sector_class(self):
+        return AtariDosWriteableSector
+
+    @property
+    def vtoc_class(self):
+        return AtariDosVTOC
+
+    @property
+    def directory_class(self):
+        return AtariDosDirectory
+
     def __str__(self):
         return "%s Atari DOS Format: %d usable sectors (%d free), %d files" % (self.header, self.total_sectors, self.unused_sectors, len(self.files))
-    
+
+    @classmethod
+    def new_header(cls, diskimage, format="ATR"):
+        if format.lower() == "atr":
+            header = AtrHeader(create=True)
+            header.check_size(diskimage.size)
+        else:
+            raise RuntimeError("Unknown header type %s" % format)
+        return header
+
+    def as_new_format(self, format="ATR"):
+        """ Create a new disk image in the specified format
+        """
+        first_data = len(self.header)
+        raw = self.rawdata[first_data:]
+        data = add_atr_header(raw)
+        newraw = SegmentData(data)
+        image = self.__class__(newraw)
+        return image
+
     vtoc_type = np.dtype([
         ('code', 'u1'),
         ('total','<u2'),
         ('unused','<u2'),
         ])
 
+    def read_header(self):
+        bytes = self.bytes[0:16]
+        try:
+            self.header = AtrHeader(bytes)
+        except errors.InvalidAtrHeader:
+            self.header = XfdHeader()
+
     def calc_vtoc_code(self):
         # From AA post: http://atariage.com/forums/topic/179868-mydos-vtoc-size/
-        num = 1 + (self.total_sectors + 80) / (self.header.sector_size * 8)
+        num = 1 + (self.total_sectors + 80) // (self.header.sector_size * 8)
         if self.header.sector_size == 128:
             if num == 1:
                 code = 2
             else:
                 if num & 1:
                     num += 1
-                code = ((num + 1) / 2) + 2
+                code = ((num + 1) // 2) + 2
         else:
             if self.total_sectors < 1024:
                 code = 2
@@ -242,17 +538,18 @@ class AtariDosDiskImage(DiskImageBase):
         self.assert_valid_sector(self.first_vtoc)
         self.num_vtoc = num
         if num < 0 or num > self.calc_vtoc_code():
-            raise InvalidDiskImage("Invalid number of VTOC sectors: %d" % num)
-        
+            raise errors.InvalidDiskImage("Invalid number of VTOC sectors: %d" % num)
+
         self.total_sectors = values[1]
         self.unused_sectors = values[2]
         if self.header.image_size == 133120:
             # enhanced density has 2nd VTOC
             self.vtoc2 = 1024
-            extra_free = self.get_sectors(self.vtoc2)[122:124].view(dtype='<u2')[0]
+            data, style = self.get_sectors(self.vtoc2)
+            extra_free = data[122:124].view(dtype='<u2')[0]
             self.unused_sectors += extra_free
-    
-    def get_directory(self):
+
+    def get_directory(self, directory=None):
         dir_bytes, style = self.get_sectors(361, 368)
         i = 0
         num = 0
@@ -261,17 +558,20 @@ class AtariDosDiskImage(DiskImageBase):
             dirent = AtariDosDirent(self, num, dir_bytes[i:i+16])
             if dirent.mydos:
                 dirent = MydosDirent(self, num, dir_bytes[i:i+16])
-            
+
             if dirent.in_use:
                 files.append(dirent)
                 if not dirent.is_sane:
                     self.all_sane = False
+                    log.debug("dirent %d not sane: %s" % (num, dirent))
             elif dirent.flag == 0:
                 break
+            if directory is not None:
+                directory.set(num, dirent)
             i += 16
             num += 1
         self.files = files
-    
+
     boot_record_type = np.dtype([
         ('BFLAG', 'u1'),
         ('BRCNT', 'u1'),
@@ -291,7 +591,7 @@ class AtariDosDiskImage(DiskImageBase):
 
     def get_boot_segments(self):
         data, style = self.get_sectors(360)
-        values = data[0:20].view(dtype=self.boot_record_type)[0]  
+        values = data[0:20].view(dtype=self.boot_record_type)[0]
         flag = int(values[0])
         segments = []
         if flag == 0:
@@ -304,46 +604,68 @@ class AtariDosDiskImage(DiskImageBase):
             code = ObjSegment(r[20:], 0, 0, addr + 20, addr + len(r), name="Boot Code")
             segments = [sectors, header, code]
         return segments
-    
+
     def get_vtoc_segments(self):
         r = self.rawdata
         segments = []
         addr = 0
         start, count = self.get_contiguous_sectors(self.first_vtoc, self.num_vtoc)
         segment = RawSectorsSegment(r[start:start+count], self.first_vtoc, self.num_vtoc, count, 128, 3, self.header.sector_size, name="VTOC")
+        segment.style[:] = get_style_bits(data=True)
+        segment.set_comment_at(0x00, "Type code")
+        segment.set_comment_at(0x01, "Total number of sectors")
+        segment.set_comment_at(0x03, "Number of free sectors")
+        segment.set_comment_at(0x05, "reserved")
+        segment.set_comment_at(0x06, "unused")
+        segment.set_comment_at(0x0a, "Sector bit map")
+        segment.set_comment_at(0x64, "unused")
         segments.append(segment)
         if self.vtoc2 > 0:
             start, count = self.get_contiguous_sectors(self.vtoc2, 1)
-            segment = RawSectorsSegment(r[start:start+count], self.vtoc2, 1, count, self.header.sector_size, name="VTOC2")
+            segment = RawSectorsSegment(r[start:start+count], self.vtoc2, 1, count, 128, 3, self.header.sector_size, name="VTOC2")
+            segment.style[:] = get_style_bits(data=True)
+            segment.set_comment_at(0x00, "Repeat of sectors 48-719")
+            segment.set_comment_at(0x44, "Sector bit map 720-1023")
+            segment.set_comment_at(0x7a, "Number of free sectors above 720")
+            segment.set_comment_at(0x7c, "unused")
             segments.append(segment)
         return segments
-    
+
     def get_directory_segments(self):
         r = self.rawdata
         segments = []
         addr = 0
         start, count = self.get_contiguous_sectors(361, 8)
         segment = RawSectorsSegment(r[start:start+count], 361, 8, count, 128, 3, self.header.sector_size, name="Directory")
+        segment.style[:] = get_style_bits(data=True)
+        index = 0
+        for filenum in range(64):
+            segment.set_comment_at(index + 0x00, "FILE #%d: Flag" % filenum)
+            segment.set_comment_at(index + 0x01, "FILE #%d: Number of sectors in file" % filenum)
+            segment.set_comment_at(index + 0x03, "FILE #%d: Starting sector number" % filenum)
+            segment.set_comment_at(index + 0x05, "FILE #%d: Filename" % filenum)
+            segment.set_comment_at(index + 0x0d, "FILE #%d: Extension" % filenum)
+            index += 16
         segments.append(segment)
         return segments
-    
+
     def get_file_segment(self, dirent):
         byte_order = []
         dirent.start_read(self)
         while True:
             bytes, last, pos, size = dirent.read_sector(self)
-            byte_order.extend(range(pos, pos + size))
+            byte_order.extend(list(range(pos, pos + size)))
             if last:
                 break
         if len(byte_order) > 0:
-            name = "%s %ds@%d" % (dirent.get_filename(), dirent.num_sectors, dirent.starting_sector)
-            verbose_name = "%s (%d sectors, first@%d) %s" % (dirent.get_filename(), dirent.num_sectors, dirent.starting_sector, dirent.verbose_info)
+            name = "%s %ds@%d" % (dirent.filename, dirent.num_sectors, dirent.starting_sector)
+            verbose_name = "%s (%d sectors, first@%d) %s" % (dirent.filename, dirent.num_sectors, dirent.starting_sector, dirent.verbose_info)
             raw = self.rawdata.get_indexed(byte_order)
             segment = DefaultSegment(raw, name=name, verbose_name=verbose_name)
         else:
-            segment = EmptySegment(self.rawdata, name=dirent.get_filename())
+            segment = EmptySegment(self.rawdata, name=dirent.filename)
         return segment
-    
+
     def get_file_segments(self):
         segments_in = DiskImageBase.get_file_segments(self)
         segments_out = []
@@ -352,27 +674,94 @@ class AtariDosDiskImage(DiskImageBase):
             try:
                 binary = AtariDosFile(segment.rawdata)
                 segments_out.extend(binary.segments)
-            except InvalidBinaryFile:
+            except errors.InvalidBinaryFile:
                 log.debug("%s not a binary file; skipping segment generation" % str(segment))
         return segments_out
 
-def get_xex(segments, runaddr):
-    total = 2
-    for s in segments:
-        total += 4 + len(s)
-    total += 6
-    bytes = np.zeros([total], dtype=np.uint8)
-    bytes[0:2] = 0xff # FFFF header
-    i = 2
-    for s in segments:
-        words = bytes[i:i+4].view(dtype='<u2')
-        words[0] = s.start_addr
-        words[1] = s.start_addr + len(s) - 1
-        i += 4
-        bytes[i:i + len(s)] = s[:]
-        i += len(s)
-    words = bytes[i:i+6].view(dtype='<u2')
-    words[0] = 0x2e0
-    words[1] = 0x2e1
-    words[2] = runaddr
-    return bytes
+
+class BootDiskImage(AtariDosDiskImage):
+    def __str__(self):
+        return "%s Boot Disk" % (self.header)
+
+    def check_size(self):
+        if self.header is None:
+            return
+        start, size = self.header.get_pos(1)
+        b = self.bytes
+        i = self.header.header_offset
+        flag = b[i:i + 2].view(dtype='<u2')[0]
+        if flag == 0xffff:
+            raise errors.InvalidDiskImage("Appears to be an executable")
+        nsec = b[i + 1]
+        bload = b[i + 2:i + 4].view(dtype='<u2')[0]
+
+        # Sanity check: number of sectors to be loaded can't be more than the
+        # lower 48k of ram because there's no way to bank switch or anything
+        # before the boot sectors are finished loading
+        max_ram = 0xc000
+        max_size = max_ram - bload
+        max_sectors = max_size // self.header.sector_size
+        if nsec > max_sectors or nsec < 1:
+            raise errors.InvalidDiskImage("Number of boot sectors out of range (tried %d, max=%d" % (nsec, max_sectors))
+        if bload > (0xc000 - (nsec * self.header.sector_size)):
+            raise errors.InvalidDiskImage("Bad boot load address")
+
+    def get_boot_sector_info(self):
+        pass
+
+    def get_vtoc(self):
+        pass
+
+    def get_directory(self, directory=None):
+        pass
+
+    boot_record_type = np.dtype([
+        ('BFLAG', 'u1'),
+        ('BRCNT', 'u1'),
+        ('BLDADR', '<u2'),
+        ('BWTARR', '<u2'),
+        ])
+
+    def get_boot_segments(self):
+        data, style = self.get_sectors(1)
+        values = data[0:6].view(dtype=self.boot_record_type)[0]
+        flag = int(values[0])
+        segments = []
+        if flag == 0:
+            num = int(values[1])
+            addr = int(values[2])
+            s = self.get_sector_slice(1, num)
+            r = self.rawdata[s]
+            header = ObjSegment(r[0:6], 0, 0, addr, addr + 6, name="Boot Header")
+            sectors = ObjSegment(r, 0, 0, addr, addr + len(r), name="Boot Sectors")
+            code = ObjSegment(r[6:], 0, 0, addr + 6, addr + len(r), name="Boot Code")
+            segments = [sectors, header, code]
+        return segments
+
+    def get_vtoc_segments(self):
+        return []
+
+    def get_directory_segments(self):
+        return []
+
+
+class AtariDiskImage(BootDiskImage):
+    def __str__(self):
+        return "%s Unidentified Contents" % (self.header)
+
+    def check_size(self):
+        if self.header is None:
+            raise errors.InvalidDiskImage("Not a known Atari disk image format")
+
+    def get_boot_segments(self):
+        return []
+
+
+def add_atr_header(bytes):
+    header = AtrHeader(create=True)
+    header.check_size(len(bytes))
+    hlen = len(header)
+    data = np.empty([hlen + len(bytes)], dtype=np.uint8)
+    data[0:hlen] = header.to_array()
+    data[hlen:] = bytes
+    return data

atrcopy/cartridge.py

@@ -2,10 +2,10 @@ from collections import defaultdict
 
 import numpy as np
 
-from errors import *
-from segments import SegmentData, EmptySegment, ObjSegment
-from diskimages import DiskImageBase
-from utils import to_numpy
+from . import errors
+from .segments import SegmentData, EmptySegment, ObjSegment
+from .diskimages import DiskImageBase
+from .utils import to_numpy
 
 import logging
 log = logging.getLogger(__name__)
@@ -95,6 +95,7 @@ known_cart_types = [
 
 known_cart_type_map = {c[0]:i for i, c in enumerate(known_cart_types)}
 
+
 def get_known_carts():
     grouped = defaultdict(list)
     for c in known_cart_types[1:]:
@@ -102,14 +103,15 @@ def get_known_carts():
         grouped[size].append(c)
     return grouped
 
+
 def get_cart(cart_type):
     try:
         return known_cart_types[known_cart_type_map[cart_type]]
     except KeyError:
-        raise InvalidDiskImage("Unsupported cart type %d" % cart_type)
+        raise errors.InvalidDiskImage("Unsupported cart type %d" % cart_type)
 
 
-class A8CartHeader(object):
+class A8CartHeader:
     # Atari Cart format described by https://sourceforge.net/p/atari800/source/ci/master/tree/DOC/cart.txt  NOTE: Big endian!
     format = np.dtype([
         ('magic', '|S4'),
@@ -117,8 +119,9 @@ class A8CartHeader(object):
         ('checksum', '>u4'),
         ('unused','>u4')
         ])
+    nominal_length = format.itemsize
     file_format = "Cart"
-    
+
     def __init__(self, bytes=None, create=False):
         self.image_size = 0
         self.cart_type = -1
@@ -136,32 +139,39 @@ class A8CartHeader(object):
         self.main_origin = 0
         self.possible_types = set()
         if create:
-            self.header_offset = 16
+            self.header_offset = self.nominal_length
             self.check_size(0)
         if bytes is None:
             return
-        
+
         if len(bytes) == 16:
             values = bytes.view(dtype=self.format)[0]
-            if values[0] != 'CART':
-                raise InvalidCartHeader
+            if values[0] != b'CART':
+                raise errors.InvalidCartHeader
             self.cart_type = int(values[1])
             self.crc = int(values[2])
-            self.header_offset = 16
+            self.header_offset = self.nominal_length
             self.set_type(self.cart_type)
         else:
-            raise InvalidCartHeader
-    
+            raise errors.InvalidCartHeader
+
     def __str__(self):
         return "%s Cartridge (atari800 type=%d size=%d, %d banks, crc=%d)" % (self.cart_name, self.cart_type, self.cart_size, self.bank_size, self.crc)
-    
+
     def __len__(self):
         return self.header_offset
-    
+
+    @property
+    def valid(self):
+        return self.cart_type != -1
+
+    def calc_crc_from_data(self, data):
+        self.crc = 0
+
     def to_array(self):
-        raw = np.zeros([16], dtype=np.uint8)
+        raw = np.zeros([self.nominal_length], dtype=np.uint8)
         values = raw.view(dtype=self.format)[0]
-        values[0] = 'CART'
+        values[0] = b'CART'
         values[1] = self.cart_type
         values[2] = self.crc
         values[3] = 0
@@ -187,50 +197,45 @@ class A8CartHeader(object):
     def check_size(self, size):
         self.possible_types = set()
         k, r = divmod(size, 1024)
-        if r == 0 or r == 16:
+        if r == 0 or r == self.nominal_length:
             for i, t in enumerate(known_cart_types):
                 valid_size = t[0]
                 if k == valid_size:
                     self.possible_types.add(i)
 
 
-class AtariCartImage(DiskImageBase):
-    def __init__(self, rawdata, cart_type, filename=""):
-        self.cart_type = cart_type
-        DiskImageBase.__init__(self, rawdata, filename)
-
+class BaseAtariCartImage(DiskImageBase):
     def __str__(self):
         return str(self.header)
-    
+
     def read_header(self):
-        bytes = self.bytes[0:16]
+        data = self.bytes[0:16]
         try:
-            self.header = A8CartHeader(bytes)
-        except InvalidCartHeader:
+            self.header = A8CartHeader(data)
+        except errors.InvalidCartHeader:
             self.header = A8CartHeader()
-            self.header.set_type(self.cart_type)
 
     def strict_check(self):
-        if self.header.cart_type != self.cart_type:
-            raise InvalidDiskImage("Cart type doesn't match type defined in header")
+        raise NotImplementedError
 
     def relaxed_check(self):
         if self.header.cart_type != self.cart_type:
             # force the header to be the specified cart type
             self.header = A8CartHeader()
             self.header.set_type(self.cart_type)
-    
+            self.check_size()
+
     def check_size(self):
-        if self.header is None:
+        if not self.header.valid:
             return
         k, rem = divmod((len(self) - len(self.header)), 1024)
-        c = get_cart(self.cart_type)
+        c = get_cart(self.header.cart_type)
         log.debug("checking type=%d, k=%d, rem=%d for %s, %s" % (self.cart_type, k, rem, c[1], c[2]))
         if rem > 0:
-            raise InvalidDiskImage("Cart not multiple of 1K")
+            raise errors.InvalidDiskImage("Cart not multiple of 1K")
         if k != c[2]:
-            raise InvalidDiskImage("Image size %d doesn't match cart type %d size %d" % (k, self.cart_type, c[2]))
-    
+            raise errors.InvalidDiskImage("Image size %d doesn't match cart type %d size %d" % (k, self.cart_type, c[2]))
+
     def parse_segments(self):
         r = self.rawdata
         i = self.header.header_offset
@@ -256,6 +261,47 @@ class AtariCartImage(DiskImageBase):
             segments.append(s)
         return segments
 
+    def create_emulator_boot_segment(self):
+        h = self.header
+        k, rem = divmod(len(self), 1024)
+        if rem == 0:
+            h.calc_crc_from_data(self.bytes)
+            data_with_header = np.empty(len(self) + h.nominal_length, dtype=np.uint8)
+            data_with_header[0:h.nominal_length] = h.to_array()
+            data_with_header[h.nominal_length:] = self.bytes
+            r = SegmentData(data_with_header)
+        else:
+            r = self.rawdata
+        s = ObjSegment(r, 0, 0, self.header.main_origin, name="Cart image")
+        return s
+
+
+class AtariCartImage(BaseAtariCartImage):
+    def __init__(self, rawdata, cart_type, filename=""):
+        c = get_cart(cart_type)
+        self.cart_type = cart_type
+        DiskImageBase.__init__(self, rawdata, filename)
+
+    def strict_check(self):
+        if not self.header.valid:
+            raise errors.InvalidDiskImage("Missing cart header")
+        if self.header.cart_type != self.cart_type:
+            raise errors.InvalidDiskImage("Cart type doesn't match type defined in header")
+
+
+class Atari8bitCartImage(AtariCartImage):
+    def strict_check(self):
+        if "5200" in self.header.cart_name:
+            raise errors.InvalidDiskImage("5200 Carts don't work in the home computers.")
+        AtariCartImage.strict_check(self)
+
+
+class Atari5200CartImage(AtariCartImage):
+    def strict_check(self):
+        if "5200" not in self.header.cart_name:
+            raise errors.InvalidDiskImage("Home computer carts don't work in the 5200.")
+        AtariCartImage.strict_check(self)
+
 
 def add_cart_header(bytes):
     header = A8CartHeader(create=True)
@@ -265,3 +311,45 @@ def add_cart_header(bytes):
     data[0:hlen] = header.to_array()
     data[hlen:] = bytes
     return data
+
+
+class RomImage(DiskImageBase):
+    def __str__(self):
+        return f"{len(self.rawdata) // 1024}k ROM image"
+
+    def read_header(self):
+        self.header = A8CartHeader()
+
+    def strict_check(self):
+        self.check_size()
+
+    def check_size(self):
+        size = len(self)
+        if (size & (size - 1)) != 0:
+            raise errors.InvalidDiskImage("ROM image not a power of 2")
+
+    def parse_segments(self):
+        r = self.rawdata
+        s = ObjSegment(r, 0, 0, self.header.main_origin, name="Main Bank")
+        self.segments = [s]
+
+    def create_emulator_boot_segment(self):
+        s = self.segments[0]
+        if s.origin == 0:
+            return None
+        return s
+
+
+class Atari2600CartImage(RomImage):
+    def __str__(self):
+        return f"{len(self.rawdata) // 1024}k Atari 2600 Cartridge"
+
+
+class Atari2600StarpathImage(RomImage):
+    def __str__(self):
+        return f"{len(self.rawdata) // 1024}k Atari 2600 Starpath Cassette"
+
+
+class VectrexCartImage(RomImage):
+    def __str__(self):
+        return f"{len(self.rawdata) // 1024}k Vectrex Cartridge"

atrcopy/container.py

@@ -0,0 +1,82 @@
+import gzip
+import bz2
+import lzma
+import io
+
+import numpy as np
+
+from . import errors
+from .segments import SegmentData
+from .utils import to_numpy
+
+
+class DiskImageContainer:
+    """Unpacker for disk image compression.
+
+    Disk images may be compressed by any number of techniques. Subclasses of
+    DiskImageContainer implement the `unpack_bytes` method which examines the
+    byte_data argument for the supported compression type, and if valid returns
+    the unpacked bytes to be used in the disk image parsing.
+    """
+    def __init__(self, data):
+        self.unpacked = self.__unpack_raw_data(data)
+
+    def __unpack_raw_data(self, data):
+        raw = data.tobytes()
+        try:
+            unpacked = self.unpack_bytes(raw)
+        except EOFError as e:
+            raise errors.InvalidContainer(e)
+        return to_numpy(unpacked)
+
+    def unpack_bytes(self, byte_data):
+        """Attempt to unpack `byte_data` using this unpacking algorithm.
+
+        `byte_data` is a byte string, and should return a byte string if
+        successfully unpacked. Conversion to a numpy array will take place
+        automatically, outside of this method.
+
+        If the data is not recognized by this subclass, raise an
+        InvalidContainer exception. This signals to the caller that a different
+        container type should be tried.
+
+        If the data is recognized by this subclass but the unpacking algorithm
+        is not implemented, raise an UnsupportedContainer exception. This is
+        different than the InvalidContainer exception because it indicates that
+        the data was indeed recognized by this subclass (despite not being
+        unpacked) and checking further containers is not necessary.
+        """
+        pass
+
+
+class GZipContainer(DiskImageContainer):
+    def unpack_bytes(self, byte_data):
+        try:
+            buf = io.BytesIO(byte_data)
+            with gzip.GzipFile(mode='rb', fileobj=buf) as f:
+                unpacked = f.read()
+        except OSError as e:
+            raise errors.InvalidContainer(e)
+        return unpacked
+
+
+class BZipContainer(DiskImageContainer):
+    def unpack_bytes(self, byte_data):
+        try:
+            buf = io.BytesIO(byte_data)
+            with bz2.BZ2File(buf, mode='rb') as f:
+                unpacked = f.read()
+        except OSError as e:
+            raise errors.InvalidContainer(e)
+        return unpacked
+
+
+class LZMAContainer(DiskImageContainer):
+    def unpack_bytes(self, byte_data):
+        try:
+            buf = io.BytesIO(byte_data)
+            with lzma.LZMAFile(buf, mode='rb') as f:
+                unpacked = f.read()
+        except lzma.LZMAError as e:
+            raise errors.InvalidContainer(e)
+        return unpacked

atrcopy/dcm.py

@@ -0,0 +1,48 @@
+import numpy as np
+
+from . import errors
+from .container import DiskImageContainer
+from .segments import SegmentData
+
+
+class DCMContainer(DiskImageContainer):
+    valid_densities = {
+        0: (720, 128),
+        1: (720, 256),
+        2: (1040, 128),
+    }
+
+    def get_next(self):
+        try:
+            data = self.raw[self.index]
+        except IndexError:
+            raise errors.InvalidContainer("Incomplete DCM file")
+        else:
+            self.index += 1
+        return data
+
+    def unpack_bytes(self, data):
+        self.index = 0
+        self.count = len(data)
+        self.raw = data
+        archive_type = self.get_next()
+        if archive_type == 0xf9 or archive_type == 0xfa:
+            archive_flags = self.get_next()
+            if archive_flags & 0x1f != 1:
+                if archive_type == 0xf9:
+                    raise errors.InvalidContainer("DCM multi-file archive combined in the wrong order")
+                else:
+                    raise errors.InvalidContainer("Expected pass one of DCM archive first")
+            density_flag = (archive_flags >> 5) & 3
+            if density_flag not in self.valid_densities:
+                raise errors.InvalidContainer(f"Unsupported density flag {density_flag} in DCM")
+        else:
+            raise errors.InvalidContainer("Not a DCM file")
+
+        # DCM decoding goes here. Currently, instead of decoding it raises the
+        # UnsupportedContainer exception, which signals to the caller that the
+        # container has been successfully identified but can't be parsed.
+        #
+        # When decoding is supported, return the decoded byte array instead of
+        # this exception.
+        raise errors.UnsupportedContainer("DCM archives are not yet supported")

atrcopy/diskimages.py

@@ -1,134 +1,106 @@
 import numpy as np
 
-from errors import *
-from segments import SegmentData, EmptySegment, ObjSegment, RawSectorsSegment
-from utils import to_numpy
+from . import errors
+from .segments import SegmentData, EmptySegment, ObjSegment, RawSectorsSegment
+from .utils import *
+from .executables import create_executable_file_data
 
-class AtrHeader(object):
-    # ATR Format described in http://www.atarimax.com/jindroush.atari.org/afmtatr.html
-    format = np.dtype([
-        ('wMagic', '<u2'),
-        ('wPars', '<u2'),
-        ('wSecSize', '<u2'),
-        ('btParsHigh', 'u1'),
-        ('dwCRC','<u4'),
-        ('unused','<u4'),
-        ('btFlags','u1'),
-        ])
-    file_format = "ATR"
-    
-    def __init__(self, bytes=None, sector_size=128, initial_sectors=3, create=False):
+import logging
+log = logging.getLogger(__name__)
+try:  # Expensive debugging
+    _xd = _expensive_debugging
+except NameError:
+    _xd = False
+
+
+class BaseHeader:
+    file_format = "generic"  # text descriptor of file format
+    sector_class = WriteableSector
+
+    def __init__(self, sector_size=256, initial_sectors=0, vtoc_sector=0, starting_sector_label=0, create=False):
         self.image_size = 0
         self.sector_size = sector_size
+        self.payload_bytes = sector_size
+        self.initial_sector_size = 0
+        self.num_initial_sectors = 0
         self.crc = 0
         self.unused = 0
         self.flags = 0
         self.header_offset = 0
-        self.initial_sector_size = sector_size
-        self.num_initial_sectors = initial_sectors
-        self.max_sectors = 0
-        if create:
-            self.header_offset = 16
-            self.check_size(0)
-        if bytes is None:
-            return
-        
-        if len(bytes) == 16:
-            values = bytes.view(dtype=self.format)[0]
-            if values[0] != 0x296:
-                raise InvalidAtrHeader
-            self.image_size = (int(values[3]) * 256 * 256 + int(values[1])) * 16
-            self.sector_size = int(values[2])
-            self.crc = int(values[4])
-            self.unused = int(values[5])
-            self.flags = int(values[6])
-            self.header_offset = 16
-        else:
-            raise InvalidAtrHeader
-    
-    def __str__(self):
-        return "%s Disk Image (size=%d (%dx%db), crc=%d flags=%d unused=%d)" % (self.file_format, self.image_size, self.max_sectors, self.sector_size, self.crc, self.flags, self.unused)
-    
+        self.starting_sector_label = starting_sector_label
+        self.max_sectors = 0  # number of sectors, -1 is unlimited
+        self.tracks_per_disk = 0
+        self.sectors_per_track = 0
+        self.first_vtoc = vtoc_sector
+        self.num_vtoc = 1
+        self.extra_vtoc = []
+        self.first_directory = 0
+        self.num_directory = 0
+
     def __len__(self):
         return self.header_offset
-    
+
     def to_array(self):
-        raw = np.zeros([16], dtype=np.uint8)
-        values = raw.view(dtype=self.format)[0]
-        values[0] = 0x296
-        paragraphs = self.image_size / 16
-        parshigh, pars = divmod(paragraphs, 256*256)
-        values[1] = pars
-        values[2] = self.sector_size
-        values[3] = parshigh
-        values[4] = self.crc
-        values[5] = self.unused
-        values[6] = self.flags
-        return raw
+        header_bytes = np.zeros([self.header_offset], dtype=np.uint8)
+        self.encode(header_bytes)
+        return header_bytes
+
+    def encode(self, header_bytes):
+        """Subclasses should override this to put the byte values into the
+        header.
+        """
+        return
+
+    def sector_is_valid(self, sector):
+        return (self.max_sectors < 0) | (sector >= self.starting_sector_label and sector < (self.max_sectors + self.starting_sector_label))
+
+    def iter_sectors(self):
+        i = self.starting_sector_label
+        while self.sector_is_valid(i):
+            pos, size = self.get_pos(i)
+            yield i, pos, size
+            i += 1
+
+    def get_pos(self, sector):
+        """Get index (into the raw data of the disk image) of start of sector
+
+        This base class method assumes the sectors are one after another, in
+        order starting from the beginning of the raw data.
+        """
+        if not self.sector_is_valid(sector):
+            raise ByteNotInFile166("Sector %d out of range" % sector)
+        pos = sector * self.sector_size + self.header_offset
+        size = self.sector_size
+        return pos, size
+
+    def sector_from_track(self, track, sector):
+        return (track * self.sectors_per_track) + sector
+
+    def track_from_sector(self, sector):
+        track, sector = divmod(sector, self.sectors_per_track)
+        return track, sector
 
     def check_size(self, size):
-        if size == 92160 or size == 92176:
-            self.image_size = 92160
-            self.sector_size = 128
-            self.initial_sector_size = 0
-            self.num_initial_sectors = 0
-        elif size == 184320 or size == 184336:
-            self.image_size = 184320
-            self.sector_size = 256
-            self.initial_sector_size = 0
-            self.num_initial_sectors = 0
-        elif size == 183936 or size == 183952:
-            self.image_size = 183936
-            self.sector_size = 256
-            self.initial_sector_size = 128
-            self.num_initial_sectors = 3
-        else:
-            self.image_size = size
-            self.sector_size = 128
-        initial_bytes = self.initial_sector_size * self.num_initial_sectors
-        self.max_sectors = ((self.image_size - initial_bytes) / self.sector_size) + self.num_initial_sectors
+        raise errors.InvalidDiskImage("BaseHeader subclasses need custom checks for size")
 
     def strict_check(self, image):
         pass
-    
-    def sector_is_valid(self, sector):
-        return sector > 0 and sector <= self.max_sectors
-    
-    def get_pos(self, sector):
-        if not self.sector_is_valid(sector):
-            raise ByteNotInFile166("Sector %d out of range" % sector)
-        if sector <= self.num_initial_sectors:
-            pos = self.num_initial_sectors * (sector - 1)
-            size = self.initial_sector_size
-        else:
-            pos = self.num_initial_sectors * self.initial_sector_size + (sector - 1 - self.num_initial_sectors) * self.sector_size
-            size = self.sector_size
-        pos += self.header_offset
-        return pos, size
+
+    def create_sector(self, data=None):
+        if data is None:
+            data = np.zeros([self.sector_size], dtype=np.uint8)
+        return self.sector_class(self.sector_size, data)
 
 
-class XfdHeader(AtrHeader):
-    file_format = "XFD"
-    
-    def __str__(self):
-        return "%s Disk Image (size=%d (%dx%db)" % (self.file_format, self.image_size, self.max_sectors, self.sector_size)
-    
-    def __len__(self):
-        return 0
-    
-    def to_array(self):
-        raw = np.zeros([0], dtype=np.uint8)
-        return raw
-
-    def strict_check(self, image):
-        size = len(image)
-        if size in [92160, 133120, 183936, 184320]:
-            return
-        raise InvalidDiskImage("Uncommon size of XFD file")
+class Bootable:
+    def create_emulator_boot_segment(self):
+        return ObjSegment(self.rawdata, 0, 0, 0)
 
 
-class DiskImageBase(object):
-    def __init__(self, rawdata, filename=""):
+class DiskImageBase(Bootable):
+    default_executable_extension = None
+
+    def __init__(self, rawdata, filename="", create=False):
         self.rawdata = rawdata
         self.bytes = self.rawdata.get_data()
         self.style = self.rawdata.get_style()
@@ -137,20 +109,40 @@ class DiskImageBase(object):
         self.header = None
         self.total_sectors = 0
         self.unused_sectors = 0
-        self.files = []
+        self.files = [] # all dirents that show up in a normal dir listing
         self.segments = []
         self.all_sane = True
-        self.setup()
+        self.default_filetype = ""
+        if create:
+            self.header = self.new_header(self)
+        else:
+            self.setup()
 
     def __len__(self):
         return len(self.rawdata)
-    
+
+    @property
+    def writeable_sector_class(self):
+        return WriteableSector
+
+    @property
+    def raw_sector_class(self):
+        return RawSectorsSegment
+
+    @property
+    def vtoc_class(self):
+        return VTOC
+
+    @property
+    def directory_class(self):
+        return Directory
+
     def set_filename(self, filename):
-        if "." in filename:
-            self.filename, self.ext = filename.rsplit(".", 1)
+        if '.' in filename:
+            self.filename, self.ext = filename.rsplit('.', 1)
         else:
-            self.filename, self.ext = filename, ""
-    
+            self.filename, self.ext = filename, ''
+
     def dir(self):
         lines = []
         lines.append(str(self))
@@ -164,6 +156,9 @@ class DiskImageBase(object):
         self.read_header()
         self.header.check_size(self.size - len(self.header))
         self.check_size()
+        self.get_metadata()
+
+    def get_metadata(self):
         self.get_boot_sector_info()
         self.get_vtoc()
         self.get_directory()
@@ -178,68 +173,55 @@ class DiskImageBase(object):
         format.
         """
         pass
-    
+
     @classmethod
     def new_header(cls, diskimage, format="ATR"):
-        if format.lower() == "atr":
-            header = AtrHeader(create=True)
-            header.check_size(diskimage.size)
-        else:
-            raise RuntimeError("Unknown header type %s" % format)
-        return header
-    
+        raise errors.NotImplementedError
+
     def as_new_format(self, format="ATR"):
         """ Create a new disk image in the specified format
         """
-        first_data = len(self.header)
-        raw = self.rawdata[first_data:]
-        data = add_atr_header(raw)
-        newraw = SegmentData(data)
-        image = self.__class__(newraw)
-        return image
-    
+        raise errors.NotImplementedError
+
     def save(self, filename=""):
         if not filename:
             filename = self.filename
             if self.ext:
-                filename += "." + self.ext
+                filename += '.' + self.ext
         if not filename:
             raise RuntimeError("No filename specified for save!")
-        bytes = self.bytes[:]
+        data = self.bytes[:]
         with open(filename, "wb") as fh:
-            bytes.tofile(fh)
-    
+            data.tofile(fh)
+
     def assert_valid_sector(self, sector):
         if not self.header.sector_is_valid(sector):
-            raise ByteNotInFile166("Sector %d out of range" % sector)
-    
+            raise errors.ByteNotInFile166("Sector %d out of range" % sector)
+
     def check_sane(self):
         if not self.all_sane:
-            raise InvalidDiskImage("Invalid directory entries; may be boot disk")
-    
+            raise errors.InvalidDiskImage("Invalid directory entries; may be boot disk")
+
     def read_header(self):
-        bytes = self.bytes[0:16]
-        try:
-            self.header = AtrHeader(bytes)
-        except InvalidAtrHeader:
-            self.header = XfdHeader()
-    
+        return BaseHeader()
+
     def check_size(self):
         pass
-    
+
     def get_boot_sector_info(self):
         pass
-    
+
     def get_vtoc(self):
+        """Get information from VTOC and populate the VTOC object"""
         pass
-    
-    def get_directory(self):
+
+    def get_directory(self, directory=None):
         pass
-    
+
     def get_raw_bytes(self, sector):
         pos, size = self.header.get_pos(sector)
         return self.bytes[pos:pos + size], pos, size
-    
+
     def get_sector_slice(self, start, end=None):
         """ Get contiguous sectors
         
@@ -255,7 +237,7 @@ class DiskImageBase(object):
             _, more = self.header.get_pos(start)
             size += more
         return slice(pos, pos + size)
-    
+
     def get_sectors(self, start, end=None):
         """ Get contiguous sectors
         
@@ -265,7 +247,7 @@ class DiskImageBase(object):
         """
         s = self.get_sector_slice(start, end)
         return self.bytes[s], self.style[s]
-    
+
     def get_contiguous_sectors(self, sector, num):
         start = 0
         count = 0
@@ -275,103 +257,151 @@ class DiskImageBase(object):
                 start = pos
             count += size
         return start, count
-    
+
     def parse_segments(self):
         r = self.rawdata
         i = self.header.header_offset
         if self.header.image_size > 0:
             self.segments.append(ObjSegment(r[0:i], 0, 0, 0, i, name="%s Header" % self.header.file_format))
-        self.segments.append(RawSectorsSegment(r[i:], 1, self.header.max_sectors, self.header.image_size, 128, 3, self.header.sector_size, name="Raw disk sectors"))
+        self.segments.append(self.raw_sector_class(r[i:], self.header.starting_sector_label, self.header.max_sectors, self.header.image_size, self.header.initial_sector_size, self.header.num_initial_sectors, self.header.sector_size, name="Raw disk sectors"))
         self.segments.extend(self.get_boot_segments())
         self.segments.extend(self.get_vtoc_segments())
         self.segments.extend(self.get_directory_segments())
         self.segments.extend(self.get_file_segments())
-    
-    boot_record_type = np.dtype([
-        ('BFLAG', 'u1'),
-        ('BRCNT', 'u1'),
-        ('BLDADR', '<u2'),
-        ('BWTARR', '<u2'),
-        ])
-    
+
     def get_boot_segments(self):
-        data, style = self.get_sectors(1)
-        values = data[0:6].view(dtype=self.boot_record_type)[0]  
-        flag = int(values[0])
-        segments = []
-        if flag == 0:
-            num = int(values[1])
-            addr = int(values[2])
-            s = self.get_sector_slice(1, num)
-            r = self.rawdata[s]
-            header = ObjSegment(r[0:6], 0, 0, addr, addr + 6, name="Boot Header")
-            sectors = ObjSegment(r, 0, 0, addr, addr + len(r), name="Boot Sectors")
-            code = ObjSegment(r[6:], 0, 0, addr + 6, addr + len(r), name="Boot Code")
-            segments = [sectors, header, code]
-        return segments
-    
+        return []
+
     def get_vtoc_segments(self):
         return []
-    
+
     def get_directory_segments(self):
         return []
-    
-    def find_file(self, filename):
+
+    def find_dirent(self, filename):
+        # check if we've been passed a dirent instead of a filename
+        if hasattr(filename, "filename"):
+            return filename
         for dirent in self.files:
-            if filename == dirent.get_filename():
-                return self.get_file(dirent)
-        return ""
-    
+            if filename == dirent.filename:
+                return dirent
+        raise errors.FileNotFound("%s not found on disk" % str(filename))
+
+    def find_file(self, filename):
+        dirent = self.find_dirent(filename)
+        return self.get_file(dirent)
+
     def get_file(self, dirent):
         segment = self.get_file_segment(dirent)
-        return segment.tostring()
-    
+        return segment.tobytes()
+
     def get_file_segment(self, dirent):
         pass
-    
+
     def get_file_segments(self):
         segments = []
         for dirent in self.files:
             try:
                 segment = self.get_file_segment(dirent)
-            except InvalidFile, e:
-                segment = EmptySegment(self.rawdata, name=dirent.get_filename(), error=str(e))
+            except errors.InvalidFile as e:
+                segment = EmptySegment(self.rawdata, name=dirent.filename, error=str(e))
             segments.append(segment)
         return segments
 
+    def create_executable_file_image(self, output_name, segments, run_addr=None):
+        try:
+            data, filetype = create_executable_file_data(output_name, segments, run_addr)
+        except errors.UnsupportedContainer:
+            data, filetype = create_executable_file_data(self.default_executable_extension, segments, run_addr)
+        return data, filetype
 
-class BootDiskImage(DiskImageBase):
-    def __str__(self):
-        return "%s Boot Disk" % (self.header)
-    
-    def check_size(self):
-        if self.header is None:
-            return
-        start, size = self.header.get_pos(1)
-        b = self.bytes
-        i = self.header.header_offset
-        flag = b[i:i + 2].view(dtype='<u2')[0]
-        if flag == 0xffff:
-            raise InvalidDiskImage("Appears to be an executable")
-        nsec = b[i + 1]
-        bload = b[i + 2:i + 4].view(dtype='<u2')[0]
-        
-        # Sanity check: number of sectors to be loaded can't be more than the
-        # lower 48k of ram because there's no way to bank switch or anything
-        # before the boot sectors are finished loading
-        max_ram = 0xc000
-        max_size = max_ram - bload
-        max_sectors = max_size / self.header.sector_size
-        if nsec > max_sectors or nsec < 3:
-            raise InvalidDiskImage("Number of boot sectors out of range")
-        if bload < 0x200 or bload > (0xc000 - (nsec * self.header.sector_size)):
-            raise InvalidDiskImage("Bad boot load address")
+    @classmethod
+    def create_boot_image(self, segments, run_addr=None):
+        raise errors.NotImplementedError
 
-def add_atr_header(bytes):
-    header = AtrHeader(create=True)
-    header.check_size(len(bytes))
-    hlen = len(header)
-    data = np.empty([hlen + len(bytes)], dtype=np.uint8)
-    data[0:hlen] = header.to_array()
-    data[hlen:] = bytes
-    return data
+    # file writing methods
+
+    def begin_transaction(self):
+        state = self.bytes[:], self.style[:]
+        return state
+
+    def rollback_transaction(self, state):
+        self.bytes[:], self.style[:] = state
+        return
+
+    def get_vtoc_object(self):
+        vtoc_segments = self.get_vtoc_segments()
+        vtoc = self.vtoc_class(self.header, vtoc_segments)
+        return vtoc
+
+    def write_file(self, filename, filetype, data):
+        """Write data to a file on disk
+
+        This throws various exceptions on failures, for instance if there is
+        not enough space on disk or a free entry is not available in the
+        catalog.
+        """
+        state = self.begin_transaction()
+        try:
+            directory = self.directory_class(self.header)
+            self.get_directory(directory)
+            dirent = directory.add_dirent(filename, filetype)
+            data = to_numpy(data)
+            sector_list = self.build_sectors(data)
+            vtoc = self.get_vtoc_object()
+            directory.save_dirent(self, dirent, vtoc, sector_list)
+            self.write_sector_list(sector_list)
+            self.write_sector_list(vtoc)
+            self.write_sector_list(directory)
+        except errors.AtrError:
+            self.rollback_transaction(state)
+            raise
+        finally:
+            self.get_metadata()
+
+    def build_sectors(self, data):
+        data = to_numpy(data)
+        sectors = BaseSectorList(self.header)
+        index = 0
+        while index < len(data):
+            count = min(self.header.payload_bytes, len(data) - index)
+            sector = self.header.create_sector(data[index:index + count])
+            sectors.append(sector)
+            index += count
+        return sectors
+
+    def write_sector_list(self, sector_list):
+        for sector in sector_list:
+            pos, size = self.header.get_pos(sector.sector_num)
+            if _xd: log.debug("writing: %s at %d" % (sector, pos))
+            self.bytes[pos:pos + size] = sector.data
+
+    def delete_file(self, filename):
+        state = self.begin_transaction()
+        try:
+            directory = self.directory_class(self.header)
+            self.get_directory(directory)
+            dirent = directory.find_dirent(filename)
+            sector_list = dirent.get_sectors_in_vtoc(self)
+            vtoc = self.get_vtoc_object()
+            directory.remove_dirent(self, dirent, vtoc, sector_list)
+            self.write_sector_list(vtoc)
+            self.write_sector_list(directory)
+        except errors.AtrError:
+            self.rollback_transaction(state)
+            raise
+        finally:
+            self.get_metadata()
+
+    def shred(self, fill_value=0):
+        state = self.begin_transaction()
+        try:
+            vtoc = self.get_vtoc_object()
+            for sector_num, pos, size in vtoc.iter_free_sectors():
+                if _xd: log.debug("shredding: sector %s at %d, fill value=%d" % (sector_num, pos, fill_value))
+                self.bytes[pos:pos + size] = fill_value
+        except errors.AtrError:
+            self.rollback_transaction(state)
+            raise
+        finally:
+            self.get_metadata()

atrcopy/dos33.py

@@ -0,0 +1,687 @@
+import numpy as np
+
+from . import errors
+from .diskimages import BaseHeader, DiskImageBase, Bootable
+from .utils import Directory, VTOC, WriteableSector, BaseSectorList, Dirent
+from .segments import DefaultSegment, EmptySegment, ObjSegment, RawTrackSectorSegment, SegmentSaver, get_style_bits, SegmentData
+from .executables import get_bsave
+
+import logging
+log = logging.getLogger(__name__)
+try:  # Expensive debugging
+    _xd = _expensive_debugging
+except NameError:
+    _xd = False
+
+
+class Dos33TSSector(WriteableSector):
+    def __init__(self, header, sector_list=None, start=None, end=None, data=None):
+        WriteableSector.__init__(self, header.sector_size, data)
+        self.header = header
+        self.used = header.sector_size
+        if data is None:
+            self.set_tslist(sector_list, start, end)
+
+    def set_tslist(self, sector_list, start, end):
+        index = 0xc
+        for i in range(start, end):
+            sector = sector_list[i]
+            t, s = self.header.track_from_sector(sector.sector_num)
+            self.data[index] = t
+            self.data[index + 1] = s
+            if _xd: log.debug("tslist entry #%d: %d, %d" % (index, t, s))
+            index += 2
+
+    def get_tslist(self):
+        index = 0xc
+        sector_list = []
+        while index < self.header.sector_size:
+            t = self.data[index]
+            s = self.data[index + 1]
+            sector_list.append(self.header.sector_from_track(t, s))
+            index += 2
+        return sector_list
+
+    @property
+    def next_sector_num(self):
+        t = self.data[1]
+        s = self.data[2]
+        return self.header.sector_from_track(t, s)
+
+    @next_sector_num.setter
+    def next_sector_num(self, value):
+        self._next_sector_num = value
+        t, s = self.header.track_from_sector(value)
+        self.data[1] = t
+        self.data[2] = s
+
+
+class Dos33VTOC(VTOC):
+    max_tracks = (256 - 0x38) // 4  # 50, but kept here in case sector size changed
+    max_sectors = max_tracks * 16
+    vtoc_bit_reorder_index = np.tile(np.arange(15, -1, -1), max_tracks) + (np.repeat(np.arange(max_tracks), 16) * 16)
+
+    def parse_segments(self, segments):
+        # VTOC stored in groups of 4 bytes starting at 0x38
+        # in bits, the sector used data is stored by track:
+        #
+        # FEDCBA98 76543210 xxxxxxxx xxxxxxxx
+        #
+        # where the x values are ignored (should be zeros). Track 0 info is
+        # found starting at 0x38, track 1 is found at 0x3c, etc.
+        #
+        # Want to convert this to an array that is a list of bits by
+        # track/sector number, i.e.:
+        #
+        # t0s0 t0s1 t0s2 t0s3 t0s4 t0s5 t0s6 t0s7 ... t1s0 t1s1 ... etc
+        #
+        # Problem: the bits are stored backwards, so a straight unpackbits will
+        # produce:
+        #
+        # t0sf t0se t0sd ...
+        #
+        # i.e. each group of 16 bits needs to be reversed.
+        self.vtoc = segments[0].data
+
+        # create a view starting at 0x38 where out of every 4 bytes, the first
+        # two are used and the second 2 are skipped. Regular slicing doesn't
+        # work like this, so thanks to stackoverflow.com/questions/33801170,
+        # reshaping it to a 2d array with 4 elements in each row, doing a slice
+        # *there* to skip the last 2 entries in each row, then flattening it
+        # gives us what we need.
+        usedbytes = self.vtoc[0x38:].reshape((-1, 4))[:,:2].flatten()
+
+        # The bits here are still ordered backwards for each track, e.g. F E D
+        # C B A 9 8 7 6 5 4 3 2 1 0
+        bits = np.unpackbits(usedbytes)
+
+        # so we need to reorder them using numpy's indexing before stuffing
+        # them into the sector map
+        self.sector_map[0:self.max_sectors] = bits[self.vtoc_bit_reorder_index]
+        if _xd: log.debug("vtoc before:\n%s" % str(self))  # expensive debugging call
+
+    def calc_bitmap(self):
+        if _xd: log.debug("vtoc after:\n%s" % str(self))  # expensive debugging call
+
+        # reverse the process from above, so swap the order of every 16 bits,
+        # turn them into bytes, then stuff them back into the vtoc. The bit
+        # reorder list is commutative, so we don't need another order here.
+        packed = np.packbits(self.sector_map[self.vtoc_bit_reorder_index])
+        vtoc = self.vtoc[0x38:].reshape((-1, 4))
+        packed = packed.reshape((-1, 2))
+        vtoc[:,:2] = packed[:,:]
+
+        # FIXME
+        self.vtoc[0x38:] = vtoc.flatten()
+        s = WriteableSector(self.sector_size, self.vtoc)
+        s.sector_num = 17 * 16
+        self.sectors.append(s)
+
+
+class Dos33Directory(Directory):
+    @property
+    def dirent_class(self):
+        return Dos33Dirent
+
+    def get_dirent_sector(self):
+        s = self.sector_class(self.sector_size)
+        data = np.zeros([0x0b], dtype=np.uint8)
+        s.add_data(data)
+        return s
+
+    def encode_empty(self):
+        return np.zeros([Dos33Dirent.format.itemsize], dtype=np.uint8)
+
+    def encode_dirent(self, dirent):
+        data = dirent.encode_dirent()
+        if _xd: log.debug("encoded dirent: %s" % data)
+        return data
+
+    def set_sector_numbers(self, image):
+        current_sector = -1
+        for sector in self.sectors:
+            current_sector, next_sector = image.get_directory_sector_links(current_sector)
+            sector.sector_num = current_sector
+            t, s = image.header.track_from_sector(next_sector)
+            sector.data[1] = t
+            sector.data[2] = s
+            if _xd: log.debug("directory sector %d -> next = %d" % (sector.sector_num, next_sector))
+            current_sector = next_sector
+
+
+class Dos33Dirent(Dirent):
+    format = np.dtype([
+        ('track', 'u1'),
+        ('sector', 'u1'),
+        ('flag', 'u1'),
+        ('name','S30'),
+        ('num_sectors','<u2'),
+        ])
+
+    def __init__(self, image, file_num=0, bytes=None):
+        Dirent.__init__(self, file_num)
+        self._file_type = 0
+        self.locked = False
+        self.deleted = False
+        self.track = 0
+        self.sector = 0
+        self.filename = ""
+        self.num_sectors = 0
+        self.is_sane = True
+        self.current_sector_index = 0
+        self.current_read = 0
+        self.sectors_seen = None
+        self.sector_map = None
+        self.parse_raw_dirent(image, bytes)
+
+    def __str__(self):
+        return "File #%-2d (%s) %03d %-30s %03d %03d" % (self.file_num, self.summary, self.num_sectors, self.filename, self.track, self.sector)
+
+    def __eq__(self, other):
+        return self.__class__ == other.__class__ and self.filename == other.filename and self.track == other.track and self.sector == other.sector and self.num_sectors == other.num_sectors
+
+    type_to_text = {
+        0x0: "T",  # text
+        0x1: "I",  # integer basic
+        0x2: "A",  # applesoft basic
+        0x4: "B",  # binary
+        0x8: "S",  # ?
+        0x10: "R",  # relocatable object module
+        0x20: "a",  # ?
+        0x40: "b",  # ?
+    }
+    text_to_type = {v: k for k, v in type_to_text.items()}
+
+    @property
+    def file_type(self):
+        """User friendly version of file type, not the binary number"""
+        return self.type_to_text.get(self._file_type, "?")
+
+    @property
+    def summary(self):
+        if self.deleted:
+            locked = "D"
+            file_type = " "
+        else:
+            locked = "*" if self.locked else " "
+            file_type = self.file_type
+        flag = "%s%s" % (locked, file_type)
+        return flag
+
+    @property
+    def verbose_info(self):
+        return self.summary
+
+    @property
+    def in_use(self):
+        return not self.deleted
+
+    @property
+    def flag(self):
+        return 0xff if self.deleted else self._file_type | (0x80 * int(self.locked))
+
+    def extra_metadata(self, image):
+        lines = []
+        ts = self.get_track_sector_list(image)
+        lines.append("track/sector list at: " + str(ts))
+        lines.append("sector map: " + str(self.sector_map))
+        return "\n".join(lines)
+
+    def parse_raw_dirent(self, image, data):
+        if data is None:
+            return
+        values = data.view(dtype=self.format)[0]
+        self.track = values[0]
+        if self.track == 0xff:
+            self.deleted = True
+            self.track = data[0x20]
+        else:
+            self.deleted = False
+        self.sector = values[1]
+        self._file_type = values[2] & 0x7f
+        self.locked = values[2] & 0x80
+        self.filename = (data[3:0x20] - 0x80).tobytes().rstrip().decode("ascii", errors='ignore')
+        self.num_sectors = int(values[4])
+        self.is_sane = self.sanity_check(image)
+
+    def encode_dirent(self):
+        data = np.zeros([self.format.itemsize], dtype=np.uint8)
+        values = data.view(dtype=self.format)[0]
+        values[0] = 0xff if self.deleted else self.track
+        values[1] = self.sector
+        values[2] = self.flag
+        n = min(len(self.filename), 30)
+        data[3:3+n] = np.frombuffer(self.filename.encode("ascii"), dtype=np.uint8) | 0x80
+        data[3+n:] = ord(' ') | 0x80
+        if self.deleted:
+            data[0x20] = self.track
+        values[4] = self.num_sectors
+        return data
+
+    def mark_deleted(self):
+        self.deleted = True
+
+    def update_sector_info(self, sector_list):
+        self.num_sectors = sector_list.num_sectors
+        self.starting_sector = sector_list.first_sector
+
+    def add_metadata_sectors(self, vtoc, sector_list, header):
+        """Add track/sector list
+        """
+        tslist = BaseSectorList(header)
+        for start in range(0, len(sector_list), header.ts_pairs):
+            end = min(start + header.ts_pairs, len(sector_list))
+            if _xd: log.debug("ts: %d-%d" % (start, end))
+            s = Dos33TSSector(header, sector_list, start, end)
+            s.ts_start, s.ts_end = start, end
+            tslist.append(s)
+        self.num_tslists = len(tslist)
+        vtoc.assign_sector_numbers(self, tslist)
+        sector_list.extend(tslist)
+        self.track, self.sector = header.track_from_sector(tslist[0].sector_num)
+        if _xd: log.debug("track/sector lists:\n%s" % str(tslist))
+
+    def sanity_check(self, image):
+        if self.deleted:
+            return True
+        if self.track == 0:
+            return False
+        s = image.header.sector_from_track(self.track, self.sector)
+        if not image.header.sector_is_valid(s):
+            return False
+        if self.num_sectors < 0 or self.num_sectors > image.header.max_sectors:
+            return False
+        return True
+
+    def get_track_sector_list(self, image):
+        tslist = BaseSectorList(image.header)
+        sector_num = image.header.sector_from_track(self.track, self.sector)
+        sector_map = []
+        while sector_num > 0:
+            image.assert_valid_sector(sector_num)
+            if _xd: log.debug("reading track/sector list at %d for %s" % (sector_num, self))
+            data, _ = image.get_sectors(sector_num)
+            sector = Dos33TSSector(image.header, data=data)
+            sector.sector_num = sector_num
+            sector_map.extend(sector.get_tslist())
+            tslist.append(sector)
+            sector_num = sector.next_sector_num
+        self.sector_map = sector_map[0:self.num_sectors - len(tslist)]
+        self.track_sector_list = tslist
+        return tslist
+
+    def get_sectors_in_vtoc(self, image):
+        self.get_track_sector_list(image)
+        sectors = BaseSectorList(image.header)
+        sectors.extend(self.track_sector_list)
+        for sector_num in self.sector_map:
+            sector = WriteableSector(image.header.sector_size, None, sector_num)
+            sectors.append(sector)
+        return sectors
+
+    def start_read(self, image):
+        if not self.is_sane:
+            raise errors.InvalidDirent("Invalid directory entry '%s'" % str(self))
+        self.get_track_sector_list(image)
+        if _xd: log.debug("start_read: %s, t/s list: %s" % (str(self), str(self.sector_map)))
+        self.current_sector_index = 0
+        self.current_read = self.num_sectors
+
+    def read_sector(self, image):
+        try:
+            sector = self.sector_map[self.current_sector_index]
+        except IndexError:
+            sector = -1  # force ByteNotInFile166 error at next read
+        if _xd: log.debug("read_sector: index %d=%d in %s" % (self.current_sector_index,sector, str(self)))
+        last = (self.current_sector_index == len(self.sector_map) - 1)
+        raw, pos, size = image.get_raw_bytes(sector)
+        bytes, num_data_bytes = self.process_raw_sector(image, raw)
+        return bytes, last, pos, num_data_bytes
+
+    def process_raw_sector(self, image, raw):
+        self.current_sector_index += 1
+        num_bytes = len(raw)
+        return raw[0:num_bytes], num_bytes
+
+    def get_filename(self):
+        return self.filename
+
+    def set_values(self, filename, filetype, index):
+        self.filename = '%-30s' % filename[0:30]
+        self._file_type = self.text_to_type.get(filetype, 0x04)
+        self.locked = False
+        self.deleted = False
+
+    def get_binary_start_address(self, image):
+        self.start_read(image)
+        data, _, _, _ = self.read_sector(image)
+        addr = int(data[0]) + 256 * int(data[1])
+        return addr
+
+
+class Dos33Header(BaseHeader):
+    file_format = "DOS 3.3"
+
+    def __init__(self):
+        BaseHeader.__init__(self, 256)
+
+    def __str__(self):
+        return "%s Disk Image (size=%d (%dx%dB)" % (self.file_format, self.image_size, self.max_sectors, self.sector_size)
+
+    def check_size(self, size):
+        if size != 143360:
+            raise errors.InvalidDiskImage("Incorrect size for DOS 3.3 image")
+        self.image_size = size
+        self.first_vtoc = 17 * 16
+        self.num_vtoc = 1
+        self.first_directory = self.first_vtoc + 15
+        self.num_directory = 8
+        self.tracks_per_disk = 35
+        self.sectors_per_track = 16
+        self.max_sectors = self.tracks_per_disk * self.sectors_per_track
+
+
+class Dos33DiskImage(DiskImageBase):
+    default_executable_extension = "BSAVE"
+
+    def __init__(self, rawdata, filename=""):
+        DiskImageBase.__init__(self, rawdata, filename)
+        self.default_filetype = "B"
+
+    def __str__(self):
+        return str(self.header)
+
+    def read_header(self):
+        self.header = Dos33Header()
+
+    @property
+    def vtoc_class(self):
+        return Dos33VTOC
+
+    @property
+    def directory_class(self):
+        return Dos33Directory
+
+    @property
+    def raw_sector_class(self):
+        return RawTrackSectorSegment
+
+    def get_boot_sector_info(self):
+        # based on logic from a2server
+        data, style = self.get_sectors(0)
+        magic = data[0:4]
+        if (magic == [1, 56, 176, 3]).all():
+            raise errors.InvalidDiskImage("ProDOS format found; not DOS 3.3 image")
+        swap_order = False
+        data, style = self.get_sectors(self.header.first_vtoc)
+        if data[3] == 3:
+            if data[1] < 35 and data[2] < 16:
+                data, style = self.get_sectors(self.header.first_vtoc + 14)
+                if data[2] != 13:
+                    log.warning("DOS 3.3 byte swap needed!")
+                    swap_order = True
+            else:
+                raise errors.InvalidDiskImage("Invalid VTOC location for DOS 3.3")
+
+
+    vtoc_type = np.dtype([
+        ('unused1', 'S1'),
+        ('cat_track','u1'),
+        ('cat_sector','u1'),
+        ('dos_release', 'u1'),
+        ('unused2', 'S2'),
+        ('vol_num', 'u1'),
+        ('unused3', 'S32'),
+        ('max_pairs', 'u1'),
+        ('unused4', 'S8'),
+        ('last_track', 'u1'),
+        ('track_dir', 'i1'),
+        ('unused5', 'S2'),
+        ('num_tracks', 'u1'),
+        ('sectors_per_track', 'u1'),
+        ('sector_size', 'u2'),
+        ])
+
+    def get_vtoc(self):
+        data, style = self.get_sectors(self.header.first_vtoc)
+        values = data[0:self.vtoc_type.itemsize].view(dtype=self.vtoc_type)[0]
+        self.header.first_directory = self.header.sector_from_track(values['cat_track'], values['cat_sector'])
+        self.header.sector_size = int(values['sector_size'])
+        if self.header.sector_size != 256:
+            log.warning(f"Nonstandard sector size {self.header.sector_size}; this is likely an error, setting to 256")
+            self.header.sector_size = 256
+        self.header.max_sectors = int(values['num_tracks']) * int(values['sectors_per_track'])
+        self.header.ts_pairs = int(values['max_pairs'])
+        self.header.dos_release = values['dos_release']
+        self.header.last_track_num = values['last_track']
+        self.header.track_alloc_dir = values['track_dir']
+        self.assert_valid_sector(self.header.first_directory)
+
+    def get_directory(self, directory=None):
+        sector = self.header.first_directory
+        num = 0
+        files = []
+        while sector > 0:
+            self.assert_valid_sector(sector)
+            if _xd: log.debug("reading catalog sector: %d" % sector)
+            values, style = self.get_sectors(sector)
+            sector = self.header.sector_from_track(values[1], values[2])
+            i = 0xb
+            while i < 256:
+                dirent = Dos33Dirent(self, num, values[i:i+0x23])
+                if dirent.flag == 0:
+                    break
+                if not dirent.is_sane:
+                    log.warning("Illegally formatted directory entry %s" % dirent)
+                    self.all_sane = False
+                elif not dirent.deleted:
+                    files.append(dirent)
+                if directory is not None:
+                    directory.set(num, dirent)
+                if _xd: log.debug("valid directory entry %s" % dirent)
+                i += 0x23
+                num += 1
+        self.files = files
+
+    def get_boot_segments(self):
+        segments = []
+        s = self.get_sector_slice(0, 0)
+        r = self.rawdata[s]
+        boot1 = ObjSegment(r, 0, 0, 0x800, name="Boot 1")
+        s = self.get_sector_slice(1, 9)
+        r = self.rawdata[s]
+        boot2 = ObjSegment(r, 0, 0, 0x3700, name="Boot 2")
+        s = self.get_sector_slice(0x0a, 0x0b)
+        r = self.rawdata[s]
+        relocator = ObjSegment(r, 0, 0, 0x1b00, name="Relocator")
+        s = self.get_sector_slice(0x0c, 0x0c + 25)
+        r = self.rawdata[s]
+        boot3 = ObjSegment(r, 0, 0, 0x1d00, name="Boot 3")
+        return [boot1, boot2, relocator, boot3]
+
+    def get_vtoc_segments(self):
+        r = self.rawdata
+        segments = []
+        addr = 0
+        start, count = self.get_contiguous_sectors(self.header.first_vtoc, 1)
+        segment = RawTrackSectorSegment(r[start:start+count], self.header.first_vtoc, 1, count, 0, 0, self.header.sector_size, name="VTOC")
+        segment.style[:] = get_style_bits(data=True)
+        segment.set_comment_at(0x00, "unused")
+        segment.set_comment_at(0x01, "Track number of next catalog sector")
+        segment.set_comment_at(0x02, "Sector number of next catalog sector")
+        segment.set_comment_at(0x03, "Release number of DOS used to format")
+        segment.set_comment_at(0x04, "unused")
+        segment.set_comment_at(0x06, "Volume number")
+        segment.set_comment_at(0x07, "unused")
+        segment.set_comment_at(0x27, "Number of track/sector pairs per t/s list sector")
+        segment.set_comment_at(0x28, "unused")
+        segment.set_comment_at(0x30, "Last track that sectors allocated")
+        segment.set_comment_at(0x31, "Track allocation direction")
+        segment.set_comment_at(0x32, "unused")
+        segment.set_comment_at(0x34, "Tracks per disk")
+        segment.set_comment_at(0x35, "Sectors per track")
+        segment.set_comment_at(0x36, "Bytes per sector")
+        index = 0x38
+        for track in range(35):
+            segment.set_comment_at(index, "Free sectors in track %d" % track)
+            index += 4
+        segments.append(segment)
+        return segments
+
+    def get_directory_segments(self):
+        byte_order = []
+        r = self.rawdata
+        segments = []
+        sector = self.header.first_directory
+        while sector > 0:
+            self.assert_valid_sector(sector)
+            if _xd: log.debug("loading directory segment from catalog sector %d" % sector)
+            raw, pos, size = self.get_raw_bytes(sector)
+            byte_order.extend(list(range(pos, pos + size)))
+            sector = self.header.sector_from_track(raw[1], raw[2])
+        raw = self.rawdata.get_indexed(byte_order)
+        segment = DefaultSegment(raw, name="Catalog")
+        segment.style[:] = get_style_bits(data=True)
+        index = 0
+        filenum = 0
+        while index < len(segment):
+            segment.set_comment_at(index + 0x00, "unused")
+            segment.set_comment_at(index + 0x01, "Track number of next catalog sector")
+            segment.set_comment_at(index + 0x02, "Sector number of next catalog sector")
+            segment.set_comment_at(index + 0x03, "unused")
+            index += 0x0b
+            for i in range(7):
+                segment.set_comment_at(index + 0x00, "FILE #%d: Track number of next catalog sector" % filenum)
+                segment.set_comment_at(index + 0x01, "FILE #%d: Sector number of next catalog sector" % filenum)
+                segment.set_comment_at(index + 0x02, "FILE #%d: File type" % filenum)
+                segment.set_comment_at(index + 0x03, "FILE #%d: Filename" % filenum)
+                segment.set_comment_at(index + 0x21, "FILE #%d: Number of sectors in file" % filenum)
+                index += 0x23
+                filenum += 1
+        segments.append(segment)
+        return segments
+
+    def get_directory_sector_links(self, sector_num):
+        if sector_num == -1:
+            sector_num = self.header.first_directory
+        self.assert_valid_sector(sector_num)
+        raw, _, _ = self.get_raw_bytes(sector_num)
+        next_sector = self.header.sector_from_track(raw[1], raw[2])
+        if _xd: log.debug("checking catalog sector %d, next catalog sector: %d" % (sector_num, next_sector))
+        if next_sector == 0:
+            raise errors.NoSpaceInDirectory("No space left in catalog")
+        return sector_num, next_sector
+
+    def get_file_segment(self, dirent):
+        byte_order = []
+        dirent.start_read(self)
+        while True:
+            bytes, last, pos, size = dirent.read_sector(self)
+            byte_order.extend(list(range(pos, pos + size)))
+            if last:
+                break
+        if len(byte_order) > 0:
+            name = "%s %03d %s" % (dirent.summary, dirent.num_sectors, dirent.filename)
+            verbose_name = "%s (%d sectors, first@%d) %s" % (dirent.filename, dirent.num_sectors, dirent.sector_map[0], dirent.verbose_info)
+            raw = self.rawdata.get_indexed(byte_order)
+            if dirent.file_type == "B":
+                addr = dirent.get_binary_start_address(self) - 4 # factor in 4 byte header
+            else:
+                addr = 0
+            segment = ObjSegment(raw, 0, 0, origin=addr, name=name, verbose_name=verbose_name)
+            if addr > 0:
+                style = segment.get_style_bits(data=True)
+                segment.style[0:4] = style
+        else:
+            segment = EmptySegment(self.rawdata, name=dirent.filename)
+        return segment
+
+
+class Dos33BinFile(Bootable):
+    """Parse a binary chunk into segments according to the DOS 3.3 binary
+    dump format
+    """
+
+    def __init__(self, rawdata):
+        self.rawdata = rawdata
+        self.size = len(rawdata)
+        self.segments = []
+        self.files = []
+
+    def __str__(self):
+        return "\n".join(str(s) for s in self.segments) + "\n"
+
+    def strict_check(self):
+        pass
+
+    def relaxed_check(self):
+        pass
+
+    def parse_segments(self):
+        r = self.rawdata
+        b = r.get_data()
+        s = r.get_style()
+        pos = 0
+        style_pos = 0
+        first = True
+        if _xd: log.debug("Initial parsing: size=%d" % self.size)
+        if len(b[pos:pos + 4]) == 4:
+            start, count = b[pos:pos + 4].view(dtype='<u2')
+            if count != self.size - 4:
+                raise errors.InvalidBinaryFile(f"Extra data after BSAVE segment: file size {self.size}, header specifies {count} bytes")
+            s[pos:pos + 4] = get_style_bits(data=True)
+            data = b[pos + 4:pos + 4 + count]
+            if len(data) == count:
+                name = "BSAVE data" % start
+            else:
+                raise errors.InvalidBinaryFile(f"Incomplete BSAVE data: expected {count}, loaded {len(data)}")
+            self.segments.append(ObjSegment(r[pos + 4:pos + 4 + count], pos, pos + 4, start, start + len(data), name))
+
+        else:
+            raise errors.InvalidBinaryFile(f"Invalid BSAVE header")
+
+    def create_emulator_boot_segment(self):
+        return self.segments[0]
+
+
+class ProdosHeader(Dos33Header):
+    file_format = "ProDOS"
+
+    def __str__(self):
+        return "%s Disk Image (size=%d) THIS FORMAT IS NOT SUPPORTED YET!" % (self.file_format, self.image_size)
+
+
+class ProdosDiskImage(DiskImageBase):
+    def __str__(self):
+        return str(self.header)
+
+    def read_header(self):
+        self.header = ProdosHeader()
+
+    def get_boot_sector_info(self):
+        # based on logic from a2server
+        data, style = self.get_sectors(0)
+        magic = data[0:4]
+        swap_order = False
+        if (magic == [1, 56, 176, 3]).all():
+            data, style = self.get_sectors(1)
+            prodos = data[3:9].tobytes()
+            if prodos == "PRODOS":
+                pass
+            else:
+                data, style = self.get_sectors(14)
+                prodos = data[3:9].tobytes()
+                if prodos == "PRODOS":
+                    swap_order = True
+                else:
+                    # FIXME: this doesn't seem to be the only way to identify a
+                    # PRODOS disk. I have example images where PRODOS occurs at
+                    # 0x21 - 0x27 in t0s14 and 0x11 - 0x16 in t0s01. Using 3 -
+                    # 9 as magic bytes was from the cppo script from
+                    # https://github.com/RasppleII/a2server but it seems that
+                    # more magic bytes might be acceptable?
+
+                    #raise errors.InvalidDiskImage("No ProDOS header info found")
+                    pass
+            raise errors.UnsupportedDiskImage("ProDOS format found but not supported")
+        raise errors.InvalidDiskImage("Not ProDOS format")

atrcopy/dummy.py

@@ -0,0 +1,44 @@
+import os
+
+import numpy as np
+
+from . import errors
+from .segments import SegmentData, EmptySegment, ObjSegment
+from .diskimages import DiskImageBase
+from .utils import to_numpy
+
+import logging
+log = logging.getLogger(__name__)
+
+
+class LocalFilesystemImage(DiskImageBase):
+    def __init__(self, path):
+        self.path = path
+
+    def __str__(self, path="."):
+        return f"Local filesystem output to: {self.path}"
+
+    def save(self, filename=""):
+        # This is to save the disk image containing the files on the disk image
+        # to the local disk, which doesn't make sense when the disk image is
+        # the filesystem.
+        pass
+
+    def find_dirent(self, name):
+        path = os.path.join(self.path, name)
+        if os.path.exists(path):
+            return True
+        raise errors.FileNotFound("%s not found on disk" % str(name))
+
+    def write_file(self, name, filetype, data):
+        path = os.path.join(self.path, name)
+        with open(path, "wb") as fh:
+            fh.write(data)
+
+    def delete_file(self, name):
+        pass
+
+
+class LocalFilesystem():
+    def __init__(self, path="."):
+        self.image = LocalFilesystemImage(path)

atrcopy/errors.py

@@ -1,32 +1,75 @@
 class AtrError(RuntimeError):
     pass
 
+
 class InvalidAtrHeader(AtrError):
     pass
 
+
 class InvalidCartHeader(AtrError):
     pass
 
+
 class InvalidDiskImage(AtrError):
+    """ Disk image is not recognized by a parser.
+
+    Usually a signal to try the next parser; this error doesn't propagate out
+    to the user much.
+    """
     pass
 
+
+class UnsupportedDiskImage(AtrError):
+    """ Disk image is recognized by a parser but it isn't supported yet.
+
+    This error does propagate out to the user.
+    """
+    pass
+
+
 class InvalidDirent(AtrError):
     pass
 
+
 class LastDirent(AtrError):
     pass
 
+
 class InvalidFile(AtrError):
     pass
 
+
 class FileNumberMismatchError164(InvalidFile):
     pass
 
+
 class ByteNotInFile166(InvalidFile):
     pass
 
+
 class InvalidBinaryFile(InvalidFile):
     pass
 
+
 class InvalidSegmentParser(AtrError):
     pass
+
+
+class NoSpaceInDirectory(AtrError):
+    pass
+
+
+class NotEnoughSpaceOnDisk(AtrError):
+    pass
+
+
+class FileNotFound(AtrError):
+    pass
+
+
+class UnsupportedContainer(AtrError):
+    pass
+
+
+class InvalidContainer(AtrError):
+    pass

atrcopy/executables.py

@@ -0,0 +1,112 @@
+import numpy as np
+
+from . import errors
+from .segments import SegmentData, EmptySegment, ObjSegment, RawSectorsSegment, DefaultSegment, SegmentedFileSegment, SegmentSaver, get_style_bits
+from .utils import *
+
+import logging
+log = logging.getLogger(__name__)
+try:  # Expensive debugging
+    _xd = _expensive_debugging
+except NameError:
+    _xd = False
+
+
+def get_xex(segments, run_addr=None):
+    segments_copy = [s for s in segments]  # don't affect the original list!
+    main_segment = None
+    sub_segments = []
+    data_style = get_style_bits(data=True)
+    total = 2
+    runad = False
+    for s in segments:
+        total += 4 + len(s)
+        if s.origin == 0x2e0:
+            runad = True
+    if not runad:
+        words = np.empty([1], dtype='<u2')
+        if run_addr:
+            found = False
+            for s in segments:
+                if run_addr >= s.origin and run_addr < s.origin + len(s):
+                    found = True
+                    break
+            if not found:
+                raise errors.InvalidBinaryFile("Run address points outside data segments")
+        else:
+            run_addr = segments[0].origin
+        words[0] = run_addr
+        r = SegmentData(words.view(dtype=np.uint8))
+        s = DefaultSegment(r, 0x2e0)
+        segments_copy[0:0] = [s]
+        total += 6
+    bytes = np.zeros([total], dtype=np.uint8)
+    rawdata = SegmentData(bytes)
+    main_segment = DefaultSegment(rawdata)
+    main_segment.data[0:2] = 0xff  # FFFF header
+    main_segment.style[0:2] = data_style
+    i = 2
+    for s in segments_copy:
+        # create new sub-segment inside new main segment that duplicates the
+        # original segment's data/style
+        new_s = DefaultSegment(rawdata[i:i+4+len(s)], s.origin)
+        words = new_s.data[0:4].view(dtype='<u2')
+        words[0] = s.origin
+        words[1] = s.origin + len(s) - 1
+        new_s.style[0:4] = data_style
+        new_s.data[4:4+len(s)] = s[:]
+        new_s.style[4:4+len(s)] = s.style[:]
+        i += 4 + len(s)
+        new_s.copy_user_data(s, 4)
+        sub_segments.append(new_s)
+    return main_segment, sub_segments
+
+
+def get_bsave(segments, run_addr=None):
+    # Apple 2 executables get executed at the first address loaded. If the
+    # run_addr is not the first byte of the combined data, have to create a
+    # new 3-byte segment with a "JMP run_addr" to go at the beginning
+    origin = 100000000
+    last = -1
+
+    for s in segments:
+        origin = min(origin, s.origin)
+        last = max(last, s.origin + len(s))
+        if _xd: log.debug("contiguous bytes needed: %04x - %04x" % (origin, last))
+    if run_addr and run_addr != origin:
+        # check if run_addr points to some location that has data
+        found = False
+        for s in segments:
+            if run_addr >= s.origin and run_addr < s.origin + len(s):
+                found = True
+                break
+        if not found:
+            raise errors.InvalidBinaryFile("Run address points outside data segments")
+        origin -= 3
+        hi, lo = divmod(run_addr, 256)
+        raw = SegmentData([0x4c, lo, hi])
+        all_segments = [DefaultSegment(raw, origin=origin)]
+        all_segments.extend(segments)
+    else:
+        all_segments = segments
+    size = last - origin
+    image = np.zeros([size + 4], dtype=np.uint8)
+    words = image[0:4].view(dtype="<u2")  # always little endian
+    words[0] = origin
+    words[1] = size
+    for s in all_segments:
+        index = s.origin - origin + 4
+        log.debug("setting data for $%04x - $%04x at index $%04x" % (s.origin, s.origin + len(s), index))
+        image[index:index + len(s)] = s.data
+    return image
+
+
+def create_executable_file_data(filename, segments, run_addr=None):
+    name = filename.lower()
+    if name.endswith("xex"):
+        base_segment, user_segments = get_xex(segments, run_addr)
+        return base_segment.data, "XEX"
+    elif name.endswith("bin") or name.endswith("bsave"):
+        data = get_bsave(segments, run_addr)
+        return data, "B"
+    raise errors.UnsupportedContainer

atrcopy/fstbt.py

@@ -0,0 +1,4 @@
+# generated file - recompile fstbt.s to make changes
+
+# append jump target (lo, hi) and sector address list before saving to disk
+fstbt = b"\x01\xa8\x8dP\xc0\x8dR\xc0\x8dW\xc0\xd0\t\xc6\xfe\x106\xa9\xd0\x8d\x0b\x08\xee\x1a\x08\xad\x86\x08\xc9\xc0\xf0d\xc9\xd0\xf0\xf2\xc9\xd1\xd0\x05\x8dT\xc0\xf0\xe9\xc9\xd2\xd0\x05\x8dU\xc0\xf0\xe0\xc9\xe0\x90\x0b)\x1f\x85\xfe\xa9$\x8d\x0b\x08\x10\xc6\x85'\xc8\xc0\x10\x90\t\xf0\x05 g\x08\xa8,\xa0\x01\x84=\xc8\xa5'\xf0\xba\x8a {\xf8\t\xc0H\xa9[H`\xe6A\x06@ o\x08\x18 t\x08\xe6@\xa5@)\x03*\x05+\xa8\xb9\x80\xc0\xa90L\xa8\xfcL"

atrcopy/kboot.py

@@ -1,8 +1,8 @@
 import numpy as np
 
-from errors import *
-from ataridos import AtariDosDirent, XexSegment
-from diskimages import DiskImageBase
+from . import errors
+from .ataridos import AtrHeader, AtariDosDirent, AtariDosDiskImage, XexSegment, get_xex
+from .segments import SegmentData
 
 
 class KBootDirent(AtariDosDirent):
@@ -10,13 +10,13 @@ class KBootDirent(AtariDosDirent):
         AtariDosDirent.__init__(self, image)
         self.in_use = True
         self.starting_sector = 4
-        self.filename = image.filename
-        if not self.filename:
-            self.filename = "KBOOT"
-        if self.filename == self.filename.upper():
-            self.ext = "XEX"
+        self.basename = image.filename
+        if not self.basename:
+            self.basename = b"KBOOT"
+        if self.basename == self.basename.upper():
+            self.ext = b"XEX"
         else:
-            self.ext = "xex"
+            self.ext = b"xex"
         start, size = image.header.get_pos(4)
         i = image.header.header_offset + 9
         count = image.bytes[i] + 256 * image.bytes[i+1] + 256 * 256 *image.bytes[i + 2]
@@ -25,8 +25,8 @@ class KBootDirent(AtariDosDirent):
         else:
             self.exe_size = count
             self.exe_start = start
-        self.num_sectors = count / 128 + 1
-    
+        self.num_sectors = count // 128 + 1
+
     def parse_raw_dirent(self, image, bytes):
         pass
 
@@ -35,14 +35,17 @@ class KBootDirent(AtariDosDirent):
         return raw[0:num_bytes], num_bytes
 
 
-class KBootImage(DiskImageBase):
+class KBootImage(AtariDosDiskImage):
     def __str__(self):
         return "%s KBoot Format: %d byte executable" % (self.header, self.files[0].exe_size)
-    
+
     def check_sane(self):
         if not self.all_sane:
-            raise InvalidDiskImage("Doesn't seem to be KBoot header")
-    
+            raise errors.InvalidDiskImage("Doesn't seem to be KBoot header")
+
+    def get_vtoc(self):
+        pass
+
     def get_directory(self):
         dirent = KBootDirent(self)
         if not dirent.is_sane:
@@ -55,27 +58,66 @@ class KBootImage(DiskImageBase):
         raw = self.rawdata[start:end]
         return XexSegment(raw, 0, 0, start, end, name="KBoot Executable")
 
-xexboot_header = '\x00\x03\x00\x07\r\x07L\r\x07\xff\xff\x00\x00\xa0\x00\x8c\t\x03\x8c\x04\x03\x8cD\x02\x8c\xe2\x02\x8c\xe3\x02\xc8\x84\t\x8c\x01\x03\xce\x06\x03\xa91\x8d\x00\x03\xa9R\x8d\x02\x03\xa9\x80\x8d\x08\x03\xa9\x01\x8d\x05\x03\xa9\xdd\x8d0\x02\xa9\x07\x8d1\x02\xa9\x00\xaa\x8d\x0b\x03\xa9\x04\x8d\n\x03 \xb6\x07\xca \x9f\x07\x85C \x9f\x07\x85D%C\xc9\xff\xf0\xf0 \x9f\x07\x85E \x9f\x07\x85F \x9f\x07\x91C\xe6C\xd0\x02\xe6D\xa5E\xc5C\xa5F\xe5D\xb0\xeb\xad\xe2\x02\r\xe3\x02\xf0\xc9\x86\x19 \x9c\x07\xa6\x19\xa0\x00\x8c\xe2\x02\x8c\xe3\x02\xf0\xb8l\xe2\x02\xad\t\x07\xd0\x0b\xad\n\x07\xd0\x03l\xe0\x02\xce\n\x07\xce\t\x07\xe0\x80\x90"\xa9@\x8d\x03\x03 Y\xe4\x10\x06\xce\x01\x07\xd0\xf1\x00\xee\n\x03\xd0\x03\xee\x0b\x03\xad\n\x03\x8d\x19\xd0\xa0\x00\xa2\x00\xbd\x00\x01\xe8`pppppF\xf2\x07p\x07ppp\x06p\x06p\x06A\xdd\x07\x00\x00\x00\x00\x00,/!$).\'\x0e\x0e\x0e\x00\x00\x00\x00\x00\xea\xf5\xed\xf0\xed\xe1\xee\x00\xec\xe5\xf6\xe5\xec\x00\xf4\xe5\xf3\xf4\xe5\xf2\x00\x00\x00\x00\x00\x00\x00&2/-\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00playermissileNcom\x00\x00\x00\x00ataripodcastNcom\x00\x00'
+    def get_boot_segments(self):
+        return []
 
-def add_xexboot_header(bytes, bootcode=None):
+    def get_vtoc_segments(self):
+        return []
+
+    def get_directory_segments(self):
+        return []
+
+    @classmethod
+    def create_boot_image(cls, segments, run_addr=None):
+        data_segment, _ = get_xex(segments)
+        payload_bytes = add_xexboot_header(data_segment.data)
+        data_bytes = np.zeros(len(payload_bytes) + 16, np.uint8)
+        data_bytes[16:] = payload_bytes[:]
+        header_bytes = data_bytes[0:16]
+        atr_header = AtrHeader(create=True)
+        atr_header.check_size(len(payload_bytes))
+        atr_header.encode(header_bytes)
+        raw = SegmentData(data_bytes)
+        atr = cls(raw, create=True)
+        return atr
+
+
+xexboot_header = b'\x00\x03\x00\x07\r\x07L\r\x07\x1c[\x00\x00\xa0\x00\x8c\t\x03\x8c\x04\x03\x8cD\x02\x8c\xe2\x02\x8c\xe3\x02\xc8\x84\t\x8c\x01\x03\xce\x06\x03\xa91\x8d\x00\x03\xa9R\x8d\x02\x03\xa9\x80\x8d\x08\x03\xa9\x01\x8d\x05\x03\xa9\xe3\x8d0\x02\x8d\x02\xd4\xa9\x07\x8d1\x02\x8d\x03\xd4\xa9\x00\xaa\x8d\x0b\x03\xa9\x04\x8d\n\x03 \xbc\x07\xca \xa5\x07\x85C \xa5\x07\x85D%C\xc9\xff\xf0\xf0 \xa5\x07\x85E \xa5\x07\x85F \xa5\x07\x91C\xe6C\xd0\x02\xe6D\xa5E\xc5C\xa5F\xe5D\xb0\xeb\xad\xe2\x02\r\xe3\x02\xf0\xc9\x86\x19 \xa2\x07\xa6\x19\xa0\x00\x8c\xe2\x02\x8c\xe3\x02\xf0\xb8l\xe2\x02\xad\t\x07\xd0\x0b\xad\n\x07\xd0\x03l\xe0\x02\xce\n\x07\xce\t\x07\xe0\x80\x90"\xa9@\x8d\x03\x03 Y\xe4\x10\x06\xce\x01\x07\xd0\xf1\x00\xee\n\x03\xd0\x03\xee\x0b\x03\xad\n\x03\x8d\x19\xd0\xa0\x00\xa2\x00\xbd\x00\x01\xe8`pppppF\xf8\x07p\x07ppp\x06p\x06p\x06A\xe3\x07\x00\x00\x00\x00\x00,/!$).\'\x0e\x0e\x0e\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00&2/-'
+
+
+def insert_bytes(data, offset, string, color):
+    s = np.frombuffer(string.upper(), dtype=np.uint8) - 32  # convert to internal
+    s = s | color
+    count = len(s)
+    tx = offset + (20 - count) // 2
+    data[tx:tx+count] = s
+
+
+def add_xexboot_header(bytes, bootcode=None, title=b"DEMO", author=b"an atari user"):
     sec_size = 128
     xex_size = len(bytes)
-    num_sectors = (xex_size + sec_size - 1) / sec_size
+    num_sectors = (xex_size + sec_size - 1) // sec_size
     padded_size = num_sectors * sec_size
     if xex_size < padded_size:
         bytes = np.append(bytes, np.zeros([padded_size - xex_size], dtype=np.uint8))
-    paragraphs = padded_size / 16
-    print xex_size, num_sectors, paragraphs, padded_size
-    
+    paragraphs = padded_size // 16
+
     if bootcode is None:
-        bootcode = np.fromstring(xexboot_header, dtype=np.uint8)
+        bootcode = np.copy(np.frombuffer(xexboot_header, dtype=np.uint8))
+    else:
+        # don't insert title or author in user supplied bootcode; would have to
+        # assume that the user supplied everything desired in their own code!
+        title = ""
+        author = ""
     bootsize = np.alen(bootcode)
     v = bootcode[9:11].view(dtype="<u2")
     v[0] = xex_size
-    print bootcode[0:16]
-    
+
     bootsectors = np.zeros([384], dtype=np.uint8)
     bootsectors[0:bootsize] = bootcode
+
+    insert_bytes(bootsectors, 268, title, 0b11000000)
+    insert_bytes(bootsectors, 308, author, 0b01000000)
+
     image = np.append(bootsectors, bytes)
-    print np.alen(image)
     return image

atrcopy/magic.py

@@ -0,0 +1,73 @@
+import numpy as np
+
+import logging
+log = logging.getLogger(__name__)
+
+
+magic = [
+    {'mime': "application/vnd.atari8bit.atr.getaway_pd",
+     'name': "Getaway Public Domain ATR",
+     'signature': [
+         (slice(8, 10), [0x82, 0x39]),
+         (slice(12, 16), [0x67, 0x21, 0x70, 0x64]),
+         ],
+    },
+
+    {'mime': "application/vnd.atari8bit.xex.getaway",
+     'name': "Getaway XEX",
+     'signature': [
+         (slice(0, 6), [0xff, 0xff, 0x80, 0x2a, 0xff, 0x8a]),
+         ],
+    },
+
+    {'mime': "application/vnd.atari8bit.atr.getaway",
+     'name': "Getaway ATR",
+     'signature': [
+         (slice(0x10, 0x19), [0x00, 0xc1, 0x80, 0x0f, 0xcc, 0x22, 0x18, 0x60, 0x0e]),
+         ],
+    },
+
+    {'mime': "application/vnd.atari8bit.atr.jumpman_level_tester",
+     'name': "Jumpman Level Tester from Omnivore",
+     'signature': [
+         (slice(0, 5), [0x96, 0x02 , 0xd0 , 0x05 , 0x80]),
+         (0x0196 + 0x3f, 0x4c),
+         (0x0196 + 0x48, 0x20),
+         (0x0196 + 0x4b, 0x60),
+         (0x0196 + 0x4c, 0xff),
+         ],
+    },
+
+    {'mime': "application/vnd.atari8bit.atr.jumpman",
+     'name': "Jumpman",
+     'signature': [
+         (slice(0, 5), [0x96, 0x02 , 0x80 , 0x16 , 0x80]),
+         (0x0810 + 0x3f, 0x4c),
+         (0x0810 + 0x48, 0x20),
+         (0x0810 + 0x4b, 0x60),
+         (0x0810 + 0x4c, 0xff),
+         ],
+    },
+]
+
+
+def check_signature(raw, sig):
+    for index, expected in sig:
+        actual = raw.data[index].tolist()
+        if actual == expected:
+            log.debug(" match at %s: %s" % (str(index), str(expected)))
+        if actual != expected:
+            log.debug(" failed at %s: %s != %s" % (str(index), str(expected), str(raw.data[index])))
+            return False
+    return True
+
+
+def guess_detail_for_mime(mime, raw, parser):
+    for entry in magic:
+        if entry['mime'].startswith(mime):
+            log.debug("checking entry for %s" % entry['mime'])
+            if check_signature(raw, entry['signature']):
+                log.debug("found match: %s" % entry['name'])
+                return entry['mime']
+    return mime
+

atrcopy/mame.py

@@ -2,10 +2,10 @@ import zipfile
 
 import numpy as np
 
-from errors import *
-from segments import SegmentData, EmptySegment, ObjSegment
-from diskimages import DiskImageBase
-from utils import to_numpy
+from . import errors
+from .segments import SegmentData, EmptySegment, ObjSegment
+from .diskimages import DiskImageBase
+from .utils import to_numpy
 
 import logging
 log = logging.getLogger(__name__)
@@ -14,13 +14,13 @@ log = logging.getLogger(__name__)
 class MameZipImage(DiskImageBase):
     def __init__(self, rawdata, filename=""):
         self.zipdata = rawdata
-        fh = self.zipdata.stringio
+        fh = self.zipdata.bufferedio
         if zipfile.is_zipfile(fh):
             with zipfile.ZipFile(fh) as zf:
                 self.check_zip_size(zf)
                 self.create_rawdata(zf)
         else:
-            raise InvalidDiskImage("Not a MAME zip file")
+            raise errors.InvalidDiskImage("Not a MAME zip file")
         DiskImageBase.__init__(self, self.rawdata, filename)
 
     def __str__(self):
@@ -34,19 +34,19 @@ class MameZipImage(DiskImageBase):
 
     def relaxed_check(self):
         pass
-    
+
     def check_zip_size(self, zf):
         for item in zf.infolist():
-            _, r = divmod(item.file_size, 256)
+            _, r = divmod(item.file_size, 16)
             if r > 0:
-                raise InvalidDiskImage("zip entry not 256 byte multiple")
+                raise errors.InvalidDiskImage("zip entry not 16 byte multiple")
 
     def create_rawdata(self, zf):
         roms = []
         segment_info = []
         offset = 0
         for item in zf.infolist():
-            rom = np.fromstring(zf.open(item).read(), dtype=np.uint8)
+            rom = np.frombuffer(zf.open(item).read(), dtype=np.uint8)
             roms.append(rom)
             segment_info.append((offset, item.file_size, item.filename, item.CRC))
             offset += item.file_size
@@ -56,7 +56,7 @@ class MameZipImage(DiskImageBase):
 
     def check_size(self):
         pass
-    
+
     def parse_segments(self):
         r = self.rawdata
         self.segments = []

atrcopy/omnivore_loader.py

@@ -0,0 +1,27 @@
+from . import find_diskimage_from_data, errors
+
+import logging
+log = logging.getLogger(__name__)
+
+
+def identify_mime(header, fh):
+    mime_type = None
+    try:
+        fh.seek(0)
+        data = fh.read()
+    except IOError as e:
+        log.debug(f"atrcopy loader: error reading entire file: {e}")
+    else:
+        try:
+            parser, mime_type = find_diskimage_from_data(data, True)
+        except (errors.UnsupportedContainer, errors.UnsupportedDiskImage, IOError) as e:
+            log.debug(f"error in atrcopy parser: {e}")
+        else:
+            log.debug(f"{parser.image}: {mime_type}")
+
+        if mime_type:
+            log.debug(f"atrcopy loader: identified {mime_type}")
+            return dict(mime=mime_type, ext="", atrcopy_parser=parser)
+        else:
+            log.debug(f"atrcopy loader: not recognized")
+    return None

atrcopy/parsers.py

@@ -1,53 +1,112 @@
+import hashlib
+
 import numpy as np
 
-from segments import SegmentData, DefaultSegment
-from diskimages import BootDiskImage
-from kboot import KBootImage
-from ataridos import AtariDosDiskImage, AtariDosFile
-from spartados import SpartaDosDiskImage
-from cartridge import AtariCartImage, get_known_carts
-from mame import MameZipImage
-from errors import *
+from .segments import SegmentData, DefaultSegment
+from .kboot import KBootImage
+from .ataridos import AtariDosDiskImage, BootDiskImage, AtariDosFile, XexContainerSegment, AtariDiskImage
+from .spartados import SpartaDosDiskImage
+from .cartridge import AtariCartImage, Atari8bitCartImage, Atari5200CartImage, get_known_carts, RomImage, Atari2600CartImage, Atari2600StarpathImage, VectrexCartImage
+from .mame import MameZipImage
+from .dos33 import Dos33DiskImage, ProdosDiskImage, Dos33BinFile
+from .standard_delivery import StandardDeliveryImage
+from . import errors
+from .magic import guess_detail_for_mime
+from . import container
+from .dcm import DCMContainer
+from .signatures import sha1_signatures
+
+import logging
+log = logging.getLogger(__name__)
 
 
-class SegmentParser(object):
+class SegmentParser:
     menu_name = ""
     image_type = None
-    
+    container_segment = DefaultSegment
+
     def __init__(self, segment_data, strict=False):
         self.image = None
         self.segments = []
         self.strict = strict
-        self.parse(segment_data)
+        self.segment_data = segment_data
+        self.parse()
 
-    def parse(self, r):
-        self.segments.append(DefaultSegment(r, 0))
+    def __str__(self):
+        lines = []
+        lines.append("%s (%s)" % (self.menu_name, self.__class__.__name__))
+        if log.isEnabledFor(logging.DEBUG):
+            lines.append("segments:")
+            for s in self.segments:
+                lines.append("  %s" % s)
+        return "\n".join(lines)
+
+    def __getstate__(self):
+        """Custom jsonpickle state save routine
+
+        This routine culls down the list of attributes that should be
+        serialized, and in some cases changes their format slightly so they
+        have a better mapping to json objects. For instance, json can't handle
+        dicts with integer keys, so dicts are turned into lists of lists.
+        Tuples are also turned into lists because tuples don't have a direct
+        representation in json, while lists have a compact representation in
+        json.
+        """
+        state = dict()
+        for key in ['segments', 'strict']:
+            state[key] = getattr(self, key)
+        return state
+
+    def __setstate__(self, state):
+        """Custom jsonpickle state restore routine
+
+        The use of jsonpickle to recreate objects doesn't go through __init__,
+        so there will be missing attributes when restoring old versions of the
+        json. Once a version gets out in the wild and additional attributes are
+        added to a segment, a default value should be applied here.
+        """
+        self.__dict__.update(state)
+
+    def parse(self):
+        r = self.segment_data
+        self.segments.append(self.container_segment(r, 0, name=self.menu_name))
         try:
+            log.debug("Trying %s" % self.image_type)
+            log.debug(self.image_type.__mro__)
             self.image = self.get_image(r)
             self.check_image()
             self.image.parse_segments()
-        except AtrError:
-            raise InvalidSegmentParser
+        except errors.UnsupportedDiskImage:
+            raise
+        except errors.AtrError as e:
+            raise errors.InvalidSegmentParser(e)
         self.segments.extend(self.image.segments)
 
+    def reconstruct_segments(self, new_rawdata):
+        self.image = self.get_image(new_rawdata)
+        self.segment_data = new_rawdata
+        for s in self.segments:
+            s.reconstruct_raw(new_rawdata)
+
     def get_image(self, r):
+        log.info(f"checking image type {self.image_type}")
         return self.image_type(r)
 
     def check_image(self):
         if self.strict:
             try:
                 self.image.strict_check()
-            except AtrError:
-                raise InvalidSegmentParser
+            except errors.AtrError as e:
+                raise errors.InvalidSegmentParser(e)
         else:
             self.image.relaxed_check()
 
 
 class DefaultSegmentParser(SegmentParser):
     menu_name = "Raw Data"
-    
-    def parse(self, r):
-        self.segments = [DefaultSegment(r, 0)]
+
+    def parse(self):
+        self.segments = [DefaultSegment(self.segment_data, 0)]
 
 
 class KBootSegmentParser(SegmentParser):
@@ -70,9 +129,15 @@ class AtariBootDiskSegmentParser(SegmentParser):
     image_type = BootDiskImage
 
 
+class AtariUnidentifiedSegmentParser(SegmentParser):
+    menu_name = "Atari Disk Image"
+    image_type = AtariDiskImage
+
+
 class XexSegmentParser(SegmentParser):
     menu_name = "XEX (Atari 8-bit executable)"
     image_type = AtariDosFile
+    container_segment = XexContainerSegment
 
 
 class AtariCartSegmentParser(SegmentParser):
@@ -82,39 +147,171 @@ class AtariCartSegmentParser(SegmentParser):
     cart_info = None
 
     def get_image(self, r):
+        log.info(f"checking cart type {self.cart_type}: {self.image_type}")
         return self.image_type(r, self.cart_type)
 
 
+class Atari8bitCartParser(AtariCartSegmentParser):
+    menu_name = "Atari Home Computer Cartridge"
+    image_type = Atari8bitCartImage
+
+
+class Atari5200CartParser(AtariCartSegmentParser):
+    menu_name = "Atari 5200 Cartridge"
+    image_type = Atari5200CartImage
+
+
+class Atari2600CartParser(SegmentParser):
+    menu_name = "Atari 2600 Cartridge"
+    image_type = Atari2600CartImage
+
+
+class Atari2600StarpathParser(SegmentParser):
+    menu_name = "Atari 2600 Starpath Cassette"
+    image_type = Atari2600StarpathImage
+
+
+class VectrexParser(SegmentParser):
+    menu_name = "Vectrex Cartridge"
+    image_type = VectrexCartImage
+
+
+class RomParser(SegmentParser):
+    menu_name = "ROM Image"
+    image_type = RomImage
+
+
 class MameZipParser(SegmentParser):
     menu_name = "MAME ROM Zipfile"
     image_type = MameZipImage
 
 
-def guess_parser_for_mime(mime, r):
+class Dos33SegmentParser(SegmentParser):
+    menu_name = "DOS 3.3 Disk Image"
+    image_type = Dos33DiskImage
+
+
+class Dos33BinSegmentParser(SegmentParser):
+    menu_name = "BIN (Apple ][ executable)"
+    image_type = Dos33BinFile
+
+
+class ProdosSegmentParser(SegmentParser):
+    menu_name = "ProDOS Disk Image"
+    image_type = ProdosDiskImage
+
+
+known_containers = [
+    container.GZipContainer,
+    container.BZipContainer,
+    container.LZMAContainer,
+    DCMContainer,
+]
+
+
+def guess_container(r, verbose=False):
+    for c in known_containers:
+        if verbose:
+            log.info(f"trying container {c}")
+        try:
+            found = c(r)
+        except errors.InvalidContainer as e:
+            continue
+        else:
+            if verbose:
+                log.info(f"found container {c}")
+            return found
+    log.info(f"image does not appear to be a container.")
+    return None
+
+
+def guess_parser_by_size(r, verbose=False):
+    found = None
+    mime = None
+    size = len(r)
+    if size in sha1_signatures:
+        sha_hash = hashlib.sha1(r.data).digest()
+        log.info(f"{size} in signature database, attempting to match {sha_hash}")
+        try:
+            match = sha1_signatures[size][sha_hash]
+        except KeyError:
+            pass
+        else:
+            mime = match[0]
+            log.info(f"found match: {match}")
+            parsers = mime_parsers[mime]
+            for parser in parsers:
+                try:
+                    found = parser(r, False)
+                    break
+                except errors.InvalidSegmentParser as e:
+                    if verbose:
+                        log.info("parser isn't %s: %s" % (parser.__name__, str(e)))
+                    pass
+        if found is None:
+            log.info(f"no matching signature")
+    else:
+        log.info(f"{size} not found in signature database; skipping sha1 matching")
+    return mime, found
+
+def guess_parser_for_mime(mime, r, verbose=False):
     parsers = mime_parsers[mime]
     found = None
     for parser in parsers:
         try:
             found = parser(r, True)
             break
-        except InvalidSegmentParser:
+        except errors.InvalidSegmentParser as e:
+            if verbose:
+                log.info("parser isn't %s: %s" % (parser.__name__, str(e)))
             pass
     return found
 
+
 def guess_parser_for_system(mime_base, r):
     for mime in mime_parse_order:
         if mime.startswith(mime_base):
             p = guess_parser_for_mime(mime, r)
             if p is not None:
+                mime = guess_detail_for_mime(mime, r, p)
                 return mime, p
     return None, None
 
+
 def iter_parsers(r):
+    container = guess_container(r.data)
+    if container is not None:
+        r = SegmentData(container.unpacked)
+    mime, parser = guess_parser_by_size(r)
+    if parser is None:
+        for mime in mime_parse_order:
+            p = guess_parser_for_mime(mime, r)
+            if p is not None:
+                parser = p
+                mime = guess_detail_for_mime(mime, r, p)
+                break
+    return mime, parser
+
+
+def parsers_for_filename(name):
+    matches = []
     for mime in mime_parse_order:
-        p = guess_parser_for_mime(mime, r)
-        if p is not None:
-            return mime, p
-    return None, None
+        parsers = mime_parsers[mime]
+        found = None
+        for parser in parsers:
+            log.debug("parser: %s = %s" % (mime, parser))
+    n = name.lower()
+    if n.endswith(".atr"):
+        matches.append(KBootImage)
+    elif n.endswith(".dsk"):
+        matches.append(StandardDeliveryImage)
+    else:
+        try:
+            _, name = name.rsplit(".", 1)
+        except ValueError:
+            pass
+        raise errors.InvalidDiskImage("no disk image formats that match '%s'" % name)
+    return matches
 
 
 mime_parsers = {
@@ -123,55 +320,107 @@ mime_parsers = {
         SpartaDosSegmentParser,
         AtariDosSegmentParser,
         AtariBootDiskSegmentParser,
+        AtariUnidentifiedSegmentParser,
         ],
     "application/vnd.atari8bit.xex": [
         XexSegmentParser,
         ],
+    "application/vnd.atari8bit.cart": [
+        Atari8bitCartParser,
+        ],
+    "application/vnd.atari5200.cart": [
+        Atari5200CartParser,
+        ],
+    "application/vnd.atari2600.cart": [
+        Atari2600CartParser,
+        ],
+    "application/vnd.atari2600.starpath": [
+        Atari2600StarpathParser,
+        ],
+    "application/vnd.vectrex": [
+        VectrexParser,
+        ],
     "application/vnd.mame_rom": [
         MameZipParser,
         ],
+    "application/vnd.rom": [
+        RomParser,
+        ],
+    "application/vnd.apple2.dsk": [
+        Dos33SegmentParser,
+        ProdosSegmentParser,
+        ],
+    "application/vnd.apple2.bin": [
+        Dos33BinSegmentParser,
+        ],
     }
 
+# Note: Atari 2600 scanning not performed here because it will match everything
 mime_parse_order = [
     "application/vnd.atari8bit.atr",
     "application/vnd.atari8bit.xex",
-    "CARTS", # Will get filled in below
+    "application/vnd.atari8bit.cart",
+    "application/vnd.atari5200.cart",
     "application/vnd.mame_rom",
+    "application/vnd.apple2.dsk",
+    "application/vnd.apple2.bin",
+    "application/vnd.rom",
+    ]
+
+# different than the above mime_parse_order, this list is the order in which
+# the mime parsers will appear in a UI. Some, like the vectrex and atari2600
+# parsers, aren't included in the parse order because they will identify
+# many things incorrectly. They are used only when parsing by size and
+# signature.
+mime_display_order = [
+    "application/vnd.atari8bit.atr",
+    "application/vnd.atari8bit.xex",
+    "application/vnd.atari8bit.cart",
+    "application/vnd.atari5200.cart",
+    "application/vnd.atari2600.cart",
+    "application/vnd.atari2600.starpath",
+    "application/vnd.vectrex",
+    "application/vnd.mame_rom",
+    "application/vnd.apple2.dsk",
+    "application/vnd.apple2.bin",
+    "application/vnd.rom",
     ]
 
 pretty_mime = {
     "application/vnd.atari8bit.atr": "Atari 8-bit Disk Image",
     "application/vnd.atari8bit.xex": "Atari 8-bit Executable",
-    "application/vnd.mame_rom": "MAME"
+    "application/vnd.atari8bit.cart": "Atari 8-bit Cartridge",
+    "application/vnd.atari5200.cart": "Atari 5200 Cartridge",
+    "application/vnd.atari2600.cart": "Atari 2600 Cartridge",
+    "application/vnd.atari2600.starpath": "Atari 2600 Starpath Cassette",
+    "application/vnd.vectrex": "GCE Vectrex Cartridge",
+    "application/vnd.mame_rom": "MAME",
+    "application/vnd.rom": "ROM Image",
+    "application/vnd.apple2.dsk": "Apple ][ Disk Image",
+    "application/vnd.apple2.bin": "Apple ][ Binary",
 }
 
 grouped_carts = get_known_carts()
 sizes = sorted(grouped_carts.keys())
-cart_order = []
 for k in sizes:
-    if k > 128:
-        key = "application/vnd.atari8bit.large_cart"
-        pretty = "Atari 8-bit Large Cartridge"
-    else:
-        key = "application/vnd.atari8bit.%dkb_cart" % k
-        pretty = "Atari 8-bit %dKB Cartridge" % k
-    if key not in mime_parsers:
-        cart_order.append(key)
-        pretty_mime[key] = pretty
-        mime_parsers[key] = []
     for c in grouped_carts[k]:
         t = c[0]
-        kclass = type("AtariCartSegmentParser%d" % t, (AtariCartSegmentParser,), {'cart_type': t, 'cart_info': c, 'menu_name': "%s Cartridge" % c[1]})
+        name = c[1]
+        if "5200" in name:
+            key = "application/vnd.atari5200.cart"
+            kclass = type("Atari5200CartSegmentParser%d" % t, (Atari5200CartParser, AtariCartSegmentParser), {'cart_type': t, 'cart_info': c, 'menu_name': "%s Cartridge" % name})
+        else:
+            key = "application/vnd.atari8bit.cart"
+            kclass = type("Atari8bitCartSegmentParser%d" % t, (Atari8bitCartParser, AtariCartSegmentParser), {'cart_type': t, 'cart_info': c, 'menu_name': "%s Cartridge" % name})
         mime_parsers[key].append(kclass)
-i = mime_parse_order.index("CARTS")
-mime_parse_order[i:i+1] = cart_order
 
 
 known_segment_parsers = [DefaultSegmentParser]
-for mime in mime_parse_order:
+for mime in mime_display_order:
     known_segment_parsers.extend(mime_parsers[mime])
 
+
 def iter_known_segment_parsers():
     yield "application/octet-stream", "", [DefaultSegmentParser]
-    for mime in mime_parse_order:
+    for mime in mime_display_order:
         yield mime, pretty_mime[mime], mime_parsers[mime]

atrcopy/spartados.py

@@ -1,9 +1,8 @@
 import numpy as np
 
-from errors import *
-from ataridos import AtariDosDirent, XexSegment
-from diskimages import DiskImageBase
-from segments import DefaultSegment, EmptySegment, ObjSegment, RawSectorsSegment, SegmentSaver
+from . import errors
+from .ataridos import AtariDosDirent, AtariDosDiskImage, XexSegment
+from .segments import DefaultSegment, EmptySegment, ObjSegment, RawSectorsSegment, SegmentSaver
 
 import logging
 log = logging.getLogger(__name__)
@@ -31,7 +30,7 @@ class SpartaDosDirent(AtariDosDirent):
             # rather the boot sector so it must be specified here.
             self.starting_sector = starting_sector
             self.is_sane = self.sanity_check(image)
-    
+
     def __str__(self):
         output = "o" if self.opened_output else "."
         subdir = "D" if self.is_dir else "."
@@ -39,8 +38,8 @@ class SpartaDosDirent(AtariDosDirent):
         deleted = "d" if self.deleted else "."
         locked = "*" if self.locked else " "
         flags = "%s%s%s%s%s %03d" % (output, subdir, in_use, deleted, locked, self.starting_sector)
-        return "File #%-2d (%s) %-8s%-3s  %8d  %s" % (self.file_num, flags, self.filename, self.ext, self.length, self.str_timestamp)
-    
+        return "File #%-2d (%s) %-8s%-3s  %8d  %s" % (self.file_num, flags, self.basename.decode('latin1'), self.ext.decode('latin1'), self.length, self.str_timestamp)
+
     @property
     def verbose_info(self):
         flags = []
@@ -50,12 +49,12 @@ class SpartaDosDirent(AtariDosDirent):
         if self.deleted: flags.append("DEL")
         if self.locked: flags.append("LOCK")
         return "flags=[%s]" % ", ".join(flags)
-    
-    def parse_raw_dirent(self, image, bytes):
-        if bytes is None:
+
+    def parse_raw_dirent(self, image, data):
+        if data is None:
             return
-        values = bytes.view(dtype=self.format)[0]  
-        flag = values['status']
+        values = data.view(dtype=self.format)[0]
+        flag = values[0]
         self.flag = flag
         self.locked = (flag&0x1) > 0
         self.hidden = (flag&0x10) > 0
@@ -64,24 +63,24 @@ class SpartaDosDirent(AtariDosDirent):
         self.deleted = (flag&0b10000) > 0
         self.is_dir = (flag&0b100000) > 0
         self.opened_output = (flag&0b10000000) > 0
-        self.starting_sector = int(values['sector'])
-        self.filename = str(values['filename']).rstrip()
+        self.starting_sector = int(values[1])
+        self.basename = bytes(values[4]).rstrip()
         if self.is_dir:
-            self.ext = ""
+            self.ext = b""
         else:
-            self.ext = str(values['ext']).rstrip()
-        self.length = 256*256*values['len_h'] + values['len_l']
-        self.date_array = tuple(bytes[17:20])
-        self.time_array = tuple(bytes[20:23])
+            self.ext = bytes(values[5]).rstrip()
+        self.length = 256*256*values[3] + values[2]
+        self.date_array = tuple(data[17:20])
+        self.time_array = tuple(data[20:23])
         self.is_sane = self.sanity_check(image)
-    
+
     def sanity_check(self, image):
         if not self.in_use:
             return True
         if not image.header.sector_is_valid(self.starting_sector):
             return False
         return True
-    
+
     @property
     def str_timestamp(self):
         str_date = "%d/%d/%d" % self.date_array
@@ -91,11 +90,11 @@ class SpartaDosDirent(AtariDosDirent):
     def start_read(self, image):
         if not self.is_sane:
             log.debug("Invalid directory entry '%s', starting_sector=%s" % (str(self), self.starting_sector))
-            raise InvalidDirent("Invalid directory entry '%s'" % str(self))
+            raise errors.InvalidDirent("Invalid directory entry '%s'" % str(self))
         self.sector_map = image.get_sector_map(self.starting_sector)
         self.sector_map_index = 0
         self.length_remaining = self.length
-    
+
     def read_sector(self, image):
         sector = self.sector_map[self.sector_map_index]
         if sector == 0:
@@ -107,18 +106,18 @@ class SpartaDosDirent(AtariDosDirent):
         return raw[0:num_data_bytes], sector == 0, pos, num_data_bytes
 
 
-class SpartaDosDiskImage(DiskImageBase):
+class SpartaDosDiskImage(AtariDosDiskImage):
     def __init__(self, *args, **kwargs):
         self.first_bitmap = 0
         self.num_bitmap = 0
         self.root_dir = 0
         self.root_dir_dirent = None
         self.fs_version = 0
-        DiskImageBase.__init__(self, *args, **kwargs)
-    
+        AtariDosDiskImage.__init__(self, *args, **kwargs)
+
     def __str__(self):
         return "%s Sparta DOS Format: %d usable sectors (%d free), %d files" % (self.header, self.total_sectors, self.unused_sectors, len(self.files))
-    
+
     boot_record_type = np.dtype([
         ('unused', 'u1'),
         ('num_boot', 'u1'),
@@ -138,12 +137,12 @@ class SpartaDosDiskImage(DiskImageBase):
         ('sector_size','u1'),
         ('fs_version','u1'),
         ])
-    
+
     sector_size_map = {0: 256,
                        1: 512,
                        0x80: 128,
                        }
-    
+
     def get_boot_sector_info(self):
         data, style = self.get_sectors(1)
         values = data[0:33].view(dtype=self.boot_record_type)[0]
@@ -160,11 +159,11 @@ class SpartaDosDiskImage(DiskImageBase):
         num = self.header.max_sectors
         self.is_sane = self.total_sectors == num and values['first_free'] <= num and self.first_bitmap <= num and self.root_dir <= num and self.fs_version in [0x11, 0x20, 0x21] and self.sector_size != -1
         if not self.is_sane:
-            raise InvalidDiskImage("Invalid SpartaDos parameters in boot header")
+            raise errors.InvalidDiskImage("Invalid SpartaDos parameters in boot header")
 
     def get_vtoc(self):
         pass
-    
+
     def get_directory(self):
         self.files = []
         dir_map = self.get_sector_map(self.root_dir)
@@ -178,7 +177,7 @@ class SpartaDosDiskImage(DiskImageBase):
             dirent = SpartaDosDirent(self, filenum + 1, s[i:i + 23])
             self.files.append(dirent)
         self.root_dir_dirent = d
-    
+
     def get_boot_segments(self):
         segments = []
         num = min(self.num_boot, 1)
@@ -192,7 +191,7 @@ class SpartaDosDiskImage(DiskImageBase):
             code = ObjSegment(r[43:], 0, 0, addr + 43, addr + len(r), name="Boot Code")
             segments.extend([header, code])
         return segments
-    
+
     def get_vtoc_segments(self):
         r = self.rawdata
         segments = []
@@ -207,7 +206,7 @@ class SpartaDosDiskImage(DiskImageBase):
         segment = RawSectorsSegment(r[start:start+count], self.first_bitmap, self.num_bitmap, count, 0, 0, self.sector_size, name="Bitmap")
         segments.append(segment)
         return segments
-    
+
     def get_sector_map(self, sector):
         m = None
         while sector > 0:
@@ -218,7 +217,7 @@ class SpartaDosDiskImage(DiskImageBase):
             else:
                 m = np.hstack((m, b[4:].view(dtype='<u2')))
         return m
-    
+
     def get_directory_segments(self):
         dirent = self.root_dir_dirent
         segment = self.get_file_segment(dirent)
@@ -226,21 +225,21 @@ class SpartaDosDiskImage(DiskImageBase):
         segment.map_width = 23
         segments = [segment]
         return segments
-    
+
     def get_file_segment(self, dirent):
         byte_order = []
         dirent.start_read(self)
         while True:
             bytes, last, pos, size = dirent.read_sector(self)
             if not last:
-                byte_order.extend(range(pos, pos + size))
+                byte_order.extend(list(range(pos, pos + size)))
             else:
                 break
         if len(byte_order) > 0:
-            name = "%s %d@%d %s" % (dirent.get_filename(), dirent.length, dirent.starting_sector, dirent.str_timestamp)
-            verbose_name = "%s (%d bytes, sector map@%d) %s %s" % (dirent.get_filename(), dirent.length, dirent.starting_sector, dirent.verbose_info, dirent.str_timestamp)
+            name = "%s %d@%d %s" % (dirent.filename, dirent.length, dirent.starting_sector, dirent.str_timestamp)
+            verbose_name = "%s (%d bytes, sector map@%d) %s %s" % (dirent.filename, dirent.length, dirent.starting_sector, dirent.verbose_info, dirent.str_timestamp)
             raw = self.rawdata.get_indexed(byte_order)
             segment = DefaultSegment(raw, name=name, verbose_name=verbose_name)
         else:
-            segment = EmptySegment(self.rawdata, name=dirent.get_filename(), error=dirent.str_timestamp)
+            segment = EmptySegment(self.rawdata, name=dirent.filename, error=dirent.str_timestamp)
         return segment

atrcopy/standard_delivery.py

@@ -0,0 +1,172 @@
+import numpy as np
+
+from . import errors
+from .segments import SegmentData
+from .diskimages import BaseHeader, DiskImageBase
+
+import logging
+log = logging.getLogger(__name__)
+
+
+class StandardDeliveryHeader(BaseHeader):
+    file_format = "Apple ][ Standard Delivery"
+
+    def __init__(self, bytes=None, sector_size=256, create=False):
+        BaseHeader.__init__(self, sector_size, create=create)
+        if bytes is None:
+            return
+
+        data = bytes[0:5]
+        if np.all(data == (0x01, 0xa8, 0xee, 0x06, 0x08)):
+            log.debug("Found 48k loader")
+        else:
+            raise errors.InvalidDiskImage("No %s boot header" % self.file_format)
+
+    def __str__(self):
+        return "Standard Delivery Boot Disk (size=%d (%dx%dB)" % (self.file_format, self.image_size, self.max_sectors, self.sector_size)
+
+    def check_size(self, size):
+        if size != 143360:
+            raise errors.InvalidDiskImage("Incorrect size for Standard Delivery image")
+        self.image_size = size
+        self.tracks_per_disk = 35
+        self.sectors_per_track = 16
+        self.max_sectors = self.tracks_per_disk * self.sectors_per_track
+
+
+class StandardDeliveryImage(DiskImageBase):
+    def __str__(self):
+        return str(self.header)
+
+    def read_header(self):
+        self.header = StandardDeliveryHeader(self.bytes[0:256])
+
+    @classmethod
+    def new_header(cls, diskimage, format="DSK"):
+        if format.lower() == "dsk":
+            header = StandardDeliveryHeader(create=True)
+            header.check_size(diskimage.size)
+        else:
+            raise RuntimeError("Unknown header type %s" % format)
+        return header
+
+    def check_size(self):
+        pass
+
+    def get_boot_sector_info(self):
+        pass
+
+    def get_vtoc(self):
+        pass
+
+    def get_directory(self, directory=None):
+        pass
+
+    @classmethod
+    def create_boot_image(cls, segments, run_addr=None):
+        raw = SegmentData(np.zeros([143360], dtype=np.uint8))
+        dsk = cls(raw, create=True)
+        if run_addr is None:
+            run_addr = segments[0].origin
+
+        chunks = []
+
+        for s in segments:
+            # find size in 256 byte chunks that start on a page boundary
+            # since the loader only deals with page boundaries
+            origin = s.origin
+            chunk_start, padding = divmod(origin, 256)
+            if (chunk_start == 0x20 or chunk_start == 0x40) and padding == 1:
+                show_hgr = False
+                padding = 0
+            else:
+                show_hgr = True
+            size = ((len(s) + padding + 255) // 256) * 256
+            chunk = np.zeros([size], dtype=np.uint8)
+            chunk[padding:padding + len(s)] = s[:]
+            chunks.append((chunk_start, chunk, show_hgr))
+            print("segment: %s, pages=%d" % (str(s), len(chunk) // 256))
+            log.debug(" last chunk=%s" % str(chunks[-1]))
+
+        # break up the chunks into sectors
+
+        # NOTE: fstbt implied that the sector order was staggered, but in
+        # AppleWin, by trial and error, works with the following order
+
+        # index = 1  # on the first track, sector 0 is reserved for boot sector
+        # sector_order = [0, 2, 4, 6, 8, 10, 12, 14, 1, 3, 5, 7, 9, 11, 13, 15]
+
+        index = 1
+        sector_order = [0, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 15]
+        track = 0
+        count = 0
+
+        sector_list = []
+        address_list = [0xd1]
+
+        boot_sector = dsk.header.create_sector()
+        boot_sector.sector_num = 0
+        boot_sector.track_num = 1
+        sector_list.append(boot_sector)
+        first_page_1 = True
+
+        for chunk_start, chunk_data, show_hgr in chunks:
+            count = len(chunk_data) // 256
+            if chunk_start == 0x20 and count == 32 and first_page_1:
+                # Assume this is an HGR screen, use interesting load effect,
+                # not the usual venetian blind
+                chunk_hi = [0x20, 0x24, 0x28, 0x2c, 0x30, 0x34, 0x38, 0x3c, 0x21, 0x25, 0x29, 0x2d, 0x31, 0x35, 0x39, 0x3d, 0x22, 0x26, 0x2a, 0x2e, 0x32, 0x36, 0x3a, 0x3e, 0x23, 0x27, 0x2b, 0x2f, 0x33, 0x37, 0x3b, 0x3f]
+                address_order = chunk_hi
+            else:
+                chunk_hi = range(chunk_start, chunk_start + count)
+                if len(chunk_hi) > 2:
+                    address_order = []
+                    for n in range(0, count, 32):
+                        subset = chunk_hi[n:n+32]
+                        if len(subset) > 2:
+                            address_order.extend([chunk_hi[n], 0xe0 + len(subset) - 1])
+                        else:
+                            address_order.extend(subset)
+                else:
+                    address_order = chunk_hi
+
+            for n in range(count):
+                i = (chunk_hi[n] - chunk_start) * 256
+                sector = dsk.header.create_sector(chunk_data[i:i+256])
+                sector.sector_num = dsk.header.sector_from_track(track, sector_order[index])
+                count += 1
+                #sector.sector_num = count
+                sector_list.append(sector)
+                # sector.data[0] = sector.sector_num
+                # sector.data[1] = hi
+                # sector.data[2:16] = 0xff
+                log.debug("%s at %02x00: %s ..." % (sector_list[-1], address_list[-1], " ".join(["%02x" % h for h in chunk_data[i:i + 16]])))
+                index += 1
+                if index >= len(sector_order):
+                    index = 0
+                    track += 1
+            address_list.extend(address_order)
+            if show_hgr:
+                if chunk_start == 0x40:
+                    address_list.append(0xd2)
+                elif chunk_start == 0x20:
+                    if not first_page_1:
+                        address_list.append(0xd1)
+                    first_page_1 = False
+
+        print("fstbt commands: %s" % ", ".join(["%02x" % i for i in address_list]))
+        boot_code = get_fstbt_code(boot_sector.data, address_list, run_addr)
+
+        dsk.write_sector_list(sector_list)
+
+        return dsk
+
+from . fstbt import fstbt
+
+def get_fstbt_code(data, address_list, run_addr):
+    pointer = len(fstbt)
+    data[0:pointer] = np.frombuffer(fstbt, dtype=np.uint8)
+    hi, lo = divmod(run_addr, 256)
+    data[pointer:pointer + 2] = (lo, hi)
+    address_list.append(0xc0)  # last sector flag
+    data[pointer + 2:pointer + 2 + len(address_list)] = address_list

atrcopy/templates/dos2dd.atr.inf

@@ -0,0 +1 @@
+{"type": "new file", "task": "hex_edit", "description": "Atari 8-bit DOS 2 double density (180K), empty VTOC", "label": "Atari DOS 2 DD (180K) blank image", "ext": "atr"}

atrcopy/templates/dos2ed+2.5.atr.inf

@@ -0,0 +1 @@
+{"type": "new file", "task": "hex_edit", "description": "Atari 8-bit DOS 2 enhanced density (130K) DOS 2.5 system disk", "label": "Atari DOS 2.5 ED (130K) system disk", "ext": "atr"}

atrcopy/templates/dos2ed.atr.inf

@@ -0,0 +1 @@
+{"type": "new file", "task": "hex_edit", "description": "Atari 8-bit DOS 2 enhanced density (130K), empty VTOC", "label": "Atari DOS 2 ED (130K) blank image", "ext": "atr"}

atrcopy/templates/dos2sd+2.0s.atr.inf

@@ -0,0 +1 @@
+{"type": "new file", "task": "hex_edit", "description": "Atari 8-bit DOS 2 single density (90K) DOS 2.0S system disk", "label": "Atari DOS 2.0S SD (90K) system disk", "ext": "atr"}

atrcopy/templates/dos2sd.atr.inf

@@ -0,0 +1 @@
+{"type": "new file", "task": "hex_edit", "description": "Atari 8-bit DOS 2 single density (90K), empty VTOC", "label": "Atari DOS 2 SD (90K) blank image", "ext": "atr"}

atrcopy/templates/dos33.dsk.inf

@@ -0,0 +1 @@
+{"type": "new file", "task": "hex_edit", "description": "Apple ][ DOS 3.3 (140K) standard RWTS, empty VTOC", "label": "Apple DOS 3.3 (140K) blank image", "ext": "dsk"}

atrcopy/templates/dos33autobrun.dsk.inf

@@ -0,0 +1 @@
+{"type": "new file", "task": "hex_edit", "description": "Apple ][ DOS 3.3 (140K) disk image for binary program development: HELLO sets fullscreen HGR and calls BRUN on user-supplied AUTOBRUN binary file", "label": "Apple DOS 3.3 (140K) AUTOBRUN image", "ext": "dsk"}

atrcopy/utils.py

@@ -1,14 +1,36 @@
 import types
+import uuid as stdlib_uuid
 
 import numpy as np
 
+from . import errors
+
+import logging
+log = logging.getLogger(__name__)
+try:  # Expensive debugging
+    _xd = _expensive_debugging
+except NameError:
+    _xd = False
+
+
+def uuid():
+    u = stdlib_uuid.uuid4()
+
+    # Force it to use unicode(py2) or str(py3) so it isn't serialized as
+    # future.types.newstr.newstr on py2
+    try:
+        u = unicode(u)
+    except:
+        u = str(u)
+    return u
+
 
 def to_numpy(value):
     if type(value) is np.ndarray:
         return value
-    elif type(value) is types.StringType:
-        return np.fromstring(value, dtype=np.uint8)
-    elif type(value) is types.ListType:
+    elif type(value) is bytes:
+        return np.copy(np.frombuffer(value, dtype=np.uint8))
+    elif type(value) is list:
     	return np.asarray(value, dtype=np.uint8)
     raise TypeError("Can't convert to numpy data")
 
@@ -17,3 +39,330 @@ def to_numpy_list(value):
     if type(value) is np.ndarray:
         return value
     return np.asarray(value, dtype=np.uint32)
+
+
+def text_to_int(text, default_base="hex"):
+    """ Convert text to int, raising exeception on invalid input
+    """
+    if text.startswith("0x"):
+        value = int(text[2:], 16)
+    elif text.startswith("$"):
+        value = int(text[1:], 16)
+    elif text.startswith("#"):
+        value = int(text[1:], 10)
+    elif text.startswith("%"):
+        value = int(text[1:], 2)
+    else:
+        if default_base == "dec":
+            value = int(text)
+        else:
+            value = int(text, 16)
+    return value
+
+
+class WriteableSector:
+    def __init__(self, sector_size, data=None, num=-1):
+        self._sector_num = num
+        self._next_sector = 0
+        self.sector_size = sector_size
+        self.file_num = 0
+        self.data = np.zeros([sector_size], dtype=np.uint8)
+        self.used = 0
+        self.ptr = self.used
+        if data is not None:
+            self.add_data(data)
+
+    def __str__(self):
+        return "sector=%d next=%d size=%d used=%d" % (self._sector_num, self._next_sector, self.sector_size, self.used)
+
+    @property
+    def sector_num(self):
+        return self._sector_num
+
+    @sector_num.setter
+    def sector_num(self, value):
+        self._sector_num = value
+
+    @property
+    def next_sector_num(self):
+        return self._next_sector_num
+
+    @sector_num.setter
+    def next_sector_num(self, value):
+        self._next_sector_num = value
+
+    @property
+    def space_remaining(self):
+        return self.sector_size - self.ptr
+
+    @property
+    def is_empty(self):
+        return self.ptr == 0
+
+    def add_data(self, data):
+        count = len(data)
+        if self.ptr + count > self.sector_size:
+            count = self.space_remaining
+        self.data[self.ptr:self.ptr + count] = data[0:count]
+        self.ptr += count
+        self.used += count
+        return data[count:]
+
+
+class BaseSectorList:
+    def __init__(self, header):
+        self.header = header
+        self.sector_size = header.sector_size
+        self.sectors = []
+
+    def __len__(self):
+        return len(self.sectors)
+
+    def __str__(self):
+        return "\n".join(" %d: %s" % (i, str(s)) for i, s in enumerate(self))
+
+    def __getitem__(self, index):
+        if index < 0 or index >= len(self):
+            raise IndexError
+        return self.sectors[index]
+
+    @property
+    def num_sectors(self):
+        return len(self.sectors)
+
+    @property
+    def first_sector(self):
+        if self.sectors:
+            return self.sectors[0].sector_num
+        return -1
+
+    @property
+    def bytes_used(self):
+        size = 0
+        for s in self:
+            size += s.used
+        return size
+
+    def append(self, sector):
+        self.sectors.append(sector)
+
+    def extend(self, sectors):
+        self.sectors.extend(sectors)
+
+
+class Dirent:
+    """Abstract base class for a directory entry
+
+    """
+
+    def __init__(self, file_num=0):
+        self.file_num = file_num
+
+    def __eq__(self, other):
+        raise errors.NotImplementedError
+
+    def extra_metadata(self, image):
+        raise errors.NotImplementedError
+
+    def mark_deleted(self):
+        raise errors.NotImplementedError
+
+    def parse_raw_dirent(self, image, bytes):
+        raise errors.NotImplementedError
+
+    def encode_dirent(self):
+        raise errors.NotImplementedError
+
+    def get_sectors_in_vtoc(self, image):
+        raise errors.NotImplementedError
+
+    def start_read(self, image):
+        raise errors.NotImplementedError
+
+    def read_sector(self, image):
+        raise errors.NotImplementedError
+
+
+class Directory(BaseSectorList):
+    def __init__(self, header, num_dirents=-1, sector_class=WriteableSector):
+        BaseSectorList.__init__(self, header)
+        self.sector_class = sector_class
+        self.num_dirents = num_dirents
+        # number of dirents may be unlimited, so use a dict instead of a list
+        self.dirents = {}
+
+    def set(self, index, dirent):
+        self.dirents[index] = dirent
+        if _xd: log.debug("set dirent #%d: %s" % (index, dirent))
+
+    def get_free_dirent(self):
+        used = set()
+        d = list(self.dirents.items())
+        if d:
+            d.sort()
+            for i, dirent in d:
+                if not dirent.in_use:
+                    return i
+                used.add(i)
+            if self.num_dirents > 0 and (len(used) >= self.num_dirents):
+                raise errors.NoSpaceInDirectory()
+            i += 1
+        else:
+            i = 0
+        used.add(i)
+        return i
+
+    def add_dirent(self, filename, filetype):
+        index = self.get_free_dirent()
+        dirent = self.dirent_class(None)
+        dirent.set_values(filename, filetype, index)
+        self.set(index, dirent)
+        return dirent
+
+    def find_dirent(self, filename):
+        if hasattr(filename, "filename"):
+            # we've been passed a dirent instead of a filename
+            for dirent in list(self.dirents.values()):
+                if dirent == filename:
+                    return dirent
+        else:
+            for dirent in list(self.dirents.values()):
+                if filename == dirent.filename:
+                    return dirent
+        raise errors.FileNotFound("%s not found on disk" % filename)
+
+    def save_dirent(self, image, dirent, vtoc, sector_list):
+        vtoc.assign_sector_numbers(dirent, sector_list)
+        dirent.add_metadata_sectors(vtoc, sector_list, image.header)
+        dirent.update_sector_info(sector_list)
+        self.calc_sectors(image)
+
+    def remove_dirent(self, image, dirent, vtoc, sector_list):
+        vtoc.free_sector_list(sector_list)
+        dirent.mark_deleted()
+        self.calc_sectors(image)
+
+    @property
+    def dirent_class(self):
+        raise errors.NotImplementedError
+
+    def calc_sectors(self, image):
+        self.sectors = []
+        self.current_sector = self.get_dirent_sector()
+        self.encode_index = 0
+
+        d = list(self.dirents.items())
+        d.sort()
+        # there may be gaps, so fill in missing entries with blanks
+        current = 0
+        for index, dirent in d:
+            for missing in range(current, index):
+                if _xd: log.debug("Encoding empty dirent at %d" % missing)
+                data = self.encode_empty()
+                self.store_encoded(data)
+            if _xd: log.debug("Encoding dirent: %s" % dirent)
+            data = self.encode_dirent(dirent)
+            self.store_encoded(data)
+            current = index + 1
+        self.finish_encoding(image)
+
+    def get_dirent_sector(self):
+        return self.sector_class(self.sector_size)
+
+    def encode_empty(self):
+        raise errors.NotImplementedError
+
+    def encode_dirent(self, dirent):
+        raise errors.NotImplementedError
+
+    def store_encoded(self, data):
+        while True:
+            if _xd: log.debug("store_encoded: %d bytes in %s" % (len(data), self.current_sector))
+            data = self.current_sector.add_data(data)
+            if len(data) > 0:
+                self.sectors.append(self.current_sector)
+                self.current_sector = self.get_dirent_sector()
+            else:
+                break
+
+    def finish_encoding(self, image):
+        if not self.current_sector.is_empty:
+            self.sectors.append(self.current_sector)
+        self.set_sector_numbers(image)
+
+    def set_sector_numbers(self, image):
+        raise errors.NotImplementedError
+
+
+class VTOC(BaseSectorList):
+    def __init__(self, header, segments=None):
+        BaseSectorList.__init__(self, header)
+
+        # sector map: 1 is free, 0 is allocated
+        self.sector_map = np.zeros([1280], dtype=np.uint8)
+        if segments is not None:
+            self.parse_segments(segments)
+
+    def __str__(self):
+        return "%s\n (%d free)" % ("\n".join(["track %02d: %s" % (i, line) for i, line in enumerate(str(self.sector_map[self.header.starting_sector_label:(self.header.tracks_per_disk*self.header.sectors_per_track) + self.header.starting_sector_label].reshape([self.header.tracks_per_disk,self.header.sectors_per_track])).splitlines())]), self.num_free_sectors)
+
+    @property
+    def num_free_sectors(self):
+        free = np.where(self.sector_map == 1)[0]
+        return len(free)
+
+    def iter_free_sectors(self):
+        for i, pos, size in self.header.iter_sectors():
+            if self.sector_map[i] == 1:
+                yield i, pos, size
+
+    def parse_segments(self, segments):
+        raise errors.NotImplementedError
+
+    def assign_sector_numbers(self, dirent, sector_list):
+        """ Map out the sectors and link the sectors together
+
+        raises NotEnoughSpaceOnDisk if the whole file won't fit. It will not
+        allow partial writes.
+        """
+        num = len(sector_list)
+        order = self.reserve_space(num)
+        if len(order) != num:
+            raise errors.InvalidFile("VTOC reserved space for %d sectors. Sectors needed: %d" % (len(order), num))
+        file_length = 0
+        last_sector = None
+        for sector, sector_num in zip(sector_list.sectors, order):
+            sector.sector_num = sector_num
+            sector.file_num = dirent.file_num
+            file_length += sector.used
+            if last_sector is not None:
+                last_sector.next_sector_num = sector_num
+            last_sector = sector
+        if last_sector is not None:
+            last_sector.next_sector_num = 0
+        sector_list.file_length = file_length
+
+    def reserve_space(self, num):
+        order = []
+        for i in range(num):
+            order.append(self.get_next_free_sector())
+        if _xd: log.debug("Sectors reserved: %s" % order)
+        self.calc_bitmap()
+        return order
+
+    def get_next_free_sector(self):
+        free = np.nonzero(self.sector_map)[0]
+        if len(free) > 0:
+            num = free[0]
+            if _xd: log.debug("Found sector %d free" % num)
+            self.sector_map[num] = 0
+            return num
+        raise errors.NotEnoughSpaceOnDisk("No space left in VTOC")
+
+    def calc_bitmap(self):
+        raise errors.NotImplementedError
+
+    def free_sector_list(self, sector_list):
+        for sector in sector_list:
+            self.sector_map[sector.sector_num] = 1
+        self.calc_bitmap()

gen-sha.py

@@ -0,0 +1,47 @@
+#!/usr/bin/env python
+import os
+import sys
+import hashlib
+from collections import defaultdict
+import pprint
+
+def parse(filename, mime):
+    data = open(filename, 'rb').read()
+    h = hashlib.sha1(data).digest()
+    name = os.path.basename(os.path.splitext(filename)[0])
+    return len(data), h, name
+
+
+
+if __name__ == '__main__':
+    source = "atrcopy/signatures.py"
+    try:
+        with open(source, 'r') as fh:
+            source_text = fh.read()
+    except OSError:
+        source_text = "sha1_signatures = {}"
+    try:
+        exec(source_text)
+    except:
+        raise
+    print(sha1_signatures)
+    mime = sys.argv[1]
+    new_signatures = defaultdict(dict)
+    new_signatures.update(sha1_signatures)
+    for filename in sys.argv[2:]:
+        print(f"parsing {filename}")
+        size, hash_string, name = parse(filename, mime)
+        print(f"{size} {hash_string} {mime} {name}")
+        new_signatures[size][hash_string] = (mime, name)
+    lines = []
+    lines.append("sha1_signatures = {")
+    for k,v in sorted(new_signatures.items()):
+        lines.append(f"{k}: {{")
+        for h,n in sorted(v.items(), key=lambda a:(a[1], a[0])):
+            lines.append(f"  {h}: {n},")
+        lines.append("},")
+    lines.append("}  # end sha1_signatures\n")
+
+    print("\n".join(lines))
+    with open(source, 'w') as fh:
+        fh.write("\n".join(lines))

scripts/atrcopy

@@ -1,6 +1,12 @@
 #!/usr/bin/env python
 
 if __name__ == "__main__":
-    import atrcopy
+    import sys
+    if sys.version_info < (3, 6, 0):
+        print("atrcopy requires Python 3.6 or greater to run; this is Python %s" % ".".join([str(v) for v in sys.version_info[0:2]]))
+        if sys.version_info[0] == 2:
+            print("Python 2 support was dropped with atrcopy 7.0, so you can either use:\n\n    pip install \"atrcopy<7.0\"\n\nto install a version compatible with Python 2, or install Python 3.6 or higher.")
+    else:
+        import atrcopy
     
-    atrcopy.run()
+        atrcopy.run()

setup.py

@@ -1,5 +1,3 @@
-from __future__ import with_statement
-
 import sys
 
 try:
@@ -7,20 +5,8 @@ try:
 except ImportError:
     from distutils.core import setup
 
-try:
-    import atrcopy
-    version = atrcopy.__version__
-except RuntimeError, e:
-    # If numpy isn't present, pull the version number from the error string
-    version = str(e).split()[1]
-
-classifiers = [
-    "Programming Language :: Python :: 2",
-    "Intended Audience :: Developers",
-    "License :: OSI Approved :: GNU General Public License (GPL)",
-    "Topic :: Software Development :: Libraries",
-    "Topic :: Utilities",
-]
+exec(compile(open('atrcopy/_version.py').read(), 'atrcopy/_version.py', 'exec'))
+exec(compile(open('atrcopy/_metadata.py').read(), 'atrcopy/_metadata.py', 'exec'))
 
 with open("README.rst", "r") as fp:
     long_description = fp.read()
@@ -31,17 +17,31 @@ else:
     scripts = ["scripts/atrcopy"]
 
 setup(name="atrcopy",
-      version=version,
-      author="Rob McMullen",
-      author_email="feedback@playermissile.com>",
-      url="https://github.com/robmcmullen/atrcopy",
-      packages=["atrcopy"],
-      scripts=scripts,
-      description="Disk image utilities for Atari 8-bit emulators",
-      long_description=long_description,
-      license="GPL",
-      classifiers=classifiers,
-      install_requires = [
-          'numpy',
-          ],
-      )
+        version=__version__,
+        author=__author__,
+        author_email=__author_email__,
+        url=__url__,
+        packages=["atrcopy"],
+        include_package_data=True,
+        scripts=scripts,
+        entry_points={"sawx.loaders": 'atrcopy = atrcopy.omnivore_loader'},
+        description="Utility to manage file systems on Atari 8-bit (DOS 2) and Apple ][ (DOS 3.3) disk images.",
+        long_description=long_description,
+        license="MPL 2.0",
+        classifiers=[
+            "Programming Language :: Python :: 3.6",
+            "Intended Audience :: Developers",
+            "License :: OSI Approved :: Mozilla Public License 2.0 (MPL 2.0)",
+            "Topic :: Software Development :: Libraries",
+            "Topic :: Utilities",
+        ],
+        python_requires = '>=3.6',
+        install_requires = [
+            'numpy',
+        ],
+        tests_require = [
+            'pytest>3.0',
+            'coverage',
+            'pytest.cov',
+        ],
+    )

test/.coveragerc

@@ -0,0 +1,20 @@
+[run]
+branch = True
+
+[report]
+# Regexes for lines to exclude from consideration
+exclude_lines =
+    # Have to re-enable the standard pragma
+    pragma: no cover
+
+    # Don't complain about missing debug-only code:
+    def __repr__
+    if self\.debug
+
+    # Don't complain if tests don't hit defensive assertion code:
+    raise AssertionError
+    raise NotImplementedError
+
+    # Don't complain if non-runnable code isn't run:
+    if 0:
+    if __name__ == .__main__.:

test/mock.py

@@ -8,6 +8,8 @@ module_dir = os.path.realpath(os.path.abspath(".."))
 if module_dir not in sys.path:
     sys.path.insert(0, module_dir)
 
+print(sys.path)
+
 import pytest
 try:
     slow = pytest.mark.skipif(

test/pytest.ini

@@ -0,0 +1,2 @@
+#[pytest]
+#addopts = --cov=atrcopy --cov-report html --cov-report term

test/test_add_file.py

@@ -0,0 +1,169 @@
+from __future__ import print_function
+from builtins import object
+import numpy as np
+
+from mock import *
+
+from atrcopy import SegmentData, AtariDosDiskImage, Dos33DiskImage
+from atrcopy import errors
+
+
+class BaseFilesystemModifyTest:
+    diskimage_type = None
+    sample_data = None
+    num_files_in_sample = 0
+
+    def setup(self):
+        rawdata = SegmentData(self.sample_data.copy())
+        self.image = self.diskimage_type(rawdata)
+
+    def check_entries(self, entries, prefix="TEST", save=None):
+        orig_num_files = len(self.image.files)
+        filenames = []
+        count = 1
+        for data in entries:
+            filename = "%s%d.BIN" % (prefix, count)
+            self.image.write_file(filename, None, data)
+            assert len(self.image.files) == orig_num_files + count
+            data2 = np.frombuffer(self.image.find_file(filename), dtype=np.uint8)
+            assert np.array_equal(data, data2[0:len(data)])
+            count += 1
+
+        # loop over them again to make sure data wasn't overwritten
+        count = 1
+        for data in entries:
+            filename = "%s%d.BIN" % (prefix, count)
+            data2 = np.frombuffer(self.image.find_file(filename), dtype=np.uint8)
+            assert np.array_equal(data, data2[0:len(data)])
+            count += 1
+            filenames.append(filename)
+
+        if save is not None:
+            self.image.save(save)
+
+        return filenames
+
+    def test_small(self):
+        assert len(self.image.files) == self.num_files_in_sample
+
+        data = np.asarray([0xff, 0xff, 0x00, 0x60, 0x01, 0x60, 1, 2], dtype=np.uint8)
+        self.image.write_file("TEST.XEX", None, data)
+        assert len(self.image.files) == self.num_files_in_sample + 1
+
+        data2 = np.frombuffer(self.image.find_file("TEST.XEX"), dtype=np.uint8)
+        assert np.array_equal(data, data2[0:len(data)])
+
+    def test_50k(self):
+        assert len(self.image.files) == self.num_files_in_sample
+
+        data = np.arange(50*1024, dtype=np.uint8)
+        self.image.write_file("RAMP50K.BIN", None, data)
+        assert len(self.image.files) == self.num_files_in_sample + 1
+
+        data2 = self.image.find_file("RAMP50K.BIN")
+        assert data.tostring() == data2
+
+    def test_many_small(self):
+        entries = [
+            np.asarray([0xff, 0xff, 0x00, 0x60, 0x01, 0x60, 1, 2], dtype=np.uint8),
+            np.arange(1*1024, dtype=np.uint8),
+            np.arange(2*1024, dtype=np.uint8),
+            np.arange(3*1024, dtype=np.uint8),
+            np.arange(4*1024, dtype=np.uint8),
+            np.arange(5*1024, dtype=np.uint8),
+            np.arange(6*1024, dtype=np.uint8),
+            np.arange(7*1024, dtype=np.uint8),
+            np.arange(8*1024, dtype=np.uint8),
+            np.arange(9*1024, dtype=np.uint8),
+            np.arange(10*1024, dtype=np.uint8),
+            ]
+        self.check_entries(entries, save="many_small.atr")
+
+    def test_big_failure(self):
+        assert len(self.image.files) == self.num_files_in_sample
+
+        data = np.arange(50*1024, dtype=np.uint8)
+        self.image.write_file("RAMP50K.BIN", None, data)
+        assert len(self.image.files) == self.num_files_in_sample + 1
+        with pytest.raises(errors.NotEnoughSpaceOnDisk):
+            huge = np.arange(500*1024, dtype=np.uint8)
+            self.image.write_file("RAMP500K.BIN", None, huge)
+        assert len(self.image.files) == self.num_files_in_sample + 1
+
+    def test_delete(self):
+        entries1 = [
+            np.arange(3*1024, dtype=np.uint8),
+            np.arange(3*1024, dtype=np.uint8),
+            np.arange(3*1024, dtype=np.uint8),
+            np.arange(3*1024, dtype=np.uint8),
+            np.arange(3*1024, dtype=np.uint8),
+            np.arange(3*1024, dtype=np.uint8),
+            np.arange(3*1024, dtype=np.uint8),
+            np.arange(3*1024, dtype=np.uint8),
+            np.arange(3*1024, dtype=np.uint8),
+            np.arange(10*1024, dtype=np.uint8),
+            np.arange(10*1024, dtype=np.uint8),
+            ]
+        entries2 = [
+            np.arange(10*1024, dtype=np.uint8),
+            np.arange(5*1024, dtype=np.uint8),
+        ]
+        
+        filenames = self.check_entries(entries1, "FIRST")
+        assert len(self.image.files) == self.num_files_in_sample + 11
+        self.image.delete_file(filenames[2])
+        self.image.delete_file(filenames[5])
+        self.image.delete_file(filenames[0])
+        self.image.delete_file(filenames[8])
+        assert len(self.image.files) == self.num_files_in_sample + 7
+
+        filename = self.check_entries(entries2, "SECOND", save="test_delete.atr")
+        assert len(self.image.files) == self.num_files_in_sample + 9
+
+    def test_delete_all(self):
+        entries1 = [
+            np.arange(3*1024, dtype=np.uint8),
+            np.arange(3*1024, dtype=np.uint8),
+            np.arange(3*1024, dtype=np.uint8),
+            np.arange(3*1024, dtype=np.uint8),
+            np.arange(3*1024, dtype=np.uint8),
+            np.arange(3*1024, dtype=np.uint8),
+            np.arange(3*1024, dtype=np.uint8),
+            np.arange(3*1024, dtype=np.uint8),
+            np.arange(3*1024, dtype=np.uint8),
+            np.arange(10*1024, dtype=np.uint8),
+            np.arange(11*1024, dtype=np.uint8),
+            np.arange(12*1024, dtype=np.uint8),
+        ]
+        for dirent in self.image.files:
+            self.image.delete_file(dirent.filename)
+        assert len(self.image.files) == 0
+
+class TestAtariDosSDImage(BaseFilesystemModifyTest):
+    diskimage_type = AtariDosDiskImage
+    sample_data = np.fromfile("../test_data/dos_sd_test1.atr", dtype=np.uint8)
+    num_files_in_sample = 5
+
+class TestAtariDosEDImage(BaseFilesystemModifyTest):
+    diskimage_type = AtariDosDiskImage
+    sample_data = np.fromfile("../test_data/dos_ed_test1.atr", dtype=np.uint8)
+    num_files_in_sample = 5
+
+class TestAtariDosDDImage(BaseFilesystemModifyTest):
+    diskimage_type = AtariDosDiskImage
+    sample_data = np.fromfile("../test_data/dos_dd_test1.atr", dtype=np.uint8)
+    num_files_in_sample = 5
+
+class TestDos33Image(BaseFilesystemModifyTest):
+    diskimage_type = Dos33DiskImage
+    sample_data = np.fromfile("../test_data/dos33_master.dsk", dtype=np.uint8)
+    num_files_in_sample = 19
+
+
+if __name__ == "__main__":
+    t = TestAtariDosSDImage()
+    for name in dir(t):
+        print(name)
+        if name.startswith("test_"):
+            t.setup()
+            getattr(t, name)()

test/test_ataridos.py

@@ -1,19 +1,34 @@
+from __future__ import print_function
+from builtins import object
 from mock import *
 
-from atrcopy import SegmentData, AtariDosFile, InvalidBinaryFile
+from atrcopy import SegmentData, AtariDosFile, DefaultSegment, XexContainerSegment, errors
 
 
-class TestAtariDosFile(object):
+
+class TestAtariDosFile:
     def setup(self):
         pass
 
     def test_segment(self):
-        bytes = [0xff, 0xff, 0x00, 0x60, 0x01, 0x60, 1, 2]
+        bytes = np.asarray([0xff, 0xff, 0x00, 0x60, 0x01, 0x60, 1, 2], dtype=np.uint8)
         rawdata = SegmentData(bytes)
-        image = AtariDosFile(rawdata)
+        container = XexContainerSegment(rawdata, 0)
+        image = AtariDosFile(container.rawdata)
         image.parse_segments()
+        print(image.segments)
         assert len(image.segments) == 1
         assert len(image.segments[0]) == 2
+        assert np.all(image.segments[0] == bytes[6:8])
+        container.resize(16)
+        for s in image.segments:
+            s.replace_data(container)
+        new_segment = DefaultSegment(rawdata[8:16])
+        new_segment[:] = 99
+        assert np.all(image.segments[0] == bytes[6:8])
+        print(new_segment[:])
+        assert np.all(new_segment[:] == 99)
+
 
     def test_short_segment(self):
         bytes = [0xff, 0xff, 0x00, 0x60, 0xff, 0x60, 1, 2]
@@ -27,7 +42,7 @@ class TestAtariDosFile(object):
         bytes = [0xff, 0xff, 0x00, 0x60, 0x00, 0x00, 1, 2]
         rawdata = SegmentData(bytes)
         image = AtariDosFile(rawdata)
-        with pytest.raises(InvalidBinaryFile):
+        with pytest.raises(errors.InvalidBinaryFile):
             image.parse_segments()
 
 

test/test_cart.py

@@ -1,38 +1,40 @@
+from __future__ import print_function
+from __future__ import division
+from builtins import object
 from mock import *
 
-from atrcopy import AtariCartImage, SegmentData, InvalidDiskImage
+from atrcopy import AtariCartImage, SegmentData, RomImage, errors
+from atrcopy.cartridge import known_cart_types
 
 
-class TestAtariCart(object):
+class TestAtariCart:
     def setup(self):
         pass
 
     def get_cart(self, k_size, cart_type):
         data = np.zeros((k_size * 1024)+16, dtype=np.uint8)
-        data[0:4].view("|a4")[0] = 'CART'
+        data[0:4].view("|a4")[0] = b'CART'
         data[4:8].view(">u4")[0] = cart_type
         return data
 
-    def test_unbanked(self):
-        carts = [
+    @pytest.mark.parametrize("k_size,cart_type", [
             (8, 1),
             (16, 2),
             (8, 21),
             (2, 57),
             (4, 58),
             (4, 59),
-        ]
-        for k_size, cart_type in carts:
-            data = self.get_cart(k_size, cart_type)
-            rawdata = SegmentData(data)
-            image = AtariCartImage(rawdata, cart_type)
-            image.parse_segments()
-            assert len(image.segments) == 2
-            assert len(image.segments[0]) == 16
-            assert len(image.segments[1]) == k_size * 1024
+        ])
+    def test_unbanked(self, k_size, cart_type):
+        data = self.get_cart(k_size, cart_type)
+        rawdata = SegmentData(data)
+        image = AtariCartImage(rawdata, cart_type)
+        image.parse_segments()
+        assert len(image.segments) == 2
+        assert len(image.segments[0]) == 16
+        assert len(image.segments[1]) == k_size * 1024
 
-    def test_banked(self):
-        carts = [
+    @pytest.mark.parametrize("k_size,main_size,banked_size,cart_type", [
             (32, 8, 8, 12),
             (64, 8, 8, 13),
             (64, 8, 8, 67),
@@ -40,16 +42,16 @@ class TestAtariCart(object):
             (256, 8, 8, 23),
             (512, 8, 8, 24),
             (1024, 8, 8, 25),
-        ]
-        for k_size, main_size, banked_size, cart_type in carts:
-            data = self.get_cart(k_size, cart_type)
-            rawdata = SegmentData(data)
-            image = AtariCartImage(rawdata, cart_type)
-            image.parse_segments()
-            assert len(image.segments) == 1 + 1 + (k_size - main_size)/banked_size
-            assert len(image.segments[0]) == 16
-            assert len(image.segments[1]) == main_size * 1024
-            assert len(image.segments[2]) == banked_size * 1024
+        ])
+    def test_banked(self, k_size, main_size, banked_size, cart_type):
+        data = self.get_cart(k_size, cart_type)
+        rawdata = SegmentData(data)
+        image = AtariCartImage(rawdata, cart_type)
+        image.parse_segments()
+        assert len(image.segments) == 1 + 1 + (k_size - main_size) //banked_size
+        assert len(image.segments[0]) == 16
+        assert len(image.segments[1]) == main_size * 1024
+        assert len(image.segments[2]) == banked_size * 1024
 
     def test_bad(self):
         k_size = 32
@@ -57,23 +59,69 @@ class TestAtariCart(object):
         # check for error because invalid data in cart image itself
         data = self.get_cart(k_size, 1337)
         rawdata = SegmentData(data)
-        with pytest.raises(InvalidDiskImage):
+        with pytest.raises(errors.InvalidDiskImage):
             image = AtariCartImage(rawdata, 1337)
-        with pytest.raises(InvalidDiskImage):
+        with pytest.raises(errors.InvalidDiskImage):
             image = AtariCartImage(rawdata, 12)
 
         # check for error with valid cart image, but invalid cart type supplied
         # to the image parser
         data = self.get_cart(k_size, 12)
         rawdata = SegmentData(data)
-        with pytest.raises(InvalidDiskImage):
+        with pytest.raises(errors.InvalidDiskImage):
             image = AtariCartImage(rawdata, 1337)
 
 
+class TestRomCart:
+    def setup(self):
+        pass
+
+    def get_rom(self, k_size):
+        data = np.zeros((k_size * 1024), dtype=np.uint8)
+        return data
+
+    @pytest.mark.parametrize("k_size", [1, 2, 4, 8, 16, 32, 64])
+    def test_typical_rom_sizes(self, k_size):
+        data = self.get_rom(k_size)
+        rawdata = SegmentData(data)
+        rom_image = RomImage(rawdata)
+        rom_image.strict_check()
+        rom_image.parse_segments()
+        assert len(rom_image.segments) == 1
+        assert len(rom_image.segments[0]) == k_size * 1024
+
+    @pytest.mark.parametrize("k_size", [1, 2, 4, 8, 16, 32, 64])
+    def test_invalid_rom_sizes(self, k_size):
+        data = np.zeros((k_size * 1024) + 17, dtype=np.uint8)
+        rawdata = SegmentData(data)
+        with pytest.raises(errors.InvalidDiskImage):
+            rom_image = RomImage(rawdata)
+
+    @pytest.mark.parametrize("cart", known_cart_types)
+    def test_conversion_to_atari_cart(self, cart):
+        cart_type = cart[0]
+        name = cart[1]
+        k_size = cart[2]
+        if "Bounty" in name:
+            return
+        data = self.get_rom(k_size)
+        rawdata = SegmentData(data)
+        rom_image = RomImage(rawdata)
+        rom_image.strict_check()
+        rom_image.parse_segments()
+        new_cart_image = AtariCartImage(rawdata, cart_type)
+        new_cart_image.relaxed_check()
+        new_cart_image.parse_segments()
+        assert new_cart_image.header.valid
+        s = new_cart_image.create_emulator_boot_segment()
+        assert len(s) == len(rawdata) + new_cart_image.header.nominal_length
+        assert s[0:4].tobytes() == b'CART'
+        assert s[4:8].view(dtype=">u4") == cart_type
 
 
 if __name__ == "__main__":
-    print "\n".join(mime_parse_order)
+    from atrcopy.parsers import mime_parse_order
+    print("\n".join(mime_parse_order))
 
     t = TestAtariCart()
     t.setup()

test/test_container.py

@@ -0,0 +1,41 @@
+from __future__ import print_function
+from builtins import object
+import numpy as np
+
+from mock import *
+
+from atrcopy import SegmentData, iter_parsers
+from atrcopy import errors
+
+
+class BaseContainerTest:
+    base_path = None
+    expected_mime = ""
+
+    @pytest.mark.parametrize("ext", ['.gz', '.bz2', '.xz', '.dcm'])
+    def test_container(self, ext):
+        pathname = self.base_path + ext
+        try:
+            sample_data = np.fromfile(pathname, dtype=np.uint8)
+        except OSError:
+            pass
+        else:
+            rawdata = SegmentData(sample_data.copy())
+            mime, parser = iter_parsers(rawdata)
+            assert mime == self.expected_mime
+            assert len(parser.image.files) == self.num_files_in_sample
+
+class TestContainerAtariDosSDImage(BaseContainerTest):
+    base_path = "../test_data/container_dos_sd_test1.atr"
+    expected_mime = "application/vnd.atari8bit.atr"
+    num_files_in_sample = 5
+
+class TestContainerAtariDosEDImage(BaseContainerTest):
+    base_path = "../test_data/container_dos_ed_test1.atr"
+    expected_mime = "application/vnd.atari8bit.atr"
+    num_files_in_sample = 5
+
+class TestContainerAtariDosDDImage(BaseContainerTest):
+    base_path = "../test_data/container_dos_dd_test1.atr"
+    expected_mime = "application/vnd.atari8bit.atr"
+    num_files_in_sample = 5

test/test_create.py

@@ -0,0 +1,84 @@
+from __future__ import print_function
+from builtins import object
+import numpy as np
+
+from mock import *
+
+from atrcopy import SegmentData, AtariDosDiskImage, Dos33DiskImage, DefaultSegment
+from atrcopy import errors
+
+
+def get_image(file_name, diskimage_type):
+    data = np.fromfile(file_name, dtype=np.uint8)
+    rawdata = SegmentData(data)
+    image = diskimage_type(rawdata)
+    return image
+
+
+class BaseCreateTest:
+    diskimage_type = None
+
+    def get_exe_segments(self):
+        data1 = np.arange(4096, dtype=np.uint8)
+        data1[1::2] = np.repeat(np.arange(16, dtype=np.uint8), 128)
+        data2 = np.arange(4096, dtype=np.uint8)
+        data2[0::4] = np.repeat(np.arange(8, dtype=np.uint8), 128)
+        raw = [
+            (data1, 0x4000),
+            (data2, 0x8000),
+        ]
+
+        segments = []
+        for data, origin in raw:
+            rawdata = SegmentData(data)
+            s = DefaultSegment(rawdata, origin)
+            segments.append(s)
+        return segments
+
+    def check_exe(self, sample_file, diskimage_type, run_addr, expected):
+        image = get_image(sample_file, diskimage_type)
+        segments = self.get_exe_segments()
+        try:
+            _ = issubclass(errors.AtrError, expected)
+            with pytest.raises(errors.InvalidBinaryFile) as e:
+                file_data, filetype = image.create_executable_file_image(sample_file, segments, run_addr)
+        except TypeError:
+            file_data, filetype = image.create_executable_file_image(sample_file, segments, run_addr)
+            print(image)
+            print(file_data, filetype)
+            assert len(file_data) == expected
+
+@pytest.mark.parametrize("sample_file", ["../test_data/dos_sd_test1.atr"])
+class TestAtariDosSDImage(BaseCreateTest):
+    diskimage_type = AtariDosDiskImage
+
+    @pytest.mark.parametrize("run_addr,expected", [
+        (0x2000, errors.InvalidBinaryFile),
+        (None, (2 + 6 + (4 + 0x1000) + (4 + 0x1000))),
+        (0x4000, (2 + 6 + (4 + 0x1000) + (4 + 0x1000))),
+        (0x8000, (2 + 6 + (4 + 0x1000) + (4 + 0x1000))),
+        (0xffff, errors.InvalidBinaryFile),
+        ])
+    def test_exe(self, run_addr, expected, sample_file):
+        self.check_exe(sample_file, self.diskimage_type, run_addr, expected)
+
+
+@pytest.mark.parametrize("sample_file", ["../test_data/dos33_master.dsk"])
+class TestDos33Image(BaseCreateTest):
+    diskimage_type = Dos33DiskImage
+
+    @pytest.mark.parametrize("run_addr,expected", [
+        (0x2000, errors.InvalidBinaryFile),
+        (None, (4 + (0x9000 - 0x4000))),
+        (0x4000, (4 + (0x9000 - 0x4000))),
+        (0x8000, (4 + 3 + (0x9000 - 0x4000))),
+        (0xffff, errors.InvalidBinaryFile),
+        ])
+    def test_exe(self, run_addr, expected, sample_file):
+        self.check_exe(sample_file, self.diskimage_type, run_addr, expected)
+
+
+if __name__ == "__main__":
+    t = TestAtariDosSDImage()
+    t.setup()
+    t.test_exe()

test/test_jsonpickle.py

@@ -1,45 +1,50 @@
+from __future__ import print_function
+from builtins import zip
+from builtins import range
+from builtins import object
 import os
 
-import jsonpickle
+import pytest
+jsonpickle = pytest.importorskip("jsonpickle")
 
 import numpy as np
 
 from atrcopy import DefaultSegment, SegmentData
 
 
-class TestJsonPickle(object):
+class TestJsonPickle:
     def setup(self):
         data = np.arange(2048, dtype=np.uint8)
         self.segment = DefaultSegment(SegmentData(data))
 
     def test_simple(self):
-        print self.segment.byte_bounds_offset(), len(self.segment)
+        print(self.segment.byte_bounds_offset(), len(self.segment))
         r2 = self.segment.rawdata[100:400]
         s2 = DefaultSegment(r2)
-        print s2.byte_bounds_offset(), len(s2), s2.__getstate__()
+        print(s2.byte_bounds_offset(), len(s2), s2.__getstate__())
         r3 = s2.rawdata[100:200]
         s3 = DefaultSegment(r3)
-        print s3.byte_bounds_offset(), len(s3), s3.__getstate__()
-        order = list(reversed(range(700, 800)))
+        print(s3.byte_bounds_offset(), len(s3), s3.__getstate__())
+        order = list(reversed(list(range(700, 800))))
         r4 = self.segment.rawdata.get_indexed(order)
         s4 = DefaultSegment(r4)
-        print s4.byte_bounds_offset(), len(s4), s4.__getstate__()
+        print(s4.byte_bounds_offset(), len(s4), s4.__getstate__())
         
         slist = [s2, s3, s4]
         for s in slist:
-            print s
+            print(s)
         j = jsonpickle.dumps(slist)
-        print j
+        print(j)
         
         slist2 = jsonpickle.loads(j)
-        print slist2
+        print(slist2)
         for s in slist2:
             s.reconstruct_raw(self.segment.rawdata)
-            print s
+            print(s)
         
         for orig, rebuilt in zip(slist, slist2):
-            print "orig", orig.data[:]
-            print "rebuilt", rebuilt.data[:]
+            print("orig", orig.data[:])
+            print("rebuilt", rebuilt.data[:])
             assert np.array_equal(orig[:], rebuilt[:])
 
 if __name__ == "__main__":

test/test_kboot.py

@@ -1,11 +1,16 @@
+from __future__ import print_function
+from __future__ import division
+from builtins import range
+from builtins import object
 import os
 
 import numpy as np
 
+from mock import *
 from atrcopy import SegmentData, KBootImage, add_xexboot_header, add_atr_header
 
 
-class TestKbootHeader(object):
+class TestKbootHeader:
     def setup(self):
         pass
 
@@ -21,9 +26,9 @@ class TestKbootHeader(object):
         rawdata = SegmentData(bytes)
         newatr = KBootImage(rawdata)
         image = newatr.bytes
-        print image[0:16]
-        paragraphs = image_size / 16
-        print newatr.header, paragraphs
+        print(image[0:16])
+        paragraphs = image_size // 16
+        print(newatr.header, paragraphs)
         assert int(image[2:4].view(dtype='<u2')) == paragraphs
         assert int(image[16 + 9:16 + 9 + 2].view('<u2')) == xex_size
         return image
@@ -34,9 +39,9 @@ class TestKbootHeader(object):
             self.check_size(data)
     
     def test_real(self):
-        data = np.fromfile("air_defense_v18.xex", dtype=np.uint8)
+        data = np.fromfile("../test_data/air_defense_v18.xex", dtype=np.uint8)
         image = self.check_size(data)
-        with open("air_defense_v18.atr", "wb") as fh:
+        with open("../test_data/air_defense_v18.atr", "wb") as fh:
             txt = image.tostring()
             fh.write(txt)
 

test/test_segment.py

@@ -1,22 +1,28 @@
+from __future__ import print_function
+from builtins import zip
+from builtins import range
+from builtins import object
 import os
 
 import numpy as np
 import pytest
 
-from atrcopy import DefaultSegment, SegmentData, get_xex, interleave_segments
+from atrcopy import DefaultSegment, SegmentData, get_xex, interleave_segments, user_bit_mask, diff_bit_mask
+from atrcopy import errors
+from functools import reduce
 
 
 def get_indexed(segment, num, scale):
     indexes = np.arange(num) * scale
     raw = segment.rawdata.get_indexed(indexes)
-    s = DefaultSegment(raw, segment.start_addr + indexes[0])
+    s = DefaultSegment(raw, segment.origin + indexes[0])
     return s, indexes
 
-class TestSegment1(object):
+class TestSegment1:
     def setup(self):
         self.segments = []
         for i in range(8):
-            data = np.ones([1024], dtype=np.uint8) * i
+            data = np.arange(1024, dtype=np.uint8) * i
             r = SegmentData(data)
             self.segments.append(DefaultSegment(r, i * 1024))
 
@@ -27,16 +33,42 @@ class TestSegment1(object):
         
         for indexes, stuff in items:
             s = [self.segments[i] for i in indexes]
-            bytes = get_xex(s, 0xbeef)
-            assert tuple(bytes[0:2]) == (0xff, 0xff)
+            s[1].style[0:500] = diff_bit_mask
+            s[1].set_comment_at(0, "comment 0")
+            s[1].set_comment_at(10, "comment 10")
+            s[1].set_comment_at(100, "comment 100")
+            print(list(s[1].iter_comments_in_segment()))
+            with pytest.raises(errors.InvalidBinaryFile):
+                seg, subseg = get_xex(s, 0xbeef)
+            seg, subseg = get_xex(s)
+            assert tuple(seg.data[0:2]) == (0xff, 0xff)
             # 2 bytes for the ffff
-            # 6 bytes for the last segment run address
             # 4 bytes per segment for start, end address
-            size = reduce(lambda a, b:a + 4 + len(b), s, 0)
-            assert len(bytes) == 2 + 6 + size
+            # An extra segment has been inserted for the run address!
+            size = reduce(lambda a, b:a + len(b), subseg, 0)
+            assert len(seg) == 2 + size
+            print(id(s[1]), list(s[1].iter_comments_in_segment()))
+            print(id(subseg[2]), list(subseg[2].iter_comments_in_segment()))
+            for i, c in s[1].iter_comments_in_segment():
+                assert c == subseg[2].get_comment(i + 4)
+            assert np.all(s[1].style[:] == subseg[2].style[4:])
+
+    def test_copy(self):
+        for s in self.segments:
+            d = s.rawdata
+            print("orig:", d.data.shape, d.is_indexed, d.data, id(d.data))
+            c = d.copy()
+            print("copy", c.data.shape, c.is_indexed, c.data, id(c.data))
+            assert c.data.shape == s.data.shape
+            assert id(c) != id(s)
+            assert np.all((c.data[:] - s.data[:]) == 0)
+            c.data[0:100] = 1
+            print(d.data)
+            print(c.data)
+            assert not np.all((c.data[:] - s.data[:]) == 0)
 
 
-class TestIndexed(object):
+class TestIndexed:
     def setup(self):
         data = np.arange(4096, dtype=np.uint8)
         data[1::2] = np.repeat(np.arange(16, dtype=np.uint8), 128)
@@ -65,7 +97,7 @@ class TestIndexed(object):
         assert not sub.rawdata.is_indexed
         for i in range(len(sub)):
             ri = sub.get_raw_index(i)
-            assert ri == sub.start_addr + i
+            assert ri == sub.origin + i
             assert sub[i] == base[ri]
         start, end = sub.byte_bounds_offset()
         assert start == 512
@@ -77,14 +109,14 @@ class TestIndexed(object):
         
         # try with elements up to 256 * 3
         s, indexes = get_indexed(sub, 256, 3)
-        print sub.data
-        print indexes
-        print s.data[:]
+        print(sub.data)
+        print(indexes)
+        print(s.data[:])
         assert s.rawdata.is_indexed
         for i in range(len(indexes)):
             ri = s.get_raw_index(i)
-            print ri, "base[ri]=%d" % base[ri], i, indexes[i], "s[i]=%d" % s[i]
-            assert ri == sub.start_addr + indexes[i]
+            print(ri, "base[ri]=%d" % base[ri], i, indexes[i], "s[i]=%d" % s[i])
+            assert ri == sub.origin + indexes[i]
             assert s[i] == base[ri]
         start, end = s.byte_bounds_offset()
         assert start == 0
@@ -94,7 +126,7 @@ class TestIndexed(object):
         s2, indexes2 = get_indexed(s, 64, 3)
         assert s2.rawdata.is_indexed
         for i in range(len(indexes2)):
-            assert s2.get_raw_index(i) == sub.start_addr + indexes2[i] * 3
+            assert s2.get_raw_index(i) == sub.origin + indexes2[i] * 3
         start, end = s.byte_bounds_offset()
         assert start == 0
         assert end == len(base)
@@ -120,9 +152,9 @@ class TestIndexed(object):
         a[1::4] = s1[1::2]
         a[2::4] = s2[0::2]
         a[3::4] = s2[1::2]
-        print list(s[:])
-        print list(a[:])
-        print s.rawdata.order
+        print(list(s[:]))
+        print(list(a[:]))
+        print(s.rawdata.order)
         assert np.array_equal(s[:], a)
         
         s = interleave_segments([s1, s2], 4)
@@ -136,28 +168,319 @@ class TestIndexed(object):
         a[6::8] = s2[2::4]
         a[7::8] = s2[3::4]
         assert np.array_equal(s[:], a)
-        
-        with pytest.raises(ValueError) as e:
-            s = interleave_segments([s1, s2], 3)
 
-        r1 = base.rawdata[512:1025]  # 513 byte segment
+    def test_interleave_not_multiple(self):
+        base = self.segment
+        r1 = base.rawdata[512:1024]  # 512 byte segment
         s1 = DefaultSegment(r1, 512)
-        r2 = base.rawdata[1024:1537]  # 513 byte segment
+        r2 = base.rawdata[1024:1536]  # 512 byte segment
         s2 = DefaultSegment(r2, 1024)
+        
+        indexes1 = r1.get_indexes_from_base()
+        verify1 = np.arange(512, 1024, dtype=np.uint32)
+        assert np.array_equal(indexes1, verify1)
+        
+        indexes2 = r2.get_indexes_from_base()
+        verify2 = np.arange(1024, 1536, dtype=np.uint32)
+        assert np.array_equal(indexes2, verify2)
+        
         s = interleave_segments([s1, s2], 3)
-        a = np.empty(len(s1) + len(s2), dtype=np.uint8)
-        a[0::6] = s1[0::3]
-        a[1::6] = s1[1::3]
-        a[2::6] = s1[2::3]
-        a[3::6] = s2[0::3]
-        a[4::6] = s2[1::3]
-        a[5::6] = s2[2::3]
+
+        # when interleave size isn't a multiple of the length, the final array
+        # will reduce the size of the input array to force it to be a multiple.
+        size = (len(s1) // 3) * 3
+        assert len(s) == size * 2
+        a = np.empty(len(s), dtype=np.uint8)
+        a[0::6] = s1[0:size:3]
+        a[1::6] = s1[1:size:3]
+        a[2::6] = s1[2:size:3]
+        a[3::6] = s2[0:size:3]
+        a[4::6] = s2[1:size:3]
+        a[5::6] = s2[2:size:3]
         assert np.array_equal(s[:], a)
 
+    def test_interleave_different_sizes(self):
+        base = self.segment
+        r1 = base.rawdata[512:768]  # 256 byte segment
+        s1 = DefaultSegment(r1, 512)
+        r2 = base.rawdata[1024:1536]  # 512 byte segment
+        s2 = DefaultSegment(r2, 1024)
+        
+        indexes1 = r1.get_indexes_from_base()
+        verify1 = np.arange(512, 768, dtype=np.uint32)
+        assert np.array_equal(indexes1, verify1)
+        
+        indexes2 = r2.get_indexes_from_base()
+        verify2 = np.arange(1024, 1536, dtype=np.uint32)
+        assert np.array_equal(indexes2, verify2)
+        
+        s = interleave_segments([s1, s2], 3)
+
+        # when interleave size isn't a multiple of the length, the final array
+        # will reduce the size of the input array to force it to be a multiple.
+        size = (min(len(s1), len(s2)) // 3) * 3
+        assert size == (256 // 3) * 3
+        assert len(s) == size * 2
+        a = np.empty(len(s), dtype=np.uint8)
+        a[0::6] = s1[0:size:3]
+        a[1::6] = s1[1:size:3]
+        a[2::6] = s1[2:size:3]
+        a[3::6] = s2[0:size:3]
+        a[4::6] = s2[1:size:3]
+        a[5::6] = s2[2:size:3]
+        assert np.array_equal(s[:], a)
+
+    def test_copy(self):
+        s, indexes = get_indexed(self.segment, 1024, 3)
+        c = s.rawdata.copy()
+        print(c.data.shape, c.is_indexed)
+        print(id(c.data.np_data), id(s.data.np_data))
+        assert c.data.shape == s.data.shape
+        assert id(c) != id(s)
+        assert np.all((c.data[:] - s.data[:]) == 0)
+        c.data[0:100] = 1
+        assert not np.all((c.data[:] - s.data[:]) == 0)
+
+
+class TestComments:
+    def setup(self):
+        data = np.ones([4000], dtype=np.uint8)
+        r = SegmentData(data)
+        self.segment = DefaultSegment(r, 0)
+        self.sub_segment = DefaultSegment(r[2:202], 2)
+
+    def test_locations(self):
+        s = self.segment
+        s.set_comment([[4,5]], "test1")
+        s.set_comment([[40,50]], "test2")
+        s.set_style_ranges([[2,100]], comment=True)
+        s.set_style_ranges([[200, 299]], data=True)
+        for i in range(1,4):
+            for j in range(1, 4):
+                # create some with overlapping regions, some without
+                r = [500*j, 500*j + 200*i + 200]
+                s.set_style_ranges([r], user=i)
+                s.set_user_data([r], i, i*10 + j)
+        r = [100, 200]
+        s.set_style_ranges([r], user=4)
+        s.set_user_data([r], 4, 99)
+        r = [3100, 3200]
+        s.set_style_ranges([r], user=4)
+        s.set_user_data([r], 4, 99)
+
+        s2 = self.sub_segment
+        print(len(s2))
+        copy = s2.get_comment_locations()
+        print(copy)
+        # comments at 4 and 40 in the original means 2 and 38 in the copy
+        orig = s.get_comment_locations()
+        assert copy[2] == orig[4]
+        assert copy[28] == orig[38]
+
+    def test_split_data_at_comment(self):
+        s = self.segment
+        s.set_style_ranges([[0,1000]], data=True)
+        for i in range(0, len(s), 25):
+            s.set_comment([[i,i+1]], "comment at %d" % i)
+
+        s2 = self.sub_segment
+        print(len(s2))
+        copy = s2.get_comment_locations()
+        print(copy)
+        # comments at 4 and 40 in the original means 2 and 38 in the copy
+        orig = s.get_comment_locations()
+        print(orig[0:200])
+        assert copy[2] == orig[4]
+        assert copy[28] == orig[38]
+
+        r = s2.get_entire_style_ranges([1], user=True)
+        print(r)
+        assert r == [((0, 23), 1), ((23, 48), 1), ((48, 73), 1), ((73, 98), 1), ((98, 123), 1), ((123, 148), 1), ((148, 173), 1), ((173, 198), 1), ((198, 200), 1)]
+
+    def test_split_data_at_comment2(self):
+        s = self.segment
+        start = 0
+        i = 0
+        for end in range(40, 1000, 40):
+            s.set_style_ranges([[start, end]], user=i)
+            start = end
+            i = (i + 1) % 8
+        for i in range(0, len(s), 25):
+            s.set_comment([[i,i+1]], "comment at %d" % i)
+
+        s2 = self.sub_segment
+        print(len(s2))
+        copy = s2.get_comment_locations()
+        print(copy)
+        # comments at 4 and 40 in the original means 2 and 38 in the copy
+        orig = s.get_comment_locations()
+        print(orig[0:200])
+        assert copy[2] == orig[4]
+        assert copy[28] == orig[38]
+
+        r = s2.get_entire_style_ranges([1], user=user_bit_mask)
+        print(r)
+        assert r == [((0, 38), 0), ((38, 48), 1), ((48, 73), 1), ((73, 78), 1), ((78, 118), 2), ((118, 158), 3), ((158, 198), 4), ((198, 200), 5)]
+
+    def test_restore_comments(self):
+        s = self.segment
+        s.set_style_ranges([[0,1000]], data=True)
+        for i in range(0, len(s), 5):
+            s.set_comment([[i,i+1]], "comment at %d" % i)
+
+        s1 = self.segment
+        print(len(s1))
+        indexes = [7,12]
+        r = s1.get_comment_restore_data([indexes])
+        print(r)
+        # force clear comments
+        s1.rawdata.extra.comments = {}
+        s1.style[indexes[0]:indexes[1]] = 0
+        r0 = s1.get_comment_restore_data([indexes])
+        print(r0)
+        for start, end, style, items in r0:
+            print(style)
+            assert np.all(style == 0)
+            for rawindex, comment in list(items.values()):
+                assert not comment
+        s1.restore_comments(r)
+        r1 = s1.get_comment_restore_data([indexes])
+        print(r1)
+        for item1, item2 in zip(r, r1):
+            print(item1)
+            print(item2)
+            for a1, a2 in zip(item1, item2):
+                print(a1, a2)
+                if hasattr(a1, "shape"):
+                    assert np.all(a1 - a2 == 0)
+                else:
+                    assert a1 == a2
+
+        s2 = self.sub_segment
+        print(len(s2))
+        indexes = [5,10]
+        r = s2.get_comment_restore_data([indexes])
+        print(r)
+        # force clear comments
+        s2.rawdata.extra.comments = {}
+        s2.style[indexes[0]:indexes[1]] = 0
+        r0 = s2.get_comment_restore_data([indexes])
+        print(r0)
+        for start, end, style, items in r0:
+            print(style)
+            assert np.all(style == 0)
+            for rawindex, comment in list(items.values()):
+                assert not comment
+        s2.restore_comments(r)
+        r2 = s2.get_comment_restore_data([indexes])
+        print(r2)
+        for item1, item2 in zip(r, r2):
+            print(item1)
+            print(item2)
+            for a1, a2 in zip(item1, item2):
+                print(a1, a2)
+                if hasattr(a1, "shape"):
+                    assert np.all(a1 - a2 == 0)
+                else:
+                    assert a1 == a2
+
+        for item1, item2 in zip(r1, r2):
+            print(item1)
+            print(item2)
+            # indexes won't be the same, but rawindexes and comments will
+            assert np.all(item1[2] - item2[2] == 0)
+            assert set(item1[3].values()) == set(item2[3].values())
+
+
+class TestResize:
+    def setup(self):
+        data = np.arange(4096, dtype=np.uint8)
+        data[1::2] = np.repeat(np.arange(16, dtype=np.uint8), 128)
+        r = SegmentData(data)
+        self.container = DefaultSegment(r, 0)
+        self.container.can_resize = True
+
+    def test_subset(self):
+        # check to see data a view of some rawdata will be the same when the
+        # rawdata is resized.
+        c = self.container
+        assert not c.rawdata.is_indexed
+        offset = 1000
+        s = DefaultSegment(c.rawdata[offset:offset + offset], 0)
+        assert not s.rawdata.is_indexed
+
+        # Check that the small view has the same data as its parent
+        for i in range(offset):
+            assert s[i] == c[i + offset]
+
+        # keep a copy of the old raw data of the subset
+        oldraw = s.rawdata.copy()
+        oldid = id(s.rawdata)
+
+        requested = 8192
+        oldsize, newsize = c.resize(requested)
+        assert newsize == requested
+        s.replace_data(c)  # s should point to the same offset in the resized data
+        assert id(s.rawdata) == oldid  # segment rawdata object should be same
+        assert id(oldraw.order) == id(s.rawdata.order)  # order the same
+        for i in range(offset):  # check values compared to parent
+            assert s[i] == c[i + offset]
+
+        # check for changes in parent/view reflected so we see that it's
+        # pointing to the same array in memory
+        newbase = c.rawdata
+        newsub = s.rawdata
+        print(c.rawdata.data[offset:offset+offset])
+        print(s.rawdata.data[:])
+        s.rawdata.data[:] = 111
+        print(c.rawdata.data[offset:offset+offset])
+        print(s.rawdata.data[:])
+        for i in range(offset):
+            assert s[i] == c[i + offset]
+
+    def test_indexed(self):
+        c = self.container
+        assert not c.rawdata.is_indexed
+        s, indexes = get_indexed(self.container, 1024, 3)
+        assert s.rawdata.is_indexed
+        for i in range(len(indexes)):
+            assert s.get_raw_index(i) == indexes[i]
+        requested = 8192
+        oldraw = s.rawdata.copy()
+        oldid = id(s.rawdata)
+        oldsize, newsize = c.resize(requested)
+        assert newsize == requested
+        s.replace_data(c)
+        assert id(s.rawdata) == oldid
+        assert id(oldraw.order) == id(s.rawdata.order)
+        for i in range(len(indexes)):
+            assert s.get_raw_index(i) == indexes[i]
+        newbase = c.rawdata
+        newsub = s.rawdata
+        print(c.rawdata.data)
+        print(s.rawdata.data[:])
+        s.rawdata.data[:] = 111
+        print(c.rawdata.data)
+        print(s.rawdata.data[:])
+        for i in range(len(indexes)):
+            assert c.rawdata.data[indexes[i]] == s.rawdata.data[i]
+
+
 
 if __name__ == "__main__":
-    t = TestIndexed()
+    # t = TestIndexed()
+    # t.setup()
+    # t.test_indexed()
+    # t.test_indexed_sub()
+    # t.test_interleave()
+    # t = TestSegment1()
+    # t.setup()
+    # t.test_xex()
+    # t.test_copy()
+    # t = TestComments()
+    # t.setup()
+    # t.test_split_data_at_comment()
+    # t.test_restore_comments()
+    t = TestResize()
     t.setup()
-    t.test_indexed()
-    t.test_indexed_sub()
-    t.test_interleave()
+    t.test_subset()

test/test_serialize.py

@@ -0,0 +1,66 @@
+from __future__ import print_function
+from builtins import range
+from builtins import object
+from builtins import str
+import os
+
+import numpy as np
+import pytest
+
+from atrcopy import DefaultSegment, SegmentData, get_xex, interleave_segments
+
+
+class TestSegment:
+    def setup(self):
+        data = np.ones([4000], dtype=np.uint8)
+        r = SegmentData(data)
+        self.segment = DefaultSegment(r, 0)
+
+    def test_getstate(self):
+        state = self.segment.__getstate__()
+        for k, v in state.items():
+            print("k=%s v=%s type=%s" % (k, v, type(v)))
+        byte_type = type(str(u'  ').encode('utf-8'))  # py2 and py3
+        try:
+            u = unicode(" ")
+        except:
+            u = str(" ")
+        assert type(state['uuid']) == type(u)
+
+    def test_extra(self):
+        s = self.segment
+        s.set_comment([[4,5]], "test1")
+        s.set_comment([[40,50]], "test2")
+        s.set_style_ranges([[2,100]], comment=True)
+        s.set_style_ranges([[200, 299]], data=True)
+        for i in range(1,4):
+            for j in range(1, 4):
+                # create some with overlapping regions, some without
+                r = [500*j, 500*j + 200*i + 200]
+                s.set_style_ranges([r], user=i)
+                s.set_user_data([r], i, i*10 + j)
+        r = [100, 200]
+        s.set_style_ranges([r], user=4)
+        s.set_user_data([r], 4, 99)
+        r = [3100, 3200]
+        s.set_style_ranges([r], user=4)
+        s.set_user_data([r], 4, 99)
+
+        out = dict()
+        s.serialize_session(out)
+        print("saved", out)
+
+        data = np.ones([4000], dtype=np.uint8)
+        r = SegmentData(data)
+        s2 = DefaultSegment(r, 0)
+        s2.restore_session(out)
+        out2 = dict()
+        s2.serialize_session(out2)
+        print("loaded", out2)
+        assert out == out2
+
+
+if __name__ == "__main__":
+    t = TestSegment()
+    t.setup()
+    t.test_getstate()

test/test_utils.py

@@ -0,0 +1,24 @@
+from builtins import object
+from mock import *
+from atrcopy import utils
+
+
+class TestTextToInt:
+    def setup(self):
+        pass
+
+    @pytest.mark.parametrize("text,expected,default_base", [
+        ("12", 0x12, "hex"),
+        ("$1234", 0x1234, "hex"),
+        ("0xffff", 0xffff, "hex"),
+        ("#12", 12, "hex"),
+        ("%11001010", 202, "hex"),
+
+        ("12", 12, "dec"),
+        ("$1234", 0x1234, "dec"),
+        ("0xffff", 0xffff, "dec"),
+        ("#12", 12, "dec"),
+        ("%11001010", 202, "dec"),
+        ])
+    def test_text_to_int(self, text, expected, default_base):
+        assert expected == utils.text_to_int(text, default_base)

test_data/test_header.s

@@ -0,0 +1,29 @@
+SDLSTL	= $0230 
+CONSOL	= $D01F 
+;
+	*= $0600 
+;
+INIT
+	LDA #<TITLEDL
+	STA SDLSTL
+	LDA #>TITLEDL
+	STA SDLSTL+1 
+	LDA #8 
+	STA CONSOL
+TTL0	LDA CONSOL
+	CMP #6 
+	BNE TTL0
+	RTS 
+;
+TITLESC
+	.SBYTE +$80 ,"   PRESS START TO   "
+	.SBYTE +$C0 ,"  CONTINUE LOADING  "
+;
+TITLEDL	.BYTE $70 ,$70 ,$70
+	.BYTE $70 ,$70 ,$70 ,$70 ,$70 ,$70 ,$70 ,$70 ,$70 ,$70 ,$47 
+	.WORD TITLESC
+	.BYTE 6 ,$41 
+	.WORD TITLEDL
+
+	*= $02E2 
+	.WORD INIT