Discussion of accurate FDS archiving

This is an archive of a topic from NESdev BBS, taken in mid-October 2019 before a server upgrade.
View original topic
Discussion of accurate FDS archiving
by on (#25466)
As has been discussed previously, the current Famicom Disk System file format (*.FDS) is flawed in several ways. It does not archive the GAP periods, block start marks, or CRCs. While it is a marginally acceptable format for interoperability with emulators, it falls far short as a format for archival and preservation. This is an especially important issue since Disk Cards are not only notoriously prone to failure, but in many cases it is impossible to get a clean dump if the game has ever been played (since additional files are written to the disk). That means that clean dumps are dependent on the ever-dwindling supply of sealed Disk Cards. If the community does not act soon, these may be lost forever. And if people are to go to the trouble of buying expensive sealed games and dumping them, then it's important to do it right. The dumps should be 100%, bit-for-bit congruent with what is actually on the disk.

I am not going to define a new file format specification at this point. Instead, I am going to bring up several thoughts I have had, while carefully reading existing documentation on the FDS. Brad Taylor's document was especially helpful.

*To get accurate dumps, the RAM-adapter would probably have to be bypassed. The RAM-adapter does its own processing on the data, and we want to get exactly what is on the disk, no more and no less.
*This means hooking the FDS cable up to the parallel port. /WRITE would be tied high, because only read operations would be desired. The program on the PC side would have to pulse the /SCAN_MEDIA line, then wait for the falling edge of /READY. The READ DATA line would be hooked up to the parallel port's ACK line, so that an IRQ would be generated whenever the drive put out a pulse. The program would measure the timing between pulses and use that to reconstruct the FDS bitstream.
*As per Brad's FDSLoader document, IRQ would be the only reliable way to read the microsecond-length pulses on a LPT port. And this means the dumping program would have to run in DOS.
*According to Brad's document, "The "-ready" signal is based on a mechanical switch inside the drive which is activated when the head is brought back to the outer most edge of the disk (the beginning). Because the switch will usually be triggered prematurely, the first 13000 bits (approx.) of data the drive will send out immediately after this switch is activated will be invalid." Does this mean that it is impossible to accurately measure the length of the initial GAP period at the start of a Disk Card?
*Reading the documentation, I am not quite clear if the start mark is a single high bit, or a byte $80. Or, since data is sent in low-endian bit format, does the adapter see the lone high bit and then derive the $80 byte from that bit plus the previous seven zeroes from the GAP period? I suppose if I actually built the thing I could look at some dumps and see what the format actually was, but right now this is all hypothetical.

I'm still trying to puzzle out how to reverse the pulse timing back into a coherent data stream. I had a few ideas but they don't seem to account for all the cases.

by on (#25475)
Many games are never written to, so accurate dumps are pretty easy to come by. Those with save files can be a pain. Sometimes you have to find where the save file is stored and look at its format to extract it and put in a blank one...not always easy or possible without something to compare it to. For example, when I dumped the Disk Fax Gold Prize Card Golf games, one had a messed up save file, but by looking at the other golf games, CCovell figured out how to fix the dump.

If you have undumped FDS games, let me know and I'll get them dumped.

-Rob

by on (#25477)
Didn't the same thing happen with regards to ripping game CDROM images? You have to do a bit-for-bit cloning of a CD, in order to get around certain forms of copy-protection, which purposely circumvent the CDROM standard. It is no surprise that a 1980s technology like FDS games would break the disk standard either by accident, for optimizations, or for copy protection.

by on (#25512)
Quickdisk was developed by Mitsumi in the 70s, so FDS is actually 1970s technology. ;)

What disk standard do you mean was broken?

-Rob

by on (#25521)
Jagasian wrote:
Didn't the same thing happen with regards to ripping game CDROM images? You have to do a bit-for-bit cloning of a CD, in order to get around certain forms of copy-protection, which purposely circumvent the CDROM standard. It is no surprise that a 1980s technology like FDS games would break the disk standard either by accident, for optimizations, or for copy protection.


It's not limited to FDS. Commodore 64 disk games (utilizing the 1541 disk drive) did this all the time. You had all kinds of crazy violations of the standard: deliberate sector errors that had to be there, custom formats that didn't follow the Commodore DOS standards, tracks with no sync marks, violations of the GCR rules... just about anything that could be done to screw up a disk was done by some publisher or another.

This is why the *.G64 standard had to be invented, to replace the older (and still more common) *.D64 standard. The latter format assumed standards compliance, so it could not replicate the weird tricks that were used by copy-protected software. You could only play games if they were hacked (which they often were, back in the day.)

rbudrick wrote:
What disk standard do you mean was broken?


The *.FDS standard is flawed because it does not preserve the length of GAP periods, nor does it contain block start marks, or block end marks (the latter of which include CRCs). *.FDS is sufficient to play back the games in an emulator. However, it is not sufficient to create a bit-for-bit identical copy of the data on the original disk. From a standpoint of historical preservation, the latter is important.

by on (#25526)
Quote:
The *.FDS standard is flawed because it does not preserve the length of GAP periods, nor does it contain block start marks, or block end marks (the latter of which include CRCs). *.FDS is sufficient to play back the games in an emulator. However, it is not sufficient to create a bit-for-bit identical copy of the data on the original disk. From a standpoint of historical preservation, the latter is important.


Well, I know there's no eof marker for the whole disk or even on individual disk files, but there are file start markers (are these the block markers you are referring to?). I know dumpers like fdsloadr just look for a long string of 0s at the end of a FDS disk to determine it's end, since there's no real marker. 99.9% of disks are only 65, 500 bytes per side anyway, so it's easy to know where most ends of disks should be. Kyuusaku has mentioned that it's possible to store 67500 bytes on one, but I think he mentioned he hasn't actually seen it done...I may be remembering his phrasing incorrectly.

For the few copy-protected games there were some cheats they used for hidden files and such (false file count markers and such).

However, I think I fail to understand how any bits are lost in the duming process for any non-save file disks. The CRCs I thought were more low level...I didn't realize they were even on the disks. When you mention GAP periods, could you be more specific? Forgive my ignorance of the term, but FDS fascinates me, and I like to learn whatever I can on it's format.

It sounds like no one has ever had a dumping method to preserve this extra data. What do you think would need to be done to change this or to create a device to do it?

I actually own a Sharp MZ-1500 computer with Quickdisk drive and a MSX II Turbo R with optional Quickdisk drive. I would love for a guru of either system to tell me how to use them to benefit FDS...I know the MZ can dump FDS somehow, but I haven't a clue how. I've seen a pic of an FDS drive hooked to an MZ 1500, but no plans or notes on it.

-Rob

by on (#25530)
rbudrick wrote:
Kyuusaku has mentioned that it's possible to store 67500 bytes on one, but I think he mentioned he hasn't actually seen it done...I may be remembering his phrasing incorrectly.

I just went most of my disk images, and the biggest image I have is 66,461 bytes. Nintendo claims the disks hold 65K or 66,650 bytes but I don't think I have any that big. Someone with a lot of time could try filling up a few disks and reporting how large they are :D

by on (#25531)
kyuusaku wrote:
I just went most of my disk images, and the biggest image I have is 66,461 bytes.


Does that figure include the GAP periods and block start/end marks? If so, what methodology did you use to obtain these dumps?

by on (#25749)
Yes it includes everything, I obtained these dumps with a Multi Game Doctor. The disk image itself contains the Nintendo header, 4 file headers and the files: a 512 byte file, a 8 byte file and two 32768 byte files.

While Multi Game Doctors dump with the gaps and all, they aren't suitable for archiving disks since the images it creates are missing the CRC values and it doesn't read past the end of the last file.

I have an old unfinished (only partially constructed) project for disk dumping/rewriting using the parallel port. It connects to the console via an extension board, and provides an additional connector to the RAM adapter. /ROMSEL is disabled for the RAM adapter so that my BIOS doesn't conflict. My BIOS is decoded to $8000 and the RAM adapter still provides WRAM at $6000. The plan was to decode a few registers to $5000 for the PC interface. The BIOS device would just set the RAM adapter up for reading, and generate an IRQ on each byte assembled, go through the whole disk and upload it to PC on the fly. Simple yet effective but more work in debugging than I was up for then and now :)