Oversize FDS disk images

I would like to release a few Game Doctor/Magicard disk images. They are mostly like regular FDS disks, the major difference being that most of them cram more than the standard 65,500 bytes into a side; the longest I have seen is 66,184 bytes on a Magicard disk. How to represent such a multi-disk set using the headered "fwNES" FDS file format, which assumes a fixed size of 65,500 bytes per side? Games use between 1 and 8 disk sides.

I would just amend the definition to "at least 65,500 bytes per side. To reach the next side, advance 65,500 bytes from the start of the current side, then further until \x01*NINTENDO-HVC* (begin of block 1) is found", which works well enough for any standard-size as any oversize disk.

The simplest solution might be to give those disks another extension and add a section on the FDS page defining the differences between .fds files and those? (rather than having non-FDS disks labelled as .fds files, assuming the hardware that uses those disks isn't identical to an actual FDS unit in behavior). It might help avoid confusion and allow emulators to tweak the behavior for either type of disks as needed.

The disks are read by the FDS disk drive and the FDS RAM adapter. The devices connect in-between the Famicom main unit and FDS RAM adapter, provide additional bank-switched RAM, and substitute the $8000-$FFFF area with that RAM and their own BIOS only once activated.

The question was not how to distinguish oversize from normal FDS disks, but how the oversize disk should be represented in the image, how the file should be laid out.

(example files removed per request)

If they are Game Doctor images, why not release them as such? (*.A, *.B, etc)


This doesn't seem very safe to me. Since this is a bastardized .FDS format that breaks compatibility anyway, just pick the simplest thing that works. Go off of the file size, or put the disk length in the header somewhere.

For the same reason we do not put multi-side licensed FDS games in a separate file per side: inconvenience. That inconvenience is even greater for Game Doctor games, because you need all sides right away at boot time.
Every Game Doctor and Magicard disk still uses the \x01*NINTENDO-HVC* disk header, so I do not follow.
The problem is that the length of each side differs, so one would either

• have to specify a size for each of the maximum of eight sides in the yet-to-be-defined header, or
• pad each side to the longest side. The side length then is derived from (fileSize -16) /numberOfSides.

The second solution would also work nicely with licensed FDS images, which is why I like it more.

_*shrug* Inconvenience of multiple files, or inconvenience of another new unsupported (except in your own emulator) format. I prefer the former, personally...


If a game happens to have 0x01,0x2A in it that isn't a header, then what? Better to avoid a format that can't handle special cases or need extra heuristics IMO.

There is nothing inconvenient about the latter --- if you do not support the devices to which these disks are targeted, you need not support the format. One goes with the other.

I can already see that this consultation was a pointless exercise, so whatever.

Welp, you asked for opinions, you got mine. I'm a fan of KISS.
It's your thing, do what you like. As you said, it's for your own emulator and not likely to see adoption anyway so go nuts and make a FDS 2.0.

No, I was not asking for opinions on whether to have several oversize disk sides in one file, but on how to best include several oversize disk sides in one file. "No answer is also a kind of an answer", so I suppose have received an answer. And I have made quite clear that my preferred solution is not FDS 2.0, but what I described in the initial post, which been successfully tested with almost three thousand FDS side dumps yielding slightly less than 900 sets.

But since it was so wonderful talking to you: a different reason for why separated side dumps are bad that I just remembered is that, at least for Magicard disks, the .B side and following dumps cannot be properly parsed without the information present in the .A side dumps.

If it requires redefining the existing format in an incompatible way, I would strongly recommend not redefining the existing format.

The specifics of what you create instead don't really matter, but there is a very good reason to not make fragile and backwards-incompatible changes.

Nothing that I described is backwards-incompatible, though one might have to take the time to ponder "the specifics" that "don't really matter" for a moment to understand why. I am using the same code for all disks, licensed and unlicensed, regular FDS or otherwise. And I need no lecture on the virtues of compatibility.

To be blunt, you described a messy fragile system, and when Loopy pointed out one of the sharp edges you pooh-poohed his concerns instead.

It seems like you come to this forum to ask for a final approval, and when someone points out a problem, instead of accepting and trying to address that problem you instead get irritated about it. It took you the better part of two days to finally cool down enough to even accept the issue that Rainwarrior identified with the NES2.0 controller field specification.

If you'll accept an arbitrary suggestion, I'd recommend using a novel header that explicitly specifies the padded disk side size. Yes, it is still an FDS disk in an FDS drive going to the FDS RAM adapter, but at the same time it is an overformatted FDS disk and that is what is not compatible with the existing definition.

To be blunt, you and he have not pointed out any fragility, because the objection was not based on the actual thing that I explained.
No, I come to the forum seeking direct answers to specific questions. I asked a very specific question: "How to represent such a multi-disk set using the headered "fwNES" FDS file format", and got no answer on that question. I received answers on how I should use a different file extension, that I should be using split files, that backwards-compatiblity is really important, and finally, that the data after 65500 bytes might contain 0x01 0x2A, even though my explanation was looking for a much longer string, as shown in the quoted section of that very post. So yeah, thanks for nothing.

I'll take the answers, such as they were, as "It cannot be represented at all, so use a new format", even as the reasons for it may not be convincing, but further discussion of them would be belaboring the point in any case.
No, it took me two days to cool down about people coming with (initially poorly-reasoned) change requests shortly after a specification has been implemented who have not said anything in the six months (!) it was laid out for comments, as I have explained to you before. And I still don't think my reaction to that was unjustified, and I have agreed to the post-implementation change only as a matter of compromise, not for having accepted the validity of the objection.

No. That is your interpretation of the events to justify your own actions. Rainwarrior's bug report was perfectly clear and perfectly unambiguous. Yes, his initial post was "poorly" reasoned, in the sense that it didn't identify the root source of the problem, but you didn't get defensive until after your mistake was identified.

You still seem to be operating the impression that everyone else is smarter and more infallible than you before you do something, and that everyone else is stupider and more fallible than you after you do something.

How kind of you. "You didn't make an argument I want to hear, so to shut you up I'll change it". Obviously upwards compatibility with previous versions of the standard can be wholly disregarded.

Your behavior is not doing yourself nor anyone else any favors.

—

You are being petty.
You opened the entire thread only stating the two-byte form, which is the thing Loopy objected to, which I see you have now edited out from the first post to provide no evidence of your misspeaking.

Searching for magic numbers within a file is fragile. Doesn't matter if it's two bytes or many.

I am not going to continue discussing a matter from another thread here. If you are of the opinion that I misbehaved in another thread and and would like to discuss that, post to the thread in question.
But the full form was already included in the quotation to which he responded, and according to your following comment, the length of the string makes no difference anyway, which I do not agree with.Actually, I have edited in the full form, not edited out. The only thing I have edited out was some blather about why I asked the question even though nobody would implement the application, which I later considered beside the point.
The "magic number" in question is not some far-fetched heuristic, but a defining characteristic of the beginning of any disk image. For the strategy to fail, an oversize disk would need to have the exact fifteen-byte string in the oversize area (not in the initial 65500 bytes of the side's data), something that has not occurred in almost three thousand floppy disk images, and that is not conceivable in any real-life case, as the disks in question contain images of data that was originally in a cartridge, which has no reason to contain that string. You would be correct however to the extent that the strategy would not be robust to somebody deliberately contriving an image in this manner.

In any case, you have made your point on the oversized disk side question, and I will see what I make of it.

I know little about how FDS files are handled, but for games with save features, don't these disks contain user data that could potentially match your magic string?

I do not see how that could happen, because only normal FDS games write user data to disk, and these games are by definition not oversized. Only Game Doctor et al. games use oversized disks, and being cartridge games transferred to disks, they do not save any data to disk, and could not do that even if somebody tried to hack them to do that, because the FDS BIOS is no longer accessible on most models once the RAM cartridge has been activated until the next power cycle. Even if it did happen, it would only be a problem if that user data, that happened to contain the \x01*NINTENDO-HVC* string, were saved in the oversized area, meaning after the original 65500 bytes of a side.

(Yes, this also means that games that originally had battery-backed WRAM effectively lose their saving ability when transferred to a Game Doctor et al. disk. For this reason, to compensate for that, later RAM cartridges included a "real-time save" functionality to compensate for that, similar to that of software emulators.)

Even if you're right, why make a format that relies on a bunch of assumptions? Simpler solutions exist, I don't know why you're doubling down on this point.

Just use an archive format like Zip, Tar, or GBFS. If it must be extractable on an NES flash cart, use GBFS.

Seems like you answered your own question. The ".fds" disk sides have a maximum of 65500 bytes, which actually is much higher than the real FDS disk capacity. I'm not sure what you're refering too, but your disks are not FDS disks so they cannot be stored in the .fds format and you have to come up with something else.

If you want a history lesson about fWNES, the FDS format (sort of), alternative Famicom Disk System file formats, the author/inventor FanWen YANG, that kinda thing, just ask -- I'm familiar with it a little bit because I'm one of the few (maybe the only?) people from the late 90s NES emulation/dev scene that spoke Mandarin. I did a history write-up but felt it wouldn't matter, but does go into Famicom disk formats prior to FDS (there are at least 3 I know of off the top of my head). Instead let's cut to it:

My view is the exact same as others stated here: what you have are technically disk images from a Game Doctor/Magicard unit from Bung. There are several Game Doctor-compatible copiers/units, all of which behave a little differently, no? Maybe they all use the same format, I don't know -- I don't have these devices. I do know that there are other companies that Bung who made such copiers (that's essentially what they are, particularly the later models), for example I think Venus Corp made one and strongly suspect Front Fareast had one. But there are also other systems that were used for dumping too -- if Donald Moore / MindRape was still with us, he'd love to talk about this -- like a particular model of Brother computer that could supposedly read Famicom disks.

The FDS file format is very clear: the per-disk-side size is 65500 bytes. And while bytes 5-15 of the 16-byte FDS header itself are all zero, they were intended to be used for mainly emulation purposes, e.g. "oh, we learned a new magical quirk about the FDS that happens only on some models, we need to use byte 5 for that purpose". If there IS something new/unique about the FDS itself that requires this data (i.e. is not Game Doctor/Magicard specific), then that's a different matter.

The FDS format is conceptually identical to the original NES format by Marat Fayzullin: much like the the 16KB PRG and 8KB CHR bank sizes of the NES format, this is another case of where the information available at the time (dumps, RE, pioneering efforts, etc.) made the best decision they could for that file format (FDS), and thus the 65500-byte-per-side definition. I know, lots of folks wish these formats had been more flexible size-wise, but they weren't. We must be pragmatic and err on the side of caution (read: ensuring full compatibility).

If that's not enough for you, here's a point to consider as a comparison:

There are many different SNES copiers from many companies, *all* of which use their own separate file format (sometimes not just a header!) and approaches, despite the ROM of the backed up game (obviously) being part of the image. You cannot take, for example, a .SMC/SWC file from a Super Wildcard and load it on a Game Doctor by simply changing the extension to .078. Conversion between these formats -- and there are *MANY* -- is commonly done with UCON (I haven't used this version myself, but the early 90s DOS version was a godsend). And we've seen people come along and try to introduce new file formats due (only in part) to those annoyances, as well as only loading ROMs with those headers removed entirely (i.e. pure ROM images) -- nobody but the most pedantic of the pedants care, what came first (or thereabouts) "won the war".

Likewise, I strongly doubt you can take a Game Doctor/Magicard dump that's "oversized" like this and put it onto another device/copier/system (particularly one not from Bung) and have it work. You'd need a file conversion utility, or at least something that strips off the bytes between 65500 and whatever the byte before 0x01 + *NINTENDO-HVC* was, to achieve a "raw" disk format. Heck, do you even know what those "oversized" bytes are for? Game Doctor units apparently had all sorts of neat features (especially the later models); could it be data the unit itself reads/benefits from? If so, well then there you have it, that's a file format specific to those unit(s) and should be retained that way. So the fact remains that with a disk side larger than 65500 bytes means they're not compatible with the FDS file format as designed in the late 90s.

I would urge you to make your own file format with a different extension (maybe you can find out what Bung preferred these be named, extension-wise?), document that, and/or make a conversion utility that can strip off the unwanted bytes to convert to FDS (or maybe you could talk to the UCON folks and have them do it, considering they already support a boatload of NES-related file formats, including one of the older Famicom disk formats I hinted at in my first paragraph).

NRS said:

* They are real Quick Disks
* In a real FDS drive (HVC-022)
* Connected to a real FDS base station (HVC-023)

The only differences are
1- Some of these disks are overformatted to hold, evidently, about 4-6 KB more on them
and
2- There's an extra piece of hardware between the Famicom and the HVC-023 that does simple bankswitching

Loopy earlier stated that these were originally distributed as raw disk images with single letters as extensions to specify side number. NRS objected on the grounds that with at least one system, the subsequent sides cannot be parsed without the first side.

So the fwNES FDS format can't handle it and I guess the more generic QD format has the same problem. You have either the native Game Doctor format or invent a new format that's going to introduce all kinds of new problems like iNES or whatever.

Using the existing format makes most sense to me. If multiple files really is a problem then combine them somehow (like Tepple's suggestions). File extensions are just a guide and shouldn't be relied on, I don't see how that can be a problem (use ".fds", ".qd", ".dsk", ".flp", ".img" or whatever).

Correct me if I'm wrong, but it doesn't seem that the Game Doctor images are even being talked about. Have we gone through the process of converting them into a current, working format or are we going to butt heads and masturbate egos?

Are they FDS or Famicom games?

(At any rate it's pretty interesting that you found over-sized images. It's just a bit silly that we have petty arguments and forget the points. I see you guys do this time to time; and unfortunately it seems like some of the smartest people here have some of the most fragile egos.)

They are just cartridge games ported/hacked to run from disk (that was the purpose of Game Doctor after all). I'm not aware of any original games made specifically for game doctor hardware. There's nothing special about the disks themselves (aside from the size), they follow the standard FDS disk format. I'm not sure that storing them as .FDS files makes sense - or if you go that route, I think you should indicate somehow that they require Game Doctor hardware to run. Nah, IDGAF

What NewRisingSun originally proposed is actually really close to my ideal format, I just think the way it's defined is sloppy.

Theoretically couldn't Famicom Disk System be added to NES 2.0 under Extended Console Type with a few reused redefinitions of the standard "PRG-ROM" area as "Total Size of Disk(s and Sides)"? Mapper 20 could be used.

There could be another reused area for number of disks and size per side. That way we're all just focusing on making NES 2.0 the all-encompassing 8-bit Nintendo format.

I would really strongly prefer that NES2.0 PRG and CHR ROMs only contain parallel random access memories. It just makes everything a horrible mess when the interpretation of the bytes varies more and more.

And then we can include cassette tapes. Just kidding, I think it's more clean to have separate file formats for separate media images. And yeah NES 2.0 is already confusing as it is.
I'm not sure how things like Datach are handled though.

We have a NES 2.0, why not an FDS 2.0? Here is how it could be implemented.

The original fwNES FDS defined a 16-byte header, but only the first five bytes are used. The use of the first five bytes would not change in FDS 2.0. But the remaining bytes would. Here is how a FDS 2.0 header might look like

46 44 53 1A 03 47 44 4D xx xx xx xx xx xx xx xx

Bytes 1-5 are the ASCII for FDS, the DOS end of File Character $1A and the Number of Disk Sides
Bytes 6-8 are the ASCII for GDM, which is intended to stand for Game Doctor/Magicard.
Bytes 9-10 represent the additional amount of storage space required by Disk Side 1
Bytes 11-12 represent the additional amount of storage space required by Disk Side 2
Bytes 13-14 represent the additional amount of storage space required by Disk Side 3
Bytes 15-16 represent the additional amount of storage space required by Disk Side 4

Now you might observe that this limits you to four disk sides whereas the Game Doctor can use at least eight. What do you do then? You tack on another header after the end of the data on Disk Side 4 and continue with the data. The second header is identical to the first until Bytes 9-10, which represent the additional storage space required by Disk Side 5.

I have never seen an FDS 1.0 image ever take up more than four Disk Sides, so compatibility should not be effected. Moreover, FDS 1.0 images will have a file size that is easily determined (65500 x # of Disk Sides + 16 byte header (optional)). But if it is then the user must do a second insert of the next image with a compatible emulator.

I think if you're going to go that route, just replace FDS+1A with GDM+1A directly. That would better let old emulators know that it's not the kind of file they think it is, and have a better chance of them producing a meaningful user error than having them incorrectly assuming it's a valid FDS file and trying to run it.

At that point, you can make the header whatever size it needs to be. There's no need to compromise to make it fit in 16 bytes, that's just seems a needless complication.

Get rid of the old "FDS" FOURCC, and make the rest of the header an appropriate size for the different format. You could still use the .FDS extension if you think that's important (though I don't, personally), but at least with a different FOURCC you don't have to make the cause of incompatibiliy so obtuse for people using older emulators. Leaving the old FOURCC on there will just "trick" them into trying to run something they can't.

Though an additional question: if there are 8 sides possible, is there any reason why two sides of the same disk would need to be different sizes? Are there even more than one size for the disks in question at all, or can all these GDM disks be accomodated by one size of dump, just like worked for the FDS?

And if so... then maybe just change the FOURCC and don't add anything else to the header? Parsing would then be identical to FDS file parsing but with a different predetermined disk length.

Is there a possibility that a side will ever exceed 80000 bytes?

Well FDS homebrew scene is possible. Probably already had a couple? Not sure what the limitation would be since FDSEmu and FDSStick are out. Who's to say there can't be a 6 disk, 12 side game?

FDS homebrew is already accommodated, and it doesn't really have a limit on sides. This is about game doctor, though if a fixed size can be used for that there'd similarly be no real limit for it either.

I suppose the obvious question is just how long does it take the FDS read head to go from the very beginning of the disk to the very end of the disk...

That times 12kB/sec should give the theoretical maximum possible to store on an overformatted disk.

The largest Game Doctor image I have is 66458 bytes (and that's really pushing the limits). The largest Game Doctor game I've seen is 12 sides. I guess 13, if you want to count the saver disk -- some Game Doctor devices do save states which adds another disk, I don't know if you want to accomodate that. If you want variable length disks and there's no real limit to the number of disks in a game, it'd be better to have a header per disk.

There is a physical limit to the disk size of course, and tweaking the motor speed / gap sizes you could squeeze out more bytes at risk of nobody being able to load your game. For homebrew images, I'd recommend staying within 64k for compatibility. That raises another interesting point - file size doesn't necessarily reflect disk size because gaps aren't included. You can easily make a disk image under 64k that doesn't fit by having hundreds of tiny files on it.

I like rainwarrior's suggestion. The FOURCC determines how it's parsed.
FDS^Z: Fixed length, 65500 bytes per side
GDM^Z: Fixed length, 66k (or whatever) per side. Put the size in the header for flexibility though.
Easy enough to expand. You can add a BIN type that includes GAP + CRC since I know some ppl have a hardon for that stuff.

Loopy's proposal sounds good.

Fancy Quickdisk (.FQD) Format Specification (removed)

This reminded me of all the different memory mappers used for NES/Famicom cart rips. Guess those will be emulated if we're maintaining this format.