NSF specification for FDS memory

I think the FDS memory specification needs a bit more definition.


1. When using bankswitching, does writing to RAM ($6000-DFFF) replace values in the original NSF ROM image? i.e. if you write to an area, bankswitch it out, and bankswitch it back in, should the written values persist?

I think the answer to this one should be "yes" for a few reasons.

For rips of actual FDS games, a yes or no seems to be irrelevant, as bankswitching during playback isn't really expected. I'm not certain, but after a short review of available source code this seems to be the dominant behaviour among NSF players, and this does not appear to be causing FDS incompatibility.

For hardware players, I think this generally simplifies the implementation, since the "no" behaviour alternative is copying over the old (untouched) block every time it is bankswitched in. Instead you For software players it is also simpler for the same reasons, though they aren't as much of a problem (no need to store the original ROM copy, FDS bankswitching does not have to be a memcpy operation). I believe this simplicity is why it's the usual implementation in NSF players. What does the PowerPak do?

This behaviour does come with a warning: any writes to the image may or may not be persistent across songs, depending on whether the player will reload between songs. Some software players reload for every track, but I wouldn't expect a hardware player to. This likely won't affect any existing FDS NSFs, but it's something that should be clarified.


2. When using multi-expansion, there are potential write conflicts.

This has been discussed over at the wiki, and there is already a recommendation there, but I'll outline it here in case anyone has comment. ( http://wiki.nesdev.com/w/index.php/NSF#Multi-chip_tunes )

This one is a little bit trickier. VRC6/VRC7/5B writes overlap with FDS RAM areas, so there is a high potential that memory gets corrupted when using FDS with these expansions. Since some NSF players do not seem to prevent this conflict, it's necessary to warn against using FDS with these other expansions, but I think it's worth making a recommendation for implementers.

Since it is normal practice in NSF implementations to ignore the mirror aliases for expansion registers and expect only the address in the NSF spec, blocking RAM writes at all active expansion addresses seems to be the lowest impact solution. The other easy solution I think is to disable all writing to FDS RAM during multi-expansion, but I don't like this as much since it takes away a useful property of the FDS. (Particularly, someone can procedurally generate DPCM samples in the $C000-DFFF region with FDS.)

This one is not particularly important to address for hardware implementations, since multi-expansion hardware has plenty of its own practical design constraints, but I think it would be good to recommend a "standard" behaviour for software implementers. This is another one where I'm curious, what does the PowerPak do?

Finally, NSF makers who want greater compatibility among players can be carefuly to put dummy data areas at the conflicting expansion addresses.



Anyhow, share your thoughts.

I have found four FDS rips that have a problem with mirrored writes due to bankswitching RAM, and after discovering these I've realized several players have different workarounds for this.

NEZPlug: memcpy instead of mirroring for FDS banks at $6000-7FFF.
FCEUX: memcpy instead of mirroring for all FDS banks.
NSFPlay: does not use bankswitch 6/7 to setup $6000-7FFF.

* Ai Senshi Nicol (FDS)(1987)(Konami).nsf
* Esper Dream (FDS)(1987)(Konami).nsf
* Kiki Kaikai - Dotou-hen (FDS)(1987)(Taito).nsf
* Seiken Psycho Calibur - Maju no Mori Densetsu (FDS)(1987)(C-Lab.)(WaveJack).nsf

All of these make writes to $6000-7FFF which are set by the NSF bankswitch values 6/7 to use the same bank as code elsewhere in memory. Ai Senshi Nicol, Esper Dream, and Seiken Psycho Calibur crash the PowerPak, which I presume uses a bankswitched RAM implementation, since memcpy is impractical. Kiki Kaikai seems to run fine on the PowerPak, strangely. I'm not sure how it survives.

I dislike the workarounds I've seen, especially since memcpy-style FDS bankswitching doesn't really seem feasible for hardware players like the PowerPak. FCEUX's approach is the most sensible, I think, in that it's consistent, and very much like what the FDS does, but I don't think this is the right approach. Why make a special case for the FDS? Bankswitched RAM works well for all the rest, and it could be made to work fine with all existing FDS games?

Instead of applying these workarounds, all of these can be easily "fixed" in the NSF by adding a 4/8k block of 0s and using those as the 6/7 banks. I think it's more appropriate to allow writes to mirror. Does anyone disagree?


I've quickly tested all the FDS NSFs in GilGalad's archive, and these ones were the only ones that seemed to have this problem. I guess most FDS NSFs don't write to $6000-7FFF, or if they do, don't mirror them anywhere except the unused $E000-FFFF range, so it isn't an issue most of the time? Was using bankswitch 6/7 as setup for $6000-7FFF a later addition to the NSF spec?

This feels like it's treading dangerously close to the flame wars that the famitracker boards at least used to see regularly (about how permissible is multi-chip).

My personal intuition in the case of multi-expansion is: You should expect it to screw everything up. If you want FDS in your multi-expansion, it should be RAM, it should persist, all mirrors should be present i.e. identical to hardware. This means you should expect maximal badness, and the NSF's playback engine should be written defensively to that end. (This basically means no naive writes to (depending on expansion) $9000..$dfff; they always should affect all hardware that is nominally writable at that address)

This is basically your last suggestion.

I think this approximately agrees with your second post.

I'm not really opposed to multi-expansion stuff, and I don't actually think it's a big pain to support in software (there are several fairly easy/simple solutions outlined on the NSF wiki page). If you're going to attempt it in hardware, well, it's fair to expect the person who attempts this to know how to recompile their own music code. I haven't actually heard of any hardware multi-expansion attempts use the FDS (excepting the PowerPak's emulated version). The use of FDS RAM is so rare in modern composed NSFs that even the simple solution of disallowing FDS RAM for multi-expansion would be rather effective. Anyhow, I'm not so concerned with there being an "official" way to deal with FDS multi-expansion.


The interaction of bankswitching and FDS, however, is giving me a big pain as I'm looking at it. It really bothers me that these NSF rips have this problem that every emulator seems to have its own workaround for instead of fixing the spec to disallow or alleviate the problem.

The following NSFs use the same bank number for multiple banks and then write to one of them (i.e. mirrored writes, if you're using bankswitched RAM). For most of them this doesn't seem to be causing problems, but the four I listed in my previous post do have a noticeable problem as a result. (I have excluded writes that mirror to a bank in $E000-FFFF, since that's presumed irrelevant to the NSF playback anyway.)

* 19 Neunzehn (FDS)(1988)(Soft Pro).nsf
* Ai Senshi Nicol (FDS)(1987)(Konami).nsf
* Big Challenge! Gun Fighter (FDS)(1989)(-)(Jaleco).nsf
* Cleopatra no Mahou (FDS)(1987)(Square)(DOG).nsf
* Esper Dream (FDS)(1987)(Konami).nsf
* Fire Bam (FDS)(1988)(HAL Laboratory, Live Planning)(HAL Laboratory).nsf
* Hao-kun no Fushigi na Tabi (FDS)(1987)(Carry Laboratory)(DOG).nsf
* Idol Hotline - Nakayama Miho no Tokimeki High School (FDS)(1987)(Nintendo R&D1)(Nintendo).nsf
* Kaettekita Mario Bros. (FDS)(1988)(Intelligent Systems, Nintendo R&D1)(Nintendo).nsf
* Kidou Keisatsu Patlabor - Dai 2 Shoutai Shutsudou Seyo! (FDS)(1989)(-)(Bandai).nsf
* Kiki Kaikai - Dotou-hen (FDS)(1987)(Taito).nsf
* Maerchen Veil (FDS)(1987)(System Sacom)(Sunsoft)(Sunsoft).nsf
* Seiken Psycho Calibur - Maju no Mori Densetsu (FDS)(1987)(C-Lab.)(WaveJack).nsf

A little later I'll probably write a simple tool to fix the problem ones, unless someone can make a compelling argument for one of the memcpy-bankswitch workarounds.

I think most of these are just the result of using 00 for all "unused" bank values even if they're being written to as RAM.

(edit: fixed the list, there were a few false-positives due to a bug in my detection code)

I'd have to say no.

NSF is a ROM based format. Once you load a bank into a certain address range, it overwrites that address range. No values remain from the previous bank or written values in case of FDS RAM. This applies to FDS rips as well.

For FDS rips, when you rip a NSF from a FDS game there are many steps you have to take. You have files instead of banks. Those need to be converted and merged together as banks.

Once a bank is loaded, $6000 - $DFFF is then treated as RAM. Anything can be written there, more or less self-modifying code/data. During play, you don't have to worry about it much as the main code is taking care of reading/writing to RAM areas(sometimes it needs to be modified). As far as the initialization is concerned, a ripper needs to account for any writes to any address no matter if it's in the FDS RAM area or not. If it's not done this way, it may not work on the real hardware. Not to mention there will be incompatibility between other NSF players. Because every programmer implements NSF emulation a bit differently in some respects.

If I ever wanted to change anything, it's allowing $6000 as a load starting point for FDS rips. Some players don't support it, some do. I start all mine at $8000 for compatibility reasons. Then manually load those banks in the address range $6000 - $7FFF. And/or use the bankswitch values in the header as well.

Those NSFs that don't work right simply need to be fixed and fixed for compatibility between software and hardware players. Although, I always try to make sure a NSF works on the real hardware. I have a PowerPak to test my NSFs.

I do realize that the PowerPak also emulates mapper controllers and some sound expansion chips. Other than how the PowerPak initializes, everything else is NES.

Well, just one thing at a time. You wrote about lots of different things. I'll answer this one for now. You may or may not agree with me. I don't emulate, I'm a ripper.

One last thing. The NSF file should NOT be modified in any way outside of loaded memory. The FDS Disk I/O functions should be completely disabled. And hacked out of a NSF.

Hmm, I didn't notice load addresses in the $6000 range were disallowed in the spec. This seems like an oversight to me, since without bankswitching turned on it seems some FDS games should require it. I know that NotSoFatso does not allow it, but are there any other players that do not? (NSFPlay, NEZPlay, FooGEP, Nestopia, FCEUX, VirtuaNES, VirtuaNSF are all fine with it.)


I understand that the FDS does not really have banks, it just has non-bankswitched RAM. The problem is that NSF as a format forces all memory configurations into 4k banks. Banks are normally just manipulated high address lines, which can create mirrors. I think it's more consistent to implement all NSF bankswitching this way (not to mention easier in hardware emulation), and not have a special case for FDS; it's entirely adequate for FDS rips, but they have to be made with this in mind.

The specific problem is this: when the bankswitch block for and FDS rip contains mirrors, the ripper must account for writes to mirrored addresses. If you don't care what goes in $6000-7FFF, don't mirror code/data in those areas. In the four problem NSFs I've identified, the bankswitch block looks like this:

01 02 00 00 00 00 00 00

The problem is the two 00 00 on the end. These rips tread $6000-7FFF as RAM, and do not read any code/data from this region. The ripper has chosen a "default" bank of 00 to put in this area instead of creating a new initialized bank. This kind of thing is outside the territory of just setting it up like the original FDS version and presuming the emulation is accurate; this is now a deliberate statement that the same data should be put in places it wouldn't normally be.

Now, areas that are empty or only used for RAM need to have -some- value in them, but what I'm arguing is it's not acceptable to just stick a randomly chosen bank in them (like 00) because it has a convenient number. If $6000-7FFF is not supposed to be filled with unique data from the disk image, and is just treated like RAM, it should be explicitly initialized. Write areas should not be mirrored.

i.e. the bank table should look like this instead:

01 02 00 00 00 00 03 04

I already outlined three different workarounds to this problem, and I think they're all inadequate. NSFPlay ignores the spec and doesn't use the last two bankswitch bytes to set up $6000-7FFF. NEZPlay treats bankswitching in $6000-7FFF as copy-to-RAM but allows mirroring in the rest. FCEUX treats all FDS bankswitching as copy-to-RAM (admirable, but I think unnecessary). All of these just hide the problem and prevent practical implementation on bankswitched RAM hardware.

So anyhow, what bothers me is that the behaviour of banks when FDS is enabled is underdefined by the spec, and we have 4 different implementations (none of which have much to do with authentic emulation of the FDS) trying to account for they grey area. The definition I'm suggesting is one that would not break playback for any of these player implementations, and is the only one that doesn't break playback on the PowerPak.


I guess there is another side of this is that I neglected to state, but it should be forbidden to take advantage of mirrored writes as well (i.e. write once, read it from two different locations), since they are impossible on the FDS, and will break some of the workarounds I mentioned.

So, at the moment I've included this information as guidelines in the wiki:

http://wiki.nesdev.com/w/index.php/NSF#FDS_Bankswitching

As you can see, I've not made it a revision to the specification, just included it as advice for someone trying to make an FDS NSF that is compatible as possible. Maybe that's enough to deal with the underdefinition in the spec itself.

I'll fix those four NSFs and post them here in a little bit.

Here are fixed versions of the 4 problem NSFs:

Ai Senshi Nicol (FDS)(1987)(Konami) a.nsf (attached)
Esper Dream (FDS)(1987)(Konami) a.nsf (attached)
Kiki Kaikai - Dotou-hen (FDS)(1987)(Taito) b.nsf (attached)
Seiken Psycho Calibur - Maju no Mori Densetsu (FDS)(1987)(C-Lab.)(WaveJack) b.nsf (link to attachment below)

All of these are now free of mirrored write problems. Seiken Psycho Calibur still will not run on a PowerPak for reasons I haven't yet determined.

I made these fixes with these python programs:
fix_fds_nsf_a.py (attached)
fix_fds_nsf_b.py (attached)

The a version creates two 0-filled banks for $6000-7FFF. Suitable if all mirrored writes are in the $6000-7FFF area and it's acceptable to initiallize that area to 0. Adds ~8k to the file.

The b version is more thorough, expanding the file to a full 32k of duplicated banks. This fixes mirrored writes in all memory regions, and doesn't change the initialization values.

There are 9 other NSFs that I have seen mirrored writes in (see above), though I had not identified a playback problem with them when listening. I will test them on the PowerPak.

Edit: all of the listed NSFs have problems on the PowerPak, except Fire Bam. Fire Bam mirrors writes from $6000 to $8000, but by stroke of luck, all the writes to the $6000 region ($60CC-67FF to be precise) are copying bytes from the mirrored location in $8000, so no problem is created.

Edit 2: attaching files (dropbox links now broken)

Edit 3: Python scripts were later disallowed on this forum. Uploading a ZIP with what I think was the original script.

FDS support was added to the spec later on after the spec had already had it's basic design. As you can see from the updates. I have no clue what was going on during the design phase but something had to be done and that's how it ended up. Those players support loading at $6000 now. Some of them did not through the years and so a few of them are left with this issue. Hence setting load at $8000 to remain compatible with those older players and those that still don't support loading at $6000. I talked to kevtris about it today and he said that $6000 is a valid load address. Even though it doesn't say it in the spec.

V1.61 - 06/27/2000 Updated spec a bit
V1.60 - 06/01/2000 Updated Sunsoft, MMC5, and Namco chip information
V1.50 - 05/28/2000 Updated FDS, added Sunsoft and Namco chips
V1.32 - 11/27/1999 Added MMC5 register locations
V1.30 - 11/14/1999 Added MMC5 audio bit, added some register info
V1.20 - 09/12/1999 VRC and FDS prelim sound info added
V1.00 - 05/11/1999 First official NSF specification file

I also talked to him about expanding the header bytes that tell which initial banks are loaded when the NSF is loaded. Probably too late to do anything about it since the spec is set in stone now.

That's happened to me a few times. Sometimes the NSF needs to be ripped again from scratch. That often solves the problem if done right. I very seldom work with an existing NSF.

There could also be a memory conflict, which is one good reason why. Any of the main three addresses could be off. Anything really. Just need to track it down. You could disassemble the PowerPak NSF ROM and look at that. I've done it.

Loopy did post the source for the NSF ROM here: http://3dscapture.com/NES/

I don't know what goes on inside the mapper, though.

I've been trying to find writes to weird addresses, or reads in the $4000-5FFF area in the FDS NSFs that crash, but haven't spotted anything unusual yet.

I found Namida no Soukoban Special (FDS)(1986)(ASCII).nsf, which reads $4015 and appears to rely on correct behaviour from this register (actually exposed a bug in NSFPlay that I will have fixed in the next version), but this works fine on the PowerPak.

Attaching Seiken Psycho Calibur NSF to update broken dropbox link above.

Well, it'd be much harder to do what Gil-Galad says in hardware, as that *requires* a memcpy, than to just treat it as bankswitched RAM, since it's just tweaking those high address lines in normal mappings.
dead link?

A real FDS rip will only need to bankswitch in INIT, which is generally kind of agnostic as to whether that's a copy or not. However, the actual "bankswitch" here is a BIOS call which has been replaced by an NSF bank register write. There's no mechanism for actually doing the hardware copy here, so the ripped version should not expect a copy operation, IMO.

Homebrew NSF becomes a problem for this, as it generally just expects bankswitching RAM semantics, and then multi-chip NSFs are an even worse problem where you should probably write protect the FDS RAM to keep it from getting clobbered by other expansion audio register writes. It's a big mess.

PowerPak and probably any "normal" hardware implementation is just going to use bankswitched RAM, so reused banks should mirror writes to match that, IMO. It was unfortunately never codified in the NSF specs how it should behave, though.


Updated.

It almost seems like it would make sense to add a bit that specifies whether FDS RAM is writeable or write-protected...

Well, the heuristic which I was going to implement in the next version of NSFPlay is just to assume write protected for multi-chip. Otherwise homebrew FDS NSF seems to be very well behaved w.r.t. writes to FDS RAM area, and of course many commercial FDS rips rely on it as RAM.

Currently in NSFPlay it's just an option to write protect $8000-DFFF, but that means you have to turn it on and off for both cases. I'm going to replace that with 3 options (auto, RW, RO).

You could add it to the NSF spec if you wanted, but I kinda think the heuristic is good enough as-is.

I was thinking of adding the heuristic to the spec once we get data on how effective it would be. If a multi-chip NSF needs to build DPCM samples at runtime, it can map the same memory into $6000-$7FFF and $C000-$DFFF and write through the $6000 and $7000 windows. But to continue refining this heuristic, I have two questions:

• MMC5 can also map (one or two separate chips of) RAM into $8000-$DFFF. Should NSF emulate that in some manner?
• Is the FDS BIOS available to FDS NSFs in any way?

As far as I know those two cases do not yet exist. I think there's hundreds of "edge case theoretical combinations" to do with multi-chip things, so I'm a bit reluctant to instigate a format revision to accommodate ideas like this.

Actual extant problems:

• Copy vs mirrored RAM semantics. Many (bad?) rips rely on the copy, and are incompatible with PowerPak at least. (Solution: make "copy" semantic an option in your player, and/or try to make agnostic rips.)
• Multi-chip NSFs clobber code/data in RAM by accident. (Solution: heuristic I proposed.)

Neither of these seem to require a format extension, in my view?

And no, the FDS Bios is not available for NSF rips. If needed at all for a rip it's very easy to work around. We've done just fine without it, and I think it's best that a BIOS not become a prerequisite for an NSF player.

There are NSFs out there that depend on copy semantics for FDS RAM? Argh.

Mildly offtopic, but my heuristic for RAM/ROM detection is to emulate both and compare the results on every read. If the values differ, the address overlaps a port and the write-protect status is undecided, the RAM emulation is discarded and write-protect is enabled. If the code isn't buggy it should never write to locations it believes to be ROM, so a write anywhere other than a port causes write-protect to be disabled. Due to a few homebrew NSFs checking for write-protect and refusing to play without it (I'm looking at you w7n) I hacked it to treat $8000 as a port.

It's a big fat kludge, but it plays everything I have, including headaches with multichip engines that initialize everything but only specify FDS in the header.

I identified and posted corrected rips of all the ones I could find earlier in this thread, but yeah unfortunately the old versions are still floating around out there.


FWIW buggy code is easier to identify on an NSF, since they're deterministic. A lot of stuff can be tested in an automated way.


Ha! That's interesting.

I would just say that multi-chip just implies write protect for FDS at $8000-$DFFF. I don't think it needs to be more complicated than that. The only common hardware implementation I know of for this is the TNS-HFC series of cartridges, and they bypass the FDS RAM just using the normal NSF setup (RAM at $6000, ROM at $8000+).

Without multi-chip there's really never any reason to write protect the FDS RAM, since there's no conflict necessitates writes to it. (Very easy to test and make sure it doesn't write by accident due to the determinism, too.)

Using the header in a heuristic is only good if the header is accurate. Unfortunately, many of them aren't. While that's arguably not my problem to solve, I do like my tools to "just work", so I figured autodetection was worth it.

If only the original standard had banned FDS RAM and forced rippers to port games to WRAM...

Oh, sorry I missed that line. That's a bit unfortunate, but I'd also probably be content to just say that's a bad NSF and shouldn't ever play.


That would have made many actual FDS rips far too onerous, IMO. The FDS RAM is good. (Though I think the oldest version of the NSF spec didn't account for FDS at all. The concession to bank $E000 at $6000 was an addition in a second revision.)

If I could go back and stop something from the standard it would have been using a bitfield for expansions instead of just allowing one. There's been a lot of good work as a result of that accident though, so maybe it was okay overall.

Seiken Psycho Calibur should be working on Powerpak now (fixed a bug in the NSF mapper).


Sounds like a good idea, so I've done the same in Powerpak. $8000-DFFF gets write protected for multi-chip.

On the NSF spec topic: Should initializing $408A be in the spec? According to this, most tunes expect a BIOS-initialized default.

Very nice to hear the PowerPak NSF player is getting an update!

Looking at my notes in NSFPlay:
https://github.com/bbbradsmith/nsfplay/blob/9aae2ae0f7b737248198911fa283fea04d8666ff/xgm/devices/Sound/nes_fds.cpp#L108

So there seem to be three BIOS writes that are important:

• $4023 = $83 (enable sound i/o)
• $4080 = $80 (volume envelope)
• $408A = $E8 (envelope speed)

I agree this should be documented somewhere, maybe the FDS Audio page on the wiki, possibly also on the NSF page.