Fare thy well iNES, thou shalt not be missed.
Details about iNES death.
I LOATHE the idea of using text based formats for things that are not text based. Writing parsers is a huge pain, and there's no reason why a properly designed binary format could not be every bit as flexible.
Not to mention an XML based format could only practically contain header information and not the actual ROM (not unless you convert all the ROM to text -- which would just be silly) which means you'll have to have the ROM image in a seperate file -- or multiple seperate files.. which ultimately just makes the format cumbersome and awkward. Of course you could .zip the files up or something, but then that's another pain in the ass the emu would have to add support for just to run a ROM.
Anyway -- I don't like this idea at all. I'd much rather see something like iNES 2.0 (what happened to that?) or even a good attempt to adapt UNIF to be more complete.
XML always seemed more trendy than functional to me.
I love Marty and he does a lot of great work -- but this is one of those things I really hope just flops and goes nowhere. I really think its success would just be a big headache. Emudevers would have a whole slew of new obstacles to overcome just to support these files, and end users would be no better off.
In that thread Marty says he spent days on an XML parser. I can write an iNES loader in about 5 minutes. And I don't really see anything there that makes it more descriptive than UNIF could be.
I'm probably in the minority though =( Judgeing from the positive response it's gotten in that thread.
Well, I'd miss iNES. It may not describe a cart accurately enough, but it's really simple to maintain. The last time I tried to do something as simple as extracting the binaries from a UNIF image I almost killed myself. To me, being able to perform such simple tasks without the aid of special software is essential. This XML approach seems simple enough though.
<iNES> The reports of my death are greatly exaggerated
I've said it before (on that thread IIRC), and I'll say it again. You cannot replace INES. I don't care if you have an idea that's a million times better - you'll never get rid of the ROMs already floating around. If it were so easy, then why is that bad dump of SMB so much easier to find than a clean dump?
The main thing I like about Marty's work is the database application. I've always wanted to see a *good* ROM utility that can fix bad headers, even convert to NES 2.0, based on an accurate, updatable database. Such a utility will be much easier to implement once a standard method is available for building the database. Indeed, the next version of Nestopia will have its database in the XML format instead of the previous binary format (which would sometimes change in structure from one release to the next). When the new version is out, I hope someone will make a ROM fixing utility that takes advantage of the new database.
The idea for ROM distribution (as I understand it) is to use a zipfile, containing one XML file along with a PRG file and a CHR file (identified by the XML file). It sounds nice, but I don't think it'll catch on. Too much stuff out there (such as the PowerPak) relies on INES, and no one is going to switch formats unless the newer style is backwards compatible. If you want XML info included with the ROM, you're better off appending it to the INES image instead of putting it in another file within an archive.
Minor letdown. When I saw the subject, I thought it was supposed to mean Marat has quit.
Disch wrote:
Of course you could .zip the files up or something, but then that's another pain in the ass the emu would have to add support for just to run a ROM.
Tell that to Sun or Microsoft. The .jar format of Java, the .sx* format of OpenOffice.org 1, the .od* format of OpenOffice.org 2, and the "OOXML" formats of Microsoft Office 2007 are all zipfiles containing XML plus extra resources. Zip libraries and XML libraries are a dime a dozen.
The zipfile would look like this:
- Classic Concentration Camp (PD).zip
- index.xml
- camp.prg
- camp.chr
- title.png
- ingame.png
- box.jpg
- label.jpg
Or index.xml could state that the PRG and CHR are in an iNES file:
- Classic Concentration Camp (PD).zip
- index.xml
- camp.nes
- title.png
- ingame.png
- box.jpg
- label.jpg
Enforcing copyleft would be easy:
- Classic Concentration Camp (PD).zip
- index.xml
- camp.prg
- camp.chr
- COPYING
- src.zip
But if we do go this route, please make the CRC or SHA-1 field optional so as not to complicate debug builds of homebrew.
The one and ONLY way to kill ines is to have an emulator that is significantly better, does not support ines, and has a place to download the already converted roms. If its not better nobody will switch. If it supports ines people won't use the new format. If people cant get every rom in the new format nobody will use the emu.
ines is mostly good enough, ines 2 is better but usually not needed, and an xml type was already tried (and generally failed) with unif. Yet another format with more complexity will not help. Add in the variations with a text description (74xx161 vs 74hc161 vs 74HC161 vs 74ls161 etc etc etc) makes the decoding far more complex. If you already have multiple files there is no reason a full description couldn't be included, just not used by the emulator.
Interestingly I didn't see anything like box art, manuals, game genie code, walkthroughs, etc to be included inside the game file, but I only looked at the first couple pages. That might be one feature that would get me to switch to a specific emulator and would be good for both playing and preserving. But again if you already have multiple files, including those in a folder along with a normal ines wouldn't be any different.
Isn't Marty a God? He has created the sunlight, the earth, the community... and everything has become a NEStopia. Now, he (?) has declared "death" of... iNES format? I would laugh a lot... but i'll just ignore.
Fx3 wrote:
Isn't Marty a God? He has created the sunlight, the earth, the community... and everything has become a NEStopia. Now, he (?) has declared "death" of... iNES format? I would laugh a lot... but i'll just ignore.
You're being a jackass because your emulator sucks and you're jealous. Please grow up.
- Actually, it was a joke from me... I'm sorry if you took my comment as... bad. Plus, it's quite strange that you flame my emulator. It's not crap, and I'm sorry if you think it is, it's not a sign of intelligence.
I apologize for my lack of humor and intelligence.
Nessie wrote:
I apologize for my lack of humor and intelligence.
Fair enough.
I'm still using basic iNES, with mapper-correction if "DiskDude!" is present.\
Fun fact: Google indexes NES roms. Google "nes diskdude" for fun.
Dwedit wrote:
Fun fact: Google indexes NES roms. Google "nes diskdude" for fun.
As far as I can tell, this is because so many web servers are configured to fall back to a "text/plain" or "text/html" content type for unrecognized file name suffixes.
What's wrong with iNES? I remember hearing some weird cartridges, which were probably pirate/unlicensed, didn't work with the iNES format cause of weird things they did or something like that. I agree that nothing will replace iNES headered NES roms unless it is backwards compatible. Even then you'll always have to support the old format.
Most people here only talks from a romz0r perspective about the format and in that way I can see why iNES never will die..
But from preservation point of view iNES sucks big time and I'm guessing this is the reason why Marty made his proposal. I pesonally would like to see a more descriptive format intended for preservation of the games/box-art/manuals for future generations when all carts have bit-rots and all box-art/manuals are gone.
OP here, sorry if OP post was a little exagerated, it was to get your attention
Why this new format with Marty and BootGod in command? Because many in the community want to get rid of that piece of junk called header embedded with the binaries. We wished emulators to be able to run raw binaries without any addition. Besides, this format allows to integrate a lot more infos that any header could, in human readable format.
FAQ:
Q:Which emulator will support that?
A:Nestopia, and hopefully more eventually
Q:Who will maintain the Database?
A:BootGod, so that you never run again a rom with a wrong mapper/herdaer
Q:Are the new roms going to be easy to find?
A:All you need to do is batch remove headers with uCON64. And yes, they will be available rather easily.
dXtr wrote:
Most people here only talks from a romz0r perspective about the format and in that way I can see why iNES never will die..
But from preservation point of view iNES sucks big time and I'm guessing this is the reason why Marty made his proposal. I pesonally would like to see a more descriptive format intended for preservation of the games/box-art/manuals for future generations when all carts have bit-rots and all box-art/manuals are gone.
But there is no reason the emulated rom cant still be ines (1 or 2), and then you add the exact same zip folder that contains the more detailed hardware description and artwork. ines is good enough for the emulator to play the game, so mapper info shouldn't be lumped in with other info when it just adds complexity. Keeping ines doesn't in any way restrict you from having and preserving those box/manual/label art files. Your new xml could even just say "use that ines rom" instead of "use that bin file" to keep everything backwards compatible.
X-or wrote:
Because many in the community want to get rid of that piece of junk called header embedded with the binaries.
People who consider there to be the One True Thing, with everything else being crap and bad, I want nothing to do with. These people cannot perceive the nuances of things, that
everything has problems, and ignore the practical issues. That style of thinking just sucks.
bunnyboy wrote:
Keeping ines doesn't in any way restrict you from having and preserving those box/manual/label art files. Your new xml could even just say "use that ines rom" instead of "use that bin file" to keep everything backwards compatible.
But iNES is EV1L!!!11!! We must be cleansed of it! But really, that'd be perfect, because then any emulators which can open iNES files embedded in a .zip would work already.
Evil... but it's the format around for many years.
Personally, UNIF looks fair enough to store the ROM and its properties.
MottZilla wrote:
What's wrong with iNES? I remember hearing some weird cartridges, which were probably pirate/unlicensed, didn't work with the iNES format cause of weird things they did or something like that. I agree that nothing will replace iNES headered NES roms unless it is backwards compatible. Even then you'll always have to support the old format.
The technical problem with INES is ambiguity. You cannot distinguish between MMC3 and MMC6 (both of which are mapper 4), nor can you disable WRAM explicitly (there's always 8K implied). Some mappers (such as 185) cannot be accurately emulated without additional info which isn't in the INES spec. You cannot declare WRAM sizes smaller than8K, nor can you declare battery-backed CHR-RAM. There's no room for VS system info, and there's no way to identify a ROM as being both NTSC and PAL. Most of these problems (if not all) can be resolved by adopting the kevtris' NES 2.0 specification, but without a reliable tool for performing the update in a database-guided manner, that spec will never go anywhere.
Further problems arise in the way actual ROMs are encoded. Many ROMs have simply bad headers, and those that have "good" headers almost never have the WRAM and PAL fields set properly. Even the battery RAM bit isn't reliable in my experience. These problems wouldn't be an issue if a reliable ROM fixer were available, but AFAIK there is none.
X-or wrote:
Because many in the community want to get rid of that piece of junk called header embedded with the binaries. We wished emulators to be able to run raw binaries without any addition.
Perhaps I'm misunderstanding, here... but from what I've been reading this new format does nothing for either of these. True the header may not be part of the binary file ... but it is part of the entire image package. And when you consider the image is likely to be zipped -- then it
might as well be embedded... since, for all practical purposes... it is.
And you won't ever be able to run NES ROMs without additional information. Just as you can't run an iNES ROM without its header... you won't be able to run these ROMs without their descriptive XML files. (At least, not without doing a CRC or hash check or something of that sort)
So I fail to see how this format accomplishes what you were talking about here.
Quote:
Besides, this format allows to integrate a lot more infos that any header could, in human readable format.
UNIF could be just as descriptive. Perhaps not as human readable -- but emulators aren't human, so that's not really a big selling point in my book. If you want a human-readable database -- make a webpage (like BootGod's). ROM binaries should be just that -- a binary.
I'm looking at this format from a practicality standpoint... and I keep asking myself these questions:
1) does this format make the
emulator more functional?
2) Will end users really be looking at these XML files in a text editor?
I certainly don't see #1 being the case. And there might be a few people interested in the #2 feature -- but again those people would (or should) be just as satisfied with a seperate source (like BootGod's site). Either way -- the additional workload that's passed on the emulator here, and the complication involved overwhelmingly outweighs these benefits, IMO.
Quote:
Q:Who will maintain the Database?
A:BootGod, so that you never run again a rom with a wrong mapper/herdaer
This is what gets me.
NEStopia has had a ROM database like forever... and it's worked well. I was planning on implimenting a database feature in my emu as well -- but stored externally so that it could be modified by others and updated without having to build a new binary.
Here, a text based, human readable format is understandable (and even a good idea). But is it really necessary to gut the ROM binaries, make a new format, and compromise backwards compatibility to do this? Of course it's not. NEStopia has been doing just fine for years -- I don't see why all of the sudden these databasing techniques aren't good enough any more.
KISS. XML is not simple or even easy to work with. Hell, it's not even the easiest text based format for a human to read. There's all this additional complication here... and really... how will anybody be better off?
Quote:
Q:Are the new roms going to be easy to find?
A:All you need to do is batch remove headers with uCON64. And yes, they will be available rather easily.
More proof of my point.
If a conversion can be done with a tool -- then why can't this conversion be done internally by NEStopia? Why the new format? It just doesn't make sense to me.
Hundred bucks says NEStopia will still even do these kinds of conversions and database lookups even after this new format is supported (because of the large userbase still using iNES ROMs). So really... what's the point?
Disch wrote:
Perhaps I'm misunderstanding, here... but from what I've been reading this new format does nothing for either of these. True the header may not be part of the binary file ... but it is part of the entire image package. And when you consider the image is likely to be zipped -- then it might as well be embedded... since, for all practical purposes... it is.
Unlike with iNES[1] or UNIF, tools for hand-editing a zipfile and XML files are distributed with all computers that run the Windows XP or Windows Vista operating system.
Quote:
UNIF could be just as descriptive.
Microsoft, Apple, and Canonical will probably never distribute UNIF tools with their operating system distributions.
Quote:
1) does this format make the emulator more functional?
2) Will end users really be looking at these XML files in a text editor?
I certainly don't see #1 being the case. And there might be a few people interested in the #2 feature -- but again those people would (or should) be just as satisfied with a seperate source (like BootGod's site).
If the database updates come from a central source, then who updates this source when CaH4e3 posts a new dump or PDRoms posts a new homebrew?
Quote:
NEStopia has been doing just fine for years -- I don't see why all of the sudden these databasing techniques aren't good enough any more.
If someone actually starts writing NES programming tutorials of the same caliber as the available Atari 2600 programming tutorials, as people are suggesting in
a recent topic in the newbie forum, we might get more NES homebrew developers on board. These tutorials will need to specify how the emulator knows what board to emulate. If we want to rely on a database for board identification even more than we do now, what will newbie homebrew developers think?
[1] Sure, you could use MS-DOS Editor in its little-known binary mode to make an iNES header, and I've done that on several occasions. But isn't XML easier than that?
tepples wrote:
Unlike with iNES[1] or UNIF, tools for hand-editing a zipfile and XML files are distributed with all computers that run the Windows XP or Windows Vista operating system.
I never said XML would be harder to edit. But ROM Hackers and other people that edit ROMs and headers would (hopefully) be familiar enough with hex editors and other such tools so that on the off and somewhat rare occasion they'd need to actually edit a header in a way that existing tools don't already provide an interface for, they'd still be able to so with minimal difficulty.
Besides.. if you want to talk about editing efficiency... a new format that is completely incompatible with existing mainstream patching systems is riddled with problems. ROM hacking and translation sites (read:
huge global communities -- more people care about hacks/translations than ROM databasing) are pretty much driven by IPS submissions -- and IPS simply will not work with this "slap a bunch of files in a .zip" approach.
Quote:
Microsoft, Apple, and Canonical will probably never distribute UNIF tools with their operating system distributions.
Your point?
A UNIF editor would still be easier to write than an XML parser.
Quote:
If the database updates come from a central source, then who updates this source when CaH4e3 posts a new dump or PDRoms posts a new homebrew?
Uhh... the site maintainer?
Regardless of what method you use ... any sort of ROM or image database is going to require some kind of central maintanance. Even if the database is a series of ROM images -- you'd still need someone to weed out improper hacks, typod/bad xml files, etc, etc. A flexible format doesn't somehow make the job of quality control and management disappear.
And in the case of a homebrew -- I fail to see why a UNIF or NES 2.0 header wouldn't suffice.
In fact -- for most homebrews, even a basic iNES header work suffice. People complain a lot about its ambiguities, but at the end of the day it really is good enough to get ROMs running. And unless some homebrewer is developing custom cartridges that function differently from existing boards, there's really nothing in a homebrew that needs to be databased in this detailed fashion.
Quote:
These tutorials will need to specify how the emulator knows what board to emulate. If we want to rely on a database for board identification even more than we do now, what will newbie homebrew developers think?
Right...
Code:
<game name="Shadowgate" system="nes.ntsc" crc="6A1F628A">
<hardware board="nes-tkrom" mapper="4">
<prg type="rom" size="128k" crc="591364C9" />
<prg type="nvram" size="8k" />
<chr type="rom" size="128k" crc="05414DD9" />
<nmt mirroring="ctrl" />
<mmc>mmc3b</mmc>
</hardware>
</game>
It's
soooo much easier explaining all of that in a tutorial than it is to explain what "mapper 4" signifies.
But at any rate... how would we be relying on databases
more than we do now? I think we'd be relying on them just as much as we do now =P
Regardless... a database is only needed where iNES isn't sufficient. You may grill me for this... but for homebrewing...
iNES is sufficient.
The lynchpin of UNIF and this XML format is that it's about preserving the original hardware. In fact, from what I'm reading in that thread, that seems to be the critical selling point here: databasing. In fact, one of the
quotes in that thread:
Quote:
iNES, any version, only cares about giving the people games to play. The goal of the XML-based format is about preserving how the hardware actually works.
This is the key distinction. Preserving hardware information is what a database is for. Getting the game to run in an emulator is what ROM image binaries are for. You're talking about two very different functions, here, and while they aren't necessarily mutually exclusive... combining them doesn't make the database portion of it any easier to maintain... nor does it offer the end user any benefit.
Databasing in a file format isn't really an advantage to homebrewers, ROM hackers, and translators unless they're producing their own custom boards. When you're talking about preserving the hardware aspect of things -- software produced after the fact which piggybacks existing boards doesn't factor in. What good does saying which specific MMC3 version X homebrew game runs on when it never had cartridges produced and would function equally well on any generic MMC3 setup? What's the point of databasing theoretical hardware that never actually existed?
This is one time where the beauty of iNES shines. Strength in simplicity. iNES may be completely lousy for databasing -- I'll agree there. But when you start talking about homebrewing and PD ROMs, you're starting to lose me. I don't see how this new format is of any greater value to that audience.
"Preserving hardware" is a gate/symbol-level schematic diagram or edge-accurate HDL file of ICs accompanied by PCB schematics and high resolution scans, not a useless XML file.
kyuusaku wrote:
"Preserving hardware" is a gate/symbol-level schematic diagram or edge-accurate HDL file of ICs accompanied by PCB schematics and high resolution scans, not a useless XML file.
Awesome speech!
And where are your "gate/symbol-level schematic diagram or edge-accurate HDL file of ICs accompanied by PCB schematics and high resolution scans" ?
BootGod is the xml author, his name alone justifies it's all but useless.
Disch wrote:
you'd still need someone to weed out improper hacks, typod/bad xml files, etc, etc. A flexible format doesn't somehow make the job of quality control and management disappear.
Which should happen with ines, but nobody wants to be the maintainer. Nestopia and other emulators could have fixed roms using their databases long ago.
Disch wrote:
In fact -- for most homebrews, even a basic iNES header work suffice. People complain a lot about its ambiguities, but at the end of the day it really is good enough to get ROMs running. And unless some homebrewer is developing custom cartridges that function differently from existing boards, there's really nothing in a homebrew that needs to be databased in this detailed fashion.
Which again could be solved with a maintainer (and probably ines 2). A new board would get a new official number instead of something picked randomly. This new format will already need a maintainer to set what options are allowed in the xml, so nothing is any different.
Until there is a netlist mapper format any custom mapper will be just as ambiguous when described in text or a number. How would the emulator be able to add support by itself using the xml description of a board that does chrram banking and unrom type prg banking with different data bits? If you require a person to add that to the emulator its no different than calling those specs mapper #x.
bunnyboy wrote:
Until there is a netlist mapper format any custom mapper will be just as ambiguous when described in text or a number. How would the emulator be able to add support by itself using the xml description of a board that does chrram banking and unrom type prg banking with different data bits? If you require a person to add that to the emulator its no different than calling those specs mapper #x.
Agreed completely. I actually often debate this very same topic with a friend on IRC. The format in which the data is interpretted by the emulator, be it in text or binary form, is ultimately arbitrary.
And if it's all arbitrary -- you might as well go with a format that's easy to work with.
Disch wrote:
bunnyboy wrote:
Until there is a netlist mapper format any custom mapper will be just as ambiguous when described in text or a number. How would the emulator be able to add support by itself using the xml description of a board that does chrram banking and unrom type prg banking with different data bits? If you require a person to add that to the emulator its no different than calling those specs mapper #x.
Agreed completely. I actually often debate this very same topic with a friend on IRC. The format in which the data is interpretted by the emulator, be it in text or binary form, is ultimately arbitrary.
And if it's all arbitrary -- you might as well go with a format that's easy to work with.
Agreed++. My opinion is to
accept new ideas, even if this one (XML) looks controversial. I still believe that UNIF
is the best choice, though it needs to be more popular between emulator authors and the community (read: players). So, the users will make the choice.
At any rate, I'm not a player. I'm interested to develop my emulator for accuracy, by analyzing the CPU, PPU, APU... and making it as closer as possible of the NES.
The emulator database is a good idea, but why can't be shared between emulation authors, like a .dat file much like MAME?
X-or wrote:
And where are your "gate/symbol-level schematic diagram or edge-accurate HDL file of ICs accompanied by PCB schematics and high resolution scans" ?
My mapper logic is in a folder on my desktop. Where is your emulator that can interpret my Verilog?
Just because bootgod is behind this format doesn't make it smart.
tepples wrote:
Microsoft, Apple, and Canonical will probably never distribute UNIF tools with their operating system distributions.
Nor will they ship with NES emulators. Your point is moot.
Am I the only one to be really tired of people want to do their own format instead of iNES, failing very miserably the ones after the others ? After all a standard is a standard no matter how much people like it, and there is no way to change this.
- "Hey, why not do a format that describe every transistors and connexions that are in the chip of a cartridge in a huge text file so that the emulator can emulate the game accurately ?"
- "No that won't be accurate enough, we absolutely need a format describing every atom in the cartridge for proper emulation !"
For me a game, so it's ROM image, is an "idea" not a material, hence the idea should be emulated no the material.
It seems like it'd make more sense to have an external database with all of this info, e.g. something along the lines of the MAME "extras". This way, the main format doesn't change at all (INES), and for people who must preserve everything, an external package of boxart/screenshot/what-have-you could be added.
Adding extra bullshit reduces the ease of hex editing or manually tweaking existing things, and reduces distributabilty.
A database could be built off the ROM image portion (using a hash/checksum/whatever) and ignoring the header, so that if broken headers are a problem they can be fixed... the tool to do this automatically could even have a "Mark as unknown/corrupt"/etc option for unknown ROM images.
...
In short: I think this format sucks.
Wouldn't this be as hard to adapt to as how America tried to switch to the metric system?
- I dunno. Anyway, the standard idea would be the PRG+CHR identified by their checksums through a database. No headers or compiled ZIP files.
Fx3 wrote:
- I dunno. Anyway, the standard idea would be the PRG+CHR identified by their checksums through a database. No headers or compiled ZIP files.
A database is useless for homebrew, translations, and hacks where the checksum will be changed. There must be a way for the end user to set the hardware instantly. No reason a prg/chr checksum database couldn't be used to FIX the bad headers of existing files tho.
Why not at the end of the *.NES file, add some bytes where the board name could be placed? Wouldn't that solve the issue with iNES? Older emulators wouldn't read the extra data, and newer ones could check for its existance and act on it if needed. I think that's a simple and effective solution, and would be very easy to use by homebrew developers and hackers. Just add something like 32 bytes at the end to put a standardized board name.
bunnyboy wrote:
Fx3 wrote:
- I dunno. Anyway, the standard idea would be the PRG+CHR identified by their checksums through a database. No headers or compiled ZIP files.
A database is useless for homebrew, translations, and hacks where the checksum will be changed. There must be a way for the end user to set the hardware instantly. No reason a prg/chr checksum database couldn't be used to FIX the bad headers of existing files tho.
Not to mention, this requires keeping an up-to-date-all-the-time download a database, so you can't just get a test ROM from somebody who's developing something and run it, first, especially if it's not catalogued yet.
It really seems like the best idea is to just convert iNES ROMs to iNES2 en masse and begin building an
external, optional database off of that.
It can even be set up like the GoodSets, with a central site to report new ROMs/broken headers/broken ROMs that could be auto-repaired with the next release.
- I don't think so. Thousands of ROMs wouldn't be changed in a short period of time, a few ones would due to the rom hacking scene, and another ones as homebrew stuff. I can't see a problem updating the database, since every emulator I know does that (MAME related). Though, it does NOT make the iNES format be gone.
The focus shouldn't be eradicating the iNES format. Rather, fixing the problems that it has introduced.
I am reminded of Geiger's SNES9x debugger, in which he requires the SMC ROM header be stripped. If the user choses "No" the emulator closes the ROM. In the end, you get two choices: modify the ROM, or don't use it. This is a bad solution to what the author thought was a real problem. Instead, the header could have been ignored internally; transparent to the user, and everyone's happy.
In this case, Disch already pointed out that an emulator supporting this XML+BIN format will be doing plenty of internal restructuring, no matter what format the actual ROM takes.
Any way, I'm starting to get off topic, but my point is don't force your magical fix-all solution onto anyone; MAKE - IT - OPTIONAL. You can have your databases and your XML+BIN ROMs, but don't bother making your emulators or tools insist that I "update" my stuff for "new features".
I think the only way this would every really catch on is if there was an "export to XML from iNES" or "export to iNES from XML" application.
The other problem is that the user DOESN'T care how the rom is stored, he just wants to play. The only people who care are developers, and unless we can make the transition seemless (which we can't because all the rom sites aren't going to just randomly change to a new format because we said so--hell we can't even decide on which format is the best), then this isn't ever going to happen.
Again, I'll remember everyone that the board name tell us nothing :
- A UNROM game can have either a 74HC32 or a 74HC08 (very uncommon), either making the high or low PRG area switched, those are different mappers, but both are UNROM. The same apply for japanese CNROM games with 8KB CHRROM
- A game can use a TLROM or MC-ACC board and still have exactly the same results, the differences between both boards doesn't exist and nobody will ever care to call them different.
- Most japanese non-Ninendo made boards have no names, or really crap names.
Finally, XML shucks and stinks and is completley overrated (exept when it comes to do HTML and still...). Why waste about 20 bytes of thext for those <stuff></stuff> when the same thing could be exprimed with 2 bits ?
Then why not use some of the unused bytes that "DiskDude!" was so kind to use?
MottZilla wrote:
Then why not use some of the unused bytes that "DiskDude!" was so kind to use?
Some of the proposed revisions to iNES already use those bytes, and the last four or so ("ude!") are reserved for detecting these nonconforming headers.
Hmm, nice to see someone (Bootgod, Martin) step up. The way I see it, this new format is meant for preservation only (iNES2 can still exist for everything else: gaming, development, romhacking). The emulator, Nestopia in this case, is used as a tool to see if the data/info extracted from the board is accurate.
"preservationists" should be aware though that this is not the ultimate format, it's done in steps. 10 years ago we thought iNES was enough, 20 years from now, the information we have currently will probably be seen as lacking.
bunnyboy wrote:
A database is useless for homebrew, translations, and hacks where the checksum will be changed. There must be a way for the end user to set the hardware instantly. No reason a prg/chr checksum database couldn't be used to FIX the bad headers of existing files tho.
Amen! I've been waiting forever for SOMEONE to write a database-driven ROM header fixer. The Mac version of Nestopia doesn't support soft IPS patching, and I HATE it when I apply a translation/hack to a ROM and watch it blow up because it has a bad header and Nestopia can no longer resolve it. A couple of years ago I wrote a small, hackish program that output to a text file the NstDatabase contents for a given .nes file, so that I could fix the header with a hex editor based on Nestopia's treatment of the ROM. All well and good, until Nestopia was updated and some fields in the database file were changed, causing my little program to fail (ugh!!).
The next version of Nestopia will have its database converted to the new XML format. I love this aspect of the XML proposal, because the format is more standardized and more easily used by other software, such as a ROM fixer. I don't care much for the zipfile container approach, but the XML database feature is very appealing to me. If my programming skills weren't so bad (it takes me forever to do a hackish job at something), I'd write a ROM fixing utility as soon as the new Nestopia version was out. I sincerely hope that someone - ANYONE - out there will FINALLY get a decent ROM tool developed for the NES!
[/rant]
Just FYI, my dumping client does have the ability to fix ROM images, but it will only work with images known by the DB. Although, because it's doing everything online, it's not very practical to do batch operations.
I am planning on making a standalone utility that can load up the XML either from Nestopia or one generated by the cart database in order to easily and efficiently repair and convert between formats.
I'm not sure where this talk of replacing iNES came from, I can't speak for Marty, but I don't know that this format is aimed at replacing anything. My client and upcoming utility will be able to convert images into whatever format you want, your not going to be forced to convert to the XML based one.
Just a few comments on some points brought up:
Writing an XML parser is a pain. It certainly is, but luckily there are dozens of free libs out there that you can drop right in. I found a nice lightweight one, incorporated into a single class that allows to to easily read,create,modify,etc XML files.
I also thought INES2 was (and still is) a great idea. If anyone is ever up to completing and documenting it (mainly the sub mapper stuff), I'll be happy to add support for that as well. It's not as flexible as the XML format, but simply the fact that it is backwards compatible makes it a huge advantage.
Viewing ROMs in a hex editor is much easier to follow with this XML format since the ROMs are stored separately and individually, so the offsets don't get messed up.
As far as IPS patching goes, one would think soft-patching would usually be the preferred way to go about it and Nestopia will still be able to use patches in this manner. However, there certainly are cases where you'd want to do hard patching as well. I could add this ability to the ROM utility.
The flexibility of the XML format is quite an advantage and you can really define a lot of detail about the hardware if you want too, even though it's not always necessary.
It already does define some pin information in cases where it is necessary. For instance to distinguish between CNROM types (m3 & m185) and X1-005 (m80 & m207). It also defines things like sound samples for games that have a built-in chip like the D7756C. I plan on doing this for some other boards as well, especially to deal with all the Konami VRC variants.
Another thing I like about the new format is the ability to have ROM "sets", as in one zip can contain all the data for a game and any revisions or regional versions. Besides being more organized, it also has the bonus of being able to share ROMs, as quite often it is only the PRG ROM that changes.
This is also the main reason my client doesn't yet build images in this format though as my db doesn't currently have any link information between regions. A lot of this data will be able to be generated automatically based on catalog ID's and shared ROMs, but there is still plenty that will have to be done by hand.
BootGod wrote:
I'm not sure where this talk of replacing iNES came from, I can't speak for Marty, but I don't know that this format is aimed at replacing anything. My client and upcoming utility will be able to convert images into whatever format you want, your not going to be forced to convert to the XML based one.
Looks like it mostly came from X-or, and some of the people on the bannister board. I bet there would be a lot less controversy if it had just been presented as a way to add additional files to the current ines, instead of a complete replacement. Even Marty in
one post said the .nes file should be left alone for backwards compatibility. Then there are people like R Belmont that say nesdev doesn't care about hardware, when the hardware info found here is more accurate than Nestopia...
BootGod wrote:
Writing an XML parser is a pain. It certainly is, but luckily there are dozens of free libs out there that you can drop right in. I found a nice lightweight one, incorporated into a single class that allows to to easily read,create,modify,etc XML files.
Feel like writing one in 6502 for a device like the PowerPak?
Handling all the different versions of zip compression wouldn't be fun either...
BootGod wrote:
Viewing ROMs in a hex editor is much easier to follow with this XML format since the ROMs are stored separately and individually, so the offsets don't get messed up.
Viewing is easier (assuming you have already replaced ines, which may or may not be the goal) but editing is certainly not. Once you make a change, your CRC/MD5/SHA/etc based database is broken. Of course that may be the idea, to make sure edited roms aren't treated as the real ones. What would a checksum do if its not for determining hardware?
BootGod wrote:
Another thing I like about the new format is the ability to have ROM "sets", as in one zip can contain all the data for a game and any revisions or regional versions. Besides being more organized, it also has the bonus of being able to share ROMs, as quite often it is only the PRG ROM that changes.
Other than the sharing of duplicate data, making sets can already be done. Would the duplicates be removed by zip anyways? Its just a normal zip file so you can toss in whatever you want, including manuals and boxes for preservation. ines in no way restricts what other files you bundle with it in a zip. This new format should in no way restrict the use of ines files instead of separate prg/chr files, since that would just be a simple entry in the xml file. With the tiny size of nes roms you could even have the nes and prg/chr files in the zip. Artwork will be FAR larger than many copies of the rom anyways.
bunnyboy wrote:
BootGod wrote:
Writing an XML parser is a pain. It certainly is, but luckily there are dozens of free libs out there that you can drop right in. I found a nice lightweight one, incorporated into a single class that allows to to easily read,create,modify,etc XML files.
Feel like writing one in 6502 for a device like the PowerPak?
Handling all the different versions of zip compression wouldn't be fun either...
Yeah you've got a point there, the PowerPak is definitely one situation where the XML format is not suitable at all. One could always use the conversion utility to write INES/INES2 files right to the memory card though.
bunnyoy wrote:
BootGod wrote:
Viewing ROMs in a hex editor is much easier to follow with this XML format since the ROMs are stored separately and individually, so the offsets don't get messed up.
Viewing is easier (assuming you have already replaced ines, which may or may not be the goal) but editing is certainly not. Once you make a change, your CRC/MD5/SHA/etc based database is broken. Of course that may be the idea, to make sure edited roms aren't treated as the real ones. What would a checksum do if its not for determining hardware?
In the ROM format aspect, the checksum doesn't really matter at all. Files inside the zip are defined in the XML by the ROM filenames, not by the checksum. So any patched ROMs will still load just fine in the format.
bunnyboy wrote:
BootGod wrote:
Another thing I like about the new format is the ability to have ROM "sets", as in one zip can contain all the data for a game and any revisions or regional versions. Besides being more organized, it also has the bonus of being able to share ROMs, as quite often it is only the PRG ROM that changes.
Other than the sharing of duplicate data, making sets can already be done. Would the duplicates be removed by zip anyways? Its just a normal zip file so you can toss in whatever you want, including manuals and boxes for preservation. ines in no way restricts what other files you bundle with it in a zip. This new format should in no way restrict the use of ines files instead of separate prg/chr files, since that would just be a simple entry in the xml file. With the tiny size of nes roms you could even have the nes and prg/chr files in the zip. Artwork will be FAR larger than many copies of the rom anyways.
The zip format doesn't do solid compression (each file is compressed individually) so no, zipping would not have the same effect as removing the duplicate files.
bunnyboy wrote:
BootGod wrote:
Writing an XML parser is a pain. It certainly is, but luckily there are dozens of free libs out there that you can drop right in.
Feel like writing one in 6502 for a device like the PowerPak?
Remember the Contiki web browser? It handles HTML, which is related to XML. So I know the NES is powerful enough to handle at least a useful subset of XML, even if not a full-blown DOM or SAX implementation. But I don't know how much RAM needs to be made writable for it to work.
Quote:
Handling all the different versions of zip compression wouldn't be fun either...
Do popular PKZIP-compatible archivers ever use any of the PKZIP 1.x codecs? But the typical 32 KiB ring buffer size of Deflate might be the biggest pain on a machine with a small address space.
Quote:
Would the duplicates be removed by zip anyways?
No. The PKZIP format, unlike some other archive formats, does not use solid archiving. A non-solid archive makes it easier for the client to decompress individual files, as the files don't depend on files earlier in the archive, but it doesn't exploit file-to-file redundancies. The tar.gz format uses solid archiving, but it still has Deflate's 32 KiB window, which is smaller than everything but the earliest NROM games. The only widely-used Free archive format I know of that has solid archiving and a window size on the same order of magnitude as an NES ROM is .7z used by 7-Zip.
Zip Store, then Zip maximum compression to get solid archiving with zip.
How is interpreting XML any different than a basic interpreter, like the one from the Apple ][?
Dwedit wrote:
Zip Store, then Zip maximum compression to get solid archiving with zip.
That would have the same drawbacks as tar.gz: the 32 KiB window limit.
Quote:
How is interpreting XML any different than a basic interpreter, like the one from the Apple ][?
Differences include the following:
- Applesoft BASIC used a 1-byte character encoding based on ASCII, while XML uses UTF-8 or UTF-16 that can address over a million unique code points.
- Applesoft BASIC's parser doesn't need to load and interpret custom DTDs that define custom entities (&stuff;).
- A program in Applesoft BASIC is a linked list of lines, each of which is a nul-terminated string of tokens and characters. An XML document can be an arbitrary tree. Processing trees directly in assembly language on an 8-bit CPU can get hairy.
We would need to define a custom subset of XML that includes what we need and excludes what we don't. At that point, we might as well just use INI format.
String comparisons in UTF-8 are identical to string comparisons in 8-bit ASCII. No piece of a high character can ever appear in ASCII, and there is no way to find one extended character inside another. For everything except counting characters, you can ignore the fact that it's not ASCII.
Dwedit wrote:
String comparisons in UTF-8 are identical to string comparisons in 8-bit ASCII.
True, but XML files can be in UTF-16 too. And is "Mario's Time Machine" the same as "Mario's Time Machine" and "Mario's Time Machine" and "Mario's Time Machine"?
Quote:
For everything except counting characters, you can ignore the fact that it's not ASCII.
Everything except counting and printing. Do we ignore the mainland Europe and French Canadian markets by printing any character in the title that's above U+007F as a '?' character?