Most of the time I tend to pass all or nearly all of the test ROMs that are available on the wiki. But I feel a little bad using them to be honest. Its like having somebody else look at your code and tell you what is wrong with it instead of finding out for yourself. I know that it rigorously tests all aspects possible that most games wouldn't and that it is technically legitimate to use them as they would work on a real NES and they don't specifically tell you how to be a programmer. But I think that it makes for a lazy programmer.
How do you guys feel?
Nope. Use all the tools available to get what you need done. It's silly not to, like not using available debugging features. All it does it take more time to get the same thing done, for no reason other than to say, "I did it this way!"
Test ROMs are nothing more than unit tests for an emulator.
They're cheating if you're programming an accurate emulator as some kind of proof of being a programmer cowboy. Of course using a compiler that reports errors in your code rather than generating a crashing executable is also cheating, as is using a high-level language (asm is a high level language to a cowboy). Real cowboy programmers program in machine language, and never need to test their code since it works on the first try. At this point it's probably clear that only
one person fits the bill.
On the other hand, tool-wielding homo sapiens use every tool they can lay their figurative hands on, including compilers with extended error checking, debuggers, automated unit tests, which includes test ROMs. All of these tools extend the power of the user, but only if used intelligently, which is where these programmers get satisfaction. They don't see the wisdom in intentionally obscuring one's perception when debugging.
...Chuck Norris can run a XBOX game on a Atari 2600 emulator.
For instance, I've just fixed a CHR bank switching bug in my MMC1 emulation. Instead of running to a test ROM to tell me what was wrong I did some old fashioned work;
1. Logged CHR ROM switching writes.
2. Listed all games with corrupt GFX.
3. Compared them all and discovered they all used the same method of bankswitching.
4. Checked/changed the code, tested new theories etc.
5. Fixed the problem!
Now that felt so good. I wouldn't have experienced that if I had used a test ROM to tell me the error. They should only be used as a very last resort or to test things that no game does.
WedNESday wrote:
Instead of running to a test ROM to tell me what was wrong...
I wouldn't have experienced that if I had used a test ROM to tell me the error.
Replace using a test ROM with asking for help on these forums. Is that cheating?
WedNESday wrote:
For instance, I've just fixed a CHR bank switching bug in my MMC1 emulation. Instead of running to a test ROM to tell me what was wrong I did some old fashioned work;
1. Logged CHR ROM switching writes.
2. Listed all games with corrupt GFX.
3. Compared them all and discovered they all used the same method of bankswitching.
4. Checked/changed the code, tested new theories etc.
5. Fixed the problem!
Now that felt so good. I wouldn't have experienced that if I had used a test ROM to tell me the error. They should only be used as a very last resort or to test things that no game does.
I don't know of any test ROM that tests CHR ROM switching(?)*, so in this case you're doing what you can with the tools you have available, anyway. No cheating; nothing novel, either. FWIW I had the very same experience and followed the very same steps you outlined when working on MMC1. MMC3, and I'm sure many other mappers. Zelda 1 worked but Zelda 2 was graphically messed up. Yada yada. In that case, I used my built-in mapper debugger view to look at what the registers were set up for, and what CHR address was currently mapped at what 1KB CHR bank, then looked for paths through the code that differed based on those settings, then found the issue, then fixed it, then celebrated.
I may be remembering the two games wrong...but I know I had issues with MMC1 games that CHR-switched either 4KB or 8KB, or switched between the two switching methods.
*If there were such a test ROM I'd imagine it'd put pictures up on the screen and have the user hit "A" for "looks right" or "B" for "looks like shit". Sort of like an eye exam..."which is better? number 1, or number 2?" Eventually with enough cases and enough user response the test ROM could point out that you likely have a bug in your 4KB CHR bank switching for mapper X. But that's just it...CHR bank switching is different for *gobs* of mappers [are the bits in the CHR register shifted to form the address or not? are there bits in the CHR register that are not part of the CHR bank mapping? does CHR mapping rely on the state of more than one register?], so testing all of the possible "problems" you could have with it is absurd to try to automate.
cpow, when I said CHR test ROM I was just talking generally.
James wrote:
WedNESday wrote:
Instead of running to a test ROM to tell me what was wrong...
I wouldn't have experienced that if I had used a test ROM to tell me the error.
Replace using a test ROM with asking for help on these forums. Is that cheating?
It is cheating if you don't try hard enough and expect others to do work for you by checking your code. Or, the documentation is inadequate. I only ever ask for help as a last resort.
cpow wrote:
I don't know of any test ROM that tests CHR ROM switching(?)*
*If there were such a test ROM I'd imagine it'd put pictures up on the screen and have the user hit "A" for "looks right" or "B" for "looks like shit". Sort of like an eye exam..."which is better? number 1, or number 2?"
It wouldn't have to. It could read CHR ROM back through $2007 (like
my multi-mapper test ROM does) or through sprite 0.
Quote:
But that's just it...CHR bank switching is different for *gobs* of mappers [are the bits in the CHR register shifted to form the address or not? are there bits in the CHR register that are not part of the CHR bank mapping? does CHR mapping rely on the state of more than one register?]
My test ROM uses mapper-specific drivers loaded into RAM. Each mapper's test driver has three routines: load an 8K chunk of CHR ROM/RAM (for the coarse CHR readback test), load an 8K chunk of PRG RAM (for the PRG RAM test), and then do the PRG ROM, IRQ, and fine CHR tests that test several combinations of registers that could affect CHR mapping.
Quote:
FWIW I had the very same experience and followed the very same steps you outlined when working on MMC1. MMC3, and I'm sure many other mappers.
My test ROM is working on MMC1, MMC3, and about ten other mappers.
Cheating at what? Is this some sort of competition?
rainwarrior wrote:
Cheating at what? Is this some sort of competition?
Cheating yourself. For me at least, programming emulators is like a big puzzle and when you use a test ROM to solve a problem without trying hard enough it's like asking someone for help on the puzzle.
The problem, of course, is that "not cheating" is likely to net you Yet Another Inaccurate Emulator instead of something that other people will actually want to use.
WedNESday wrote:
Cheating yourself. For me at least, programming emulators is like a big puzzle and when you use a test ROM to solve a problem without trying hard enough it's like asking someone for help on the puzzle.
Interesting. So you want to make your program so clear and simple (or have such a good picture in your mind that it can be large and complex) that you can see any problems and not need any other tools to help find them. This is in contrast to the approach of programming in ways that plan on having tool assistance when debugging. "No way I can look at my code and be sure it emulates this properly, so I'll just use this tool to test that aspect."
It's an interesting trade-off, and gets to the core of what one wants out of the experience. A puzzle as you say where the point is to use only one's mind and minimal tools (text editor with search?) to diagnose problems. Or to see how much one can extend one's abilities with the best tools available. The latter tends to use tools to make difficult problems simple, by giving one a clear view of something that was obscured using one's unaided perception abilities, e.g. a debugger showing variable values at run-time, or a test ROM setting up certain conditions then seeing what happens if a particular thing is done. The fun as you say of a puzzle is that it's challenging only if you're unaided. Getting good at using programming tools to make challenging problems trivial tends to eliminate the fun from solving challenging problems, at least the ones eliminated. Personally I find no joy in working hard at a problem that is only made hard due to not being able to see everything and thus bumping around aimlessly in the dark, especially when I know that there is a flashlight available.
I've had experience on both ends. There were no test ROMs on the SNES, so I had to figure out the behaviors through my own hardware tests. Whereas pretty much 100% of my NES emulator is from the Wiki and an official opcode tester.
What I've found was that I have a much deeper understanding of how the SNES works. My NES emulator is full of weird edge cases that I don't even remotely understand, but they're well known. Things like non-linear APU mixing. It's voodoo to me.
The SNES work is more frustrating, but upon solving it, is much more enjoyable and rewarding.
Overall, I like the idea of test ROMs, but I don't like the way they are used. Case in point:
http://tasvideos.org/EmulatorResources/ ... Tests.htmlAccording to that chart, the emulator I spent one week developing ranks higher than FCEUX and VirtuaNES. I don't think anyone here believes that. If I were to use test ROMs, I'm sure I could push it up to the top-end as well.
But, as important as test ROMs are, and I think we should all pass them wherever possible, people need to keep in mind that these test ROMs are usually extremely pedantic, often testing things that are virtually never used. And test ROMs are
far from comprehensive. The games are the reason we are here, and it's that which should be the center of our focus.
For instance, what good is an emulator that passes 100% of test ROMs, but doesn't support anything but NROM? Would you trade 3% on that accuracy scorecard for dozens of supported peripherals, FDS support, etc?
Some sort of ranking system for the importance of passing each test ROM would probably go a long way.
Good point byuu.
Of course, if I had real hardware to test I could just pop in a flash cart and find out myself without the need to come here for help. Remember, the thrill of the chase is what counts not the catch itself. The fun is in writing the emulator not having written an emulator. If it were too easy or too quick or without the occasional problem, then there would be no fun and we would just use other people's emulators.
Test roms isn't some magical thing that will make all bugs in your code disappear. They do help, but not that much as it seems to be imposed in this thread. Considering how much other kinds of help is involved into any software development anyway (experience of other people and their creations - compilers, text editors, etc), I don't think it is even worth of discussion. If you want to have extra fun catching bugs, just don't use test ROMs; if you want to get result a little bit sooner, use them and all other possible debug tools. Execution logs of another emulator is actually WAY more valuable for this purpose.
Correct. Test ROMs don't make bugs magically disappear, rather they point out where your code is going wrong and that is what I don't like.
Of course having the ability to port a homebrew test ROMs to actual hardware negates everything and IMHO is the best option.
Actually, test ROMs don't really point at exact place of a problem. They give a direction for search, but they can't detect a wide range of potential problems, and can give false reports, either giving you a false direction, or false sense of correctness (report OK while emulation is in fact wrong). There were threads on this forum with such cases.
byuu wrote:
But, as important as test ROMs are, and I think we should all pass them wherever possible, people need to keep in mind that these test ROMs are usually extremely pedantic, often testing things that are virtually never used. And test ROMs are far from comprehensive. The games are the reason we are here, and it's that which should be the center of our focus.
+1
byuu wrote:
For instance, what good is an emulator that passes 100% of test ROMs, but doesn't support anything but NROM? Would you trade 3% on that accuracy scorecard for dozens of supported peripherals, FDS support, etc?
I'd deprioritize FDS support because I imagine that even fewer people own the means to dump FDS disks than own a Kazzo or CopyNES to dump carts. Support for a modern FDS-like board that people can actually get from infiniteneslives.com might be a better way forward, at least for people who use emulators to run homebrew.
byuu wrote:
Some sort of ranking system for the importance of passing each test ROM would probably go a long way.
You're invited to do so on the wiki. For example, nestest is pretty important because CPU bugs will kill games very dead very quickly.
WedNESday wrote:
The fun is in writing the emulator not having written an emulator.
i see an emulator as a tool for developers to test their games and to promote their games. If you can improve the tools, that frees up more time for making other tools and for making games themselves.
Shiru wrote:
If you want to have extra fun catching bugs, just don't use test ROMs; if you want to get result a little bit sooner, use them and all other possible debug tools. Execution logs of another emulator is actually WAY more valuable for this purpose.
Better yet, execution logs of another emulator running a test ROM, like that well-known Nintendulator log of nestest.
> I'd deprioritize FDS support because I imagine that even fewer people own the means to dump FDS disks than own a Kazzo or CopyNES to dump carts.
Well, we all know that 99.99% of users get all their game images illegally. Not even exaggerating the decimal places.
More likely you could deprioritize it because most emulation users tend to not speak Japanese (around one in sixty people in the world do, and emulation isn't quite as huge in Japan as it is elsewhere.)
> You're invited to do so on the wiki.
Thanks, but I'm not really qualified. If you add a SNESdev Wiki, I could do so with any test ROMs produced for it.
WedNESday wrote:
Of course, if I had real hardware to test I could just pop in a flash cart and find out myself without the need to come here for help.
My frustration with incorrect APU information is what pushed me to sacrifice my Zelda cartridge as a devcart.
I'd love for everyone to have low-cost devcarts to test things themselves. Up to a point it's fun.
WedNESday wrote:
But I think that it makes for a lazy programmer.
No, it makes for a resourceful programmer. It makes for a lazy researcher, but unless you have the skills and tools to do a detailed examination of the NES or cartridge hardware - what else can you do?
byuu wrote:
If you add a SNESdev Wiki, I could do so with any test ROMs produced for it.
Someone else added one.
miker00lz wrote:
WedNESday wrote:
But I think that it makes for a lazy programmer.
No, it makes for a resourceful programmer. It makes for a lazy researcher, but unless you have the skills and tools to do a detailed examination of the NES or cartridge hardware - what else can you do?
Resourceful yes. Lazy researcher, yes. If you don't have access to hardware homebrewing then you ask in the forums. But only after studying the documentation.
I never said that getting help from one source or the other wasn't allowed, just to be used as a last resort after you've studied hard and tried your best.
It is interesting the extreme spectrums that exist.
The more you put into doing your own research and running your own tests, the more the emulator becomes an expression of something you've created, rather than copied.
You start from the beginning, with a pure black box. At best, you can determine the CPU type and find an official programming guide for it. But you have to work out the memory mapping, every register, etc from scratch.
Then you have increasingly thorough and useful documentation.
Then you have a huge swath of test ROMs telling you what to fix and how to fix it.
Then you have entire processor cores, where you just write the glue code to bind them all together (fairly common in the Genesis scene.)
Then you basically have entire emulators as libraries, where you write a frontend for it.
Each level probably thinks of themselves as "emulator authors", and hell maybe they are, even the frontend people. But they're definitely distinct tiers. And depending on which tier you are on, you probably ask why the lower tiers even bother.
It seems very much to mirror the progressions of programmers in general: the EE logic gate types/Verilog programmers -> the assembly programmers -> the C programmers -> the C++ programmers -> the Ruby/Python programmers.
I can't fathom a world in which a test rom would be considered cheating. *Everything* is a test rom, it's just a question of how tightly controlled the situations are. They're a way of determining, fairly directly, the answer to the question "What did the Real Thing do in this particular case". It's the difference between theoretical physics and applied. One of those considers the problem, writes it down, thinks really hard, and writes down the answer. The other smashes some protons together to see what comes out.
Both are needed, and both are valid approaches. Personally, as a developer, I much prefer debugging cases with clean, reproducible tests. They leave a lot more energy left afterwards to develop more cool stuff. It does feel awesome when you finally track down and squish some evil, obscure bug through Guru level thought experiments, elbow-deep postmortem examination, and a brutal bit of luck on timing a reset or whatnot, but the only thing one feels like doing afterwards is celebrating, and not looking at code for a while.
As for the pile of test roms we have available, they are deceptive. My FPGA NES passed every sprite 0 test there was, cleared almost all of the timing and CPU tests (precise timing of the even/odd dropped dot still elude it), and all the APU tests. It still failed cart.nes and smb1 immediately. Things they didn't catch:
Sprite evaluation continuing with rendering disabled, clobbering the OAM address
No sprites showing up at all after hitting 8 on a scanline
Backwards, page-crossing BCCs going forwards instead
Envelope loop flag being hooked up to the half frame pulse instead of the reg bit
No testsuite is completely bugtight.
re: byuu's last post
Some application developers use frameworks, some write straight Win32 or Xlib code. Some use C++, some use C#. All are still application developers.
In byuu's progression, I've always seen myself as someone who finds things that are almost complete, but lacking those final little things to make them basically complete, those little details that make a big difference, and tries to fill them in.
As for the test ROMs, I've generally tried to target things that are difficult to check through other means, rather than cover all the basic things thoroughly. I see it as a matter of effort versus value. It's easy enough to use SMB as a test to get all those things working (and have fun watching the game go from garbage to working).
Indeed, it also helped reel in the usual delusions of grandeur. Now to finish stuffing some test code into the bios to diagnose where the mapper units are exploding...
I feel that test ROMs are a legitimate tool for getting emulators working. For someone like me who knew pretty much NOTHING about the 6502, they were a VALUABLE debugging tool in figuring out where my code was going wrong. If nestest didn't exist I'd probably STILL be debugging my emulator and hoping it was correct.