koitsu wrote:
I'd absolutely love a response from rainwarrior on this one
Automated testing and human / QA testing are not mutually exclusive things. You can do both. The questions are how much, and how, and those are very subjective questions that don't have easy answers.
Ron Gilbert wrote a pretty good article about it a little while ago that aligns pretty well with how I feel about it:
Unit Testing GamesBasically, I think some amount of automated testing is worth doing, but games are a fairly unique area of software development where most of the meaningful testing work cannot be automated, so there is not nearly as much value in unit tests etc. in games as it is for most other types of software. (I wrote an
an article about testing Lizard, but I didn't say a whole lot about automation for that reason.)
The other thing is that automated tests can help a lot more when more people are working on the code. You can decide how stable the part of the code you're responsible for is, and document it, etc. but most things aren't isolated, and other people will be changing things you don't expect. Unit tests are often a good way to set up a canary for yourself to tell you if something that was out of your hands changed unexpectedly. When it's just me working on the code, though... I usually know what I'm changing. I mean, sometimes I don't know, sometimes I forget stuff, but it's a world apart from working on a big team. Unit tests simply have more value in collaborative programming, and this should be a factor in deciding the "how much" question.
As far as what I
did for automated testing in Lizard: not a lot. I had a lot of compile/assemble time asserts. I also had runtime asserts on various things. On the NES version I actually used BRK to implement runtime asserts. These things aren't as automated as something like unit testing, but they're still doing a significant amount of validation and testing more or less automatically while I'm already doing manual testing or other compiling. The runtime asserts were very liberal on the PC version, and very sparing on the NES version though. The PC ones get automatically removed in the release build, but the NES ones mostly stayed put. (Most problems affected both versions, so verification in one still benefited the other platform.)
Many times I recorded a TAS movie in FCEUX, and on the PC I made it automatically log input each time I played. When something tricky to execute would come up in testing, this made it much easier to reproduce and debug the problem. I could go back, set up breakpoints etc. and then just replay the input movie. This was a semi-automated testing that was incredibly useful on a few specific occasions, but not something I was doing regularly.
If I manage to make another large NES game I'll probably do some sort of automated unit test, actually. Not sure what form it will take. You can run FCEUX with a TAS and a Lua script from the command line, and have the lua script output results to a file. (Hmm, does FCEUX needs a "terminate via Lua" function?) Most likely I'd have a separate unit test mode, not running TAS on the regular game. Ideally something I could remove easily if I needed the ROM space. You might remember that arcade machines tend to have a series of tests on power on, and often have
debug test modes.
In my professional game work, every major project I was on had unit tests in some form. It was in no way a substitute for QA / manual testing, but it certainly had enough value to justify doing. You'll write 100 unit tests to catch 1 bug. Most of them are never going to fail, but you won't know which is going to fail. That's why you write them. If you can set up a framework that's easy to add tests to, the reward to added work ratio can easily pay off, here.
Actually, similar to the "rubber duck debugging" idea, often the act of writing a unit test for some new code will immediately find bugs in it. I almost want to say that this collateral benefit is even more valuable than the rare cases where a unit test fails due to later changes. Learning to write good unit tests takes experience, both in terms of knowing what is most useful to test, but also just how to write them quickly and effectively, so it also pays to get in the habit of it just to gain practice here.
As an example of what a unit test might look like: testing the inventory system for an RPG might involve generating a random list of items, adding them to the inventory, making sure they're all there and in the expected quantities, maybe another test for removing items, etc.
Unit tests tend to be only good at finding very isolated bugs. They tend to be very bad at finding interactions between systems. Like if changing a few frames of a character's swing animation makes a boss on level 4 impossible to beat, there's probably never going to be a practical automated test for that kind of effect, but human testing will find it. Remember also: to catch this 1 break you would have written 99 other useless tests. It's really very difficult to capture all the constraints needed for any complex task in a game.
Similarly there's all kinds of bugs where, e.g. maybe treasure chests and torches share code in an unexpected way, so that opening a treasure chest accidentally lights a torch elsewhere or vice versa. This isn't an intentional change, but it is the kind of bug you see very often in games. A unit test would just test the torch, and just test the treasure chest, and find no problem. If they have to be used together to verify, it tends to be out of scope for automated testing, but QA on the other hand can be fairly good at spotting this kind of thing. In general unit tests are much better at warning about deliberate functional changes than accidental ones.
So... anyhow I think it's potentially worth doing, but really depends on the scope of your project, and your level of experience, etc. It's not high enough in value that it can't be ignored, really. Lizard turned out fine (I think?) without unit testing, but I actually wish I'd implemented some. For smaller projects I probably wouldn't bother, but maybe it's good practice-- possibly the most important thing is just learning how to write unit tests quickly and effectively. If they're too big of a work drain, they don't pay off.