So earlier today I was finally bothered by SMB3 not working on my emulator enough to do something about it. So I took a fresh look and quickly figured out there was an IRQ problem. With that out of the way there's a new problem.
Certain sprites that should be hidden behind the background, aren't. For example the Mushrooms when they float up out of blocks are being drawn over and not under the block they are coming out of. The monsters that come out of the green pipes also are being drawn over the pipe and not under. But Mario when you get on a white block and hold down, DOES go behind the background just fine. Also, other games have no problem with this, for example Legend of Zelda has Link entering caves just fine where he's entering the cave which is a all black tile and switches priority and then sinks downward behind the ground tile.
Does anyone have any idea what could be wrong with SMB3? Why would some sprites do their BG priority wrong? If anyone has any ideas that would be helpful. Things it could be, things to check for, etc.
If I understand your description of the problem, it happens when you have 1. a lower-index, back-priority sprite, 2. a higher-index, front-priority sprite, and 3. a background pixel, all at the same screen location.
Each sprite has two values that affect priority: the index of the sprite within OAM (0 to 63), and the priority bit (attribute 2 bit 5, set to 0 for front or 1 for back). On the NES PPU, putting a back-priority sprite in front of a front-priority sprite lets the background show through and cover up the front-priority sprite. Super Mario Bros. 3 uses this for powerups sprouting from blocks, putting a blank block "behind" the block at a low index and then putting the powerup "behind" that at a high index.
I'm guessing you think the NES PPU handles priority like this:
- Front priority sprites in front
- The background plane in the middle
- Back priority sprites in back
What really happens in the NES PPU is conceptually more like this:
- Sprites get drawn front (lower index) to back (higher index), taking the first opaque pixel that matches. Priority does not affect ordering in this buffer but is saved with each pixel.
- Background gets drawn to a separate buffer.
- For each pixel in the background buffer, the corresponding sprite pixel replaces it only if the sprite pixel is opaque and front priority.
Thanks tepples. That is exactly the response I needed. What you described for how I currently draw the screen is correct.
So if I understand this right there is a sprite behind the block in background, that is drawn over the sprite above it, but because the sprite drawn over it is behind the background, the background is what is drawn. Hopefully that sounds right.
So to fix this will require reworking how I draw sprites. Atleast now I know the reason for this. Before your post I wasn't sure what it could be at all.
You're welcome.
Wikified.
I'd like to know of more games that use this trick for masking sprites. I know that the Sega Master System, for example, has a priority bit for each background tile, so that they can be either in front or behind all sprites. The NES doesn't have that, but similar effects can be achieved by using that priority trick. Most of the time it's just too much trouble though, so you don't see layering effects on the NES that often.
Castlevania uses the sprite priority trick for bonus items rising out of the ground (like at the beginning when you jump past the castle and get a money bag)
I see... The thing is that all examples so far are games that use it in a very simple way, with rectangular masks. I'm not aware of a single game that uses complex shapes as masks. That would allow for some nice layering effects.
I can understand why nobody did it... first, sprites are a very expensive resource on the NES, since only 8 can be displayed in a scanline. Some sort of advanced detection could be used to only use masks at the edges of complex shapes, and skipping the rendering of individual sprites that would be completely hidden by the foreground object. That would waste a lot of CPU time though. And there is also the fact that the masks would need their own patterns, stealing tiles that would otherwise be used on actual graphics.
Not likely intentional, but in SMB3 when Mario is behind the walk block background and an enemy walks infront of him you see a complex masking/outline of Mario using the background that is neat.
I've rewritten how sprites are handled in my emulator and SMB3 seems to work fine now. It wasn't particularly difficult to do either.
Time Lord uses triangle masks in the second area.
I never actually confirmed this... but I always figured that Blaster Master does this trick to hide the lower half of your body when you walk behind a wall when in the top-down view areas.