Don't know if you're still having this problem... but...
'Color 0' is transparent. That is, color 0 of the color index determined by the pattern tables -- before attribute bits are added. Therefore, $3Fx0, $3Fx4, $3Fx8, and $3FxC are never drawn as part of the BG or sprites, because they all represent 'color 0' of their respective attribute set.
Therefore, transparency can be determined by that "pixel_color & 3" thing you're doing. If pixel_color & 3 yields a nonzero value, the pixel is opaque, otherwise it's transparent.
In my emu, I have prioritizing logic done in a manner similar to the following:
Code:
a = current_bg_pixel;
b = current_sprite_pixel;
if(!(a & 3)) a = 0;
if(!(b & 3)) b = 0;
if(b && sprite_has_bg_prio)
{
if(a) // BG and Spr pixel both opaque, but BG has prio
output = a; // so output it
else // only sprite pixel opaque, output it
output = b;
}
else
{
if(b) // sprite pixel opaque and has priority
output = b; // output it
else if(a) // sprite pixel transparent.. but BG opaque...
output = a; // output it
else
output = 0; // both transparent, output $3F00
}
That's the jist of it -- I added a few things to make it easier to understand.
If you want the actual, unaltered code:
Code:
a = nBGRender[nScanCyc + nXScroll];
b = nSpRender[nScanCyc];
if(nScanCyc < nBGClip) a = 0;
if(nScanCyc < nSpClip) b = 0;
if((b & 0x40) && (a) && (nScanCyc != 255)) // sprite-0 hit
n2002Status |= 0x40;
if(b & 0x80) //low sprite prio
{
if(!a) a = b & 0x1F;
}
else if(b) //high sprite prio
a = b & 0x1F;
PIXEL_OUT(a);