i'm not sure what else to call it. it's what games do to componesate when more than 8 sprites on a scanline. Since looping through each of the 40 tiles or whatever would realistically be on sceen at any given time is going to be really expensive, are there any tricks to getting the job done with less time spent on it? The algorithm in my head is nasty and slow and most likely won't work very well in practice.
I've heard a few things. One way is when your game takes game objects and puts them into your sprite DMA page, you could reserve the order. Another thing I've heard is if you aren't using sprite 0 hit flag for anything, you can set the OBJ RAM address before doing sprite DMA and it would start filling OBJ ram from the point you set. So you could increase/modify the starting point to try to cycle sprites.
I never messed around with it enough to figure out a particularly good method.
Bregalad mentioned a good idea, though you'll have to put in extra checks to avoid overwriting sprite #0 if you are using it for timing purposes.
Assuming you have a sprite drawing routine that completely updates the OAM page every frame (and you're not just doing something like INC $303 to move a sprite to the right), you do this:
(Say $300 is your OAM page)
Put first sprite at $300,x , x = 0
Put second sprite at $300,x , x= 12
Put third sprite at $300,x , x = 24
Increasing x by 12 each time so you eventually are at:
Put twenty-first sprite at $300,x , x = 252
Put twenty-second sprite at $300,x , x = 8
Because when you add 12 to x, since it's an 8-bit register, it wraps around to 8. That shuffles the priority a little bit. It does it even more where if you don't use all 64 sprites, you begin the next frame where you left off. Say you only drew 22 sprites. X would be at 20. So you'd draw the first sprite at $300,x where x = 20, then you'd draw at x = 32, etc. I use this method and it gives me lots of cycling. Also, every frame I switch the order in which I draw objects. So if there are 4 objects on screen, say the player, and three enemies, I'll draw them in one frame like this: first the player, then enemy 1, enemy 2, then enemy 3. The next frame I'll draw first enemy 3, then enemy 2, enemy 1, and then the player. This creates more cycling.
However, sometimes there'll be a certain amount of sprites on screen where it ends up creating no cycling at all. I've seen this happen, and it's not good. To prevent this, you can come up with a psuedo-random number generator between numbers 0 and 63. Whatever number this turns up, multiply by 4 and start drawing sprites from there.
I get you are hardcoding OAM bytes to specific on-screen sprites? That's never good. If you treated the OAM page more dynamically, you probably wouldn't have any problems, as it would be just a matter of copying/rendering your sprites there in a different order.
If priorities between sprites doesn't matter, then this is easy to do.
I guess the good idea I had Celius mentionned was that after initialising the OAM index to 0 (and optionally put a sprite zero for it in here), instead of increasing the index by 4 as you'd normally do, you increase it by any number that is multiple of 4 but not multiple of 8. That way you'll still acess to all sprites but in a "random" order (altough it's not really random) and eventually when you reach sprite 0 again. When this happens, you know that the OAM has been filled and that it's ready for the next DMA (when you draw all sprites you'll need, you'd want to fill all Y indexes with $f0 until the index is 0 again, and that both with and without cycling).
Initialising the increment amount to 4 and then add 8 to it each frame gives awesome pseudo-random sprite cycling. If you want to disable cycling for a particular reason (for example sprite pritorities matters in a cutscene), you'd just want to reset the inc amount to 4 and never touch it while the cycling is disabled.
If the priority between sprites matters and that you want cycing, you'll have to ressort to more complex methods. You'd want to do the normal increase by 4 index, but draw the objects in an order you choose. For the game I'm coding, priorities between sprites matters a lot as it has top-down perspective, so a sprite that is below another have to have higher priority. For that reason I sort the objects in function of their Y index before drawing them, but the metatiles themselves drawns in an alternate order every so frame. This doesn't give me awesome cycling, but I can't come with anything much better.