Here's a crazy thought: Could that emulator be used to take the color out of GBC games?
R = G = B = .3R + .6G + .1B also takes the color out of games for any platform that uses RGB color.
But that wouldn't be the same as using original GB palettes, right? (I imagine that's the idea here)
It would also be possible to do something like:
1) Examine each of the color palettes, calculate the brightness of each color
2) Based on the brightness, try to map this to an equivalent DMG shade of gray when drawing
3) Hope for the best
It's something that crossed my mind, although I'm not certain how well that would or wouldn't work. Can't say until I've tested it.
BUT, I think strat was asking something more along the lines of "can GB Enhanced replace a game's original colorized graphics with my custom grayscale graphics?" The answer is complicated. Yes, in practice one should be able to dump the game's colorized graphics, convert them to grayscale in an editor, and have GB Enhanced upload them when it plays a GBC titles. GB Enhanced doesn't care whether the uploaded images are grayscale or not; it's just pixels to the program, so any style will work (neon, pastel, metal, etc).
However, GB Enhanced can only experimentally dump and load sprites from GBC titles (and BG dumping/loading hasn't been implemented at all). The whole process of adequately hashing GBC tiles is more involving than DMG tiles, and I never got around to finishing custom graphics support for GBC games. The work is mostly complete in a separate branch, but I started working on the successor to GB Enhanced (unsurprisingly called GB Enhanced+, or GBE+). I'm aiming on making it a DMG/GBC/GBA emulator with the ability to replace graphics for all 3 handhelds. I'm knee-deep in GBA emulation at the moment, but rest assured GBE+ will improve upon GB Enhanced. It'll also have a GUI, which is somewhat essential for extensive editing. So what strat is looking to do will be possible, eventually.
I've been working with games on GBE for quite awhile now. Yesterday I found a way around the duplicate tile problem but it's pretty much a chr hack. It works well enough though. I've also edited the source to package everything into a nice little way for releases. There are some glaring issues that I can't figure out how to fix which is disappointing. LCD brightness I think it is. The tiles dump but don't load. Not sure why...
Anyways here's an unofficial build of Game Boy Enhanced from August.
https://drive.google.com/file/d/0B5aZT1MLfWJsY185RTF3TDZBZ0U/view?usp=sharingLooking forward to GBE-plus!
When you say duplicate tile issue, I assume you're talking about the issue brought up on Google Code (involving Metroid II?) I'd love to know (on a technical level) just what exactly is causing that and how to reliably reproduce it myself. I think root may be an issue with XORing the palette data onto the tile's hash. That may not be enough to gaurantee that collisions don't happen.
In the end though, I may just have to prefix the hash with the DMG palette represented in hex (which should have been my 1st choice anyway). This is similar to what's required for dumping and loading GBC sprites (which need color palette data to be prefixed to the base hash to create a unique hash). I'll look at the code you posted later and see what's up. I may just have to break compatibility in GBE+ (but it's simple enough to add an option to for backwards compatibility).
GBE+ is going well btw. Almost 90 days of constant commits to the project have lead to a lot of progress. A lot of GBA games boot up and run just fine, and I haven't even emulated all of the DMA channels, interrupts, or timers. I never expected to get this far this soon. To put things in perspective, this time last year, I was struggling with GB sound emulation, hadn't touched the GBC yet, and didn't even have the foundations worked out for HD/replacement graphics.
I would love support for SNES as well, Because I may want to enhance the graphics as an option for my SNES hacks (currently focused on SMW). Assuming 256 (255+Transparent), 16 (15+Transparent) and/or 4 (3+Transparent) greyscale colors depending on certain modes.
That is, Without killing backwards compatibility.
NES is already done by someone else, but could be better, so that's redundant, unless completion is required to a degree!
That reminds me, I still haven't published my general theory about Tile Replacement for systems with tile-based graphics (the SNES included). The paper I'm working on outlines the methods I used, how they are implemented, the specific challenges that require more advanced algorithms and techniques to overcome (dealing with color is a big one) and stuff of that nature. Whenever I got around to publishing it, I had hoped it would help others create support in other emulators (Sega Genesis, Wonderswan, whatever floats your boat). As it concerns emulation, I'm pretty much dedicated to working with Nintendo's portables, but my work could easily be applicable to any other tile-based console. Currently I'm laying out how the theory needs to be improved to account for some challenges specific to the GBA, and from what I can tell it is graphically similar to a lot of what the SNES was capable of, so anyone willing could hack SNES9x or higan if they had the time.
How do you plan for your technique to handle text in Hamtaro for GBC and GBA, Pokémon for GBA, or RHDE for NES in PocketNES on GBA?
hint: non-8x8 fonts
Here is a proof of concept for pack releases. It's very hacky. IPS patching random pixel noise data over the CHR to ensure all tiles have a unique hash resulting in garbage data if played without loaded sprites. However with the new custom tiles it works as intended. Working with it graphically to make new art is very time consuming though.
v0.1
https://drive.google.com/file/d/0B5aZT1MLfWJsbFF4cDdaTDhEV1U/view?usp=sharingedit: I will be periodically updating the demo. Maybe
@Teeples - Unfortunately, there is no cute way to handle those cases. If the games split the text between OBJ and BG tiles, that's how the emulator will dump them and that's how users have to edit them. That would potentially result in hundreds of unique tiles. Ideally, any sort of mass editing will be through batch processing.
There is a very WIP idea I came up with as an alternative solution. This method involves post-processing the final layers to identify specific patterns. Whenever that pattern is encountered, the emulator replaces that pixel on the relevant layers. The patterns can be letters or any symbol really, so it could work regardless of whether the font character is split between tiles. After replacing the pixel data, the layers could be merged, accounting for things like transperencies and stuff like that.
It has several drawbacks in that scaled, rotated, and scrolled text may not match the pattern, rendering this method ineffective. It would also be CPU intensive, so it would better be suited as a pixel shader. But even without this method, GBE is able to handle games like Hamtaro just fine for replacing text. It's just that it might not be entirely user friendly.
@GGuy - The cumbersome nature is to be expected. You're essentially working with the CLI and/or a basic text file. I mean, if you want to dump the whole BG, you have to click on it tile-by-tile! The usability is pretty rough without a GUI.
Oh yeah Shonumi the duplicate tile issue I was talking about is the solid color one and duplicate tiles in the CHR that are the exact same but used elsewhere in the game.
I'm not sure if the door and leg thing still happens with Metroid II since that was awhile ago and I believe you changed the hashing method since then. I will check it out using the latest source build and update you.
Edit: Nope still happens.
These two are hashed the same it seems.
How to reproduce the issue: Dump the sprites and walk a few frames, load the dumped sprites, walk and that's it!
Output file is 993993993993993993993993
Congratulations on finding a
hash collision. There are two ways to
resolve this: chaining and quadratic probing.
The thing is, it doesn't look like a hash collision at all on closer inspection. It's just that the hashing itself is incorrect. That is to say the algorithm generates two separate hashes (in stand-alone tests) but for some reason when applied in real-time it generates the same hash for the sprites GGuy pointed out (which I've reproduced now, thanks GGuy!).
The hashes are actually an entire recreation of the pelleted pixel data (in string format) for each 8x8 or 8x16 tile and it uses the two OBJ palletes as a 16-bit salt. Hash collisions are theoretically possible, but most likely only if you tried to match up garbage data with the real graphics. The issue seems to be that the tile in question (Samus' leg) has an OBJ tile number of $5F (data is then at $85F0), and the door has an OBJ tile number of $F5 (data is then at $8F50). For whatever reason, the code is then trying to look at one but not the other.
EDIT: On even closer inspection, it appears that GBE finds the hash for one (the door) and thinks it's similar enough to the next (Samus' leg) that it just ignores the latter. To avoid needlessly re-dumping existing tiles, GBE stores a list of which hashes have been dumped, so if it encounters the hash, it skips the dumping process. The door sprite is actually loaded into OAM (but not drawn) when you start the game right after the title screen. GBE sees this first and dumps it, the stores the hash in its list. It encounters Samus' leg sprite, hashes it, but sees that it's similar to an existing one and passes it up. The solution was to change the salt to account for OBJ palettes
AND sprite tile numbers. So
you were right all along GGuy The solution breaks compatibility with existing dumped tiles, since the hashes are now completely different, although a feature for backwards compatibility can be added in GBE+.
Most of this fancy-schmancy technobabblewhatsit is way over my head, but I heard the words "graphics" and "replace" and "exceed the color limits" somewhere in there, so this looks like my kind of thing.
Once this is all finished and bugtested, I would love to create demo game for you! You can check my signature for some examples of my work and email me once this is finished...Oh, but do you have any ideas in mind of a game that would work well for a demo? Preferably something small so I can polish the game to a mirror shine.
Yay I helped fix something! Hopefully that is also the solution to palette shifting aka color cycling (or whatever it is... bank switching?) sprites when damaged or getting the star in Super Mario Land 2 as well as screen fading on tiles that is used on screen changes.
Welcome DragonDePlatino! I would suggest a stacker puzzle game for something small. Or perhaps Revenge of the Gator if puzzles aren't your thing.
I'm not really familiar with Game Boy homebrew, only NES homebrew. Would an NES game in PocketNES count?
@GGuy - This solution only masks the problem of why GBE thinks the two sprites are the same. It still has drawbacks (if the sprite is the same, but uses two different tile numbers at different times, you have one redundant tile that is also mandatory to edit).I still have not figured out the source. I mean, I know what it is doing on a high-level, but the way I am looking at the code says that really shouldn't happen. But since you have ID'ed the bad dumps, I can debug it properly. Using the OBJ palettes is supposed to account for palette changes (where the sprite itself stays the same, but the shades of gray change). But I suspect it is broken atm, for whatever reason. For example, when you get to the 1st warp star in Kirby's Dreamland, you should get dumps for each flashing version. But.. you don't
@Dragon - If you want to make a demo, I think it would be great. The only major thing to avoid would be solid colors, unless you don't mind that they can't be properly replaced just yet. Also, avoid crazy midscanline rendering (a la Prehistorik Man). Other than that, be as creative as you want.
@Tepples - GBE should work with (most) GB ROMs. I am not familiar with PocketNES, but if it can run as a GB ROM, GBE can grab the BG and OBJ tiles.
PocketNES runs as a Game Boy Advance ROM. It's not for the classic Game Boy.
Okay, thanks for the quick explanation
So how feasible would it be for GBE+ (the next iteration of this project, with GBA support) to run PocketNES and replace the graphics? Very plausible, in theory. Replacing graphics in GBA games itself is not a big deal on a technical level. As long as PocketNES uses the tile based modes (Mode 0-2 irrc) dumping and loading is possible based on my current methodology. The same would apply to any.emulation Nintendo themselves made (e-reader NES games) as well as ports (NES Classics) . To GBE+, it would just be another GBA game. Now, the tricky part is to be an accurate enough GBA emulator to emulate a GBA emulating an NES. That would be biggest hurdle; I bet PocketNES is pretty picky about the emulated hardware.
I see. Well I guess I am going to stick with the noise chr replacement thing I came up with then. I just need the flashing to work and then I'm set to release graphic packs.
I actually fixed palette changes last night. Just a simple oversight. It totally breaks compatibility though. The standard hashing algorithm is still in flux despite the 1.0 release, so these things are unavoidable. Hopefully I can finalize the algorithm soon. Until then, any changes I make are liable to break existing work.
Isn't there any way to convert the old files to the new format?
Eh, incompatibility ain't an issue as long as you back up all of the art you've drawn so far. Really, that's where a bulk of the work goes.
By the way, I decided for the demo I'd like to reskin Kirby's Dream Land. It's a very short game and there characters are tiny so replacing their graphics would be super-simple. But before I draw an entire sheet, can I get a confirmation that something like this would work in your emulator?
@Sik - Not without knowing the 16-bit salt used to create the final hashes. It's just two bytes (MMIO registers OBJ palettes). You might get lucky if the base hash before salting is 0 (the salt is just XORed onto the hash) or if a game picks two OBJ palettes and sticks to it, then you can derive the base hash easily. But even then there would be no (imo acceptable) way to make a standalone tool for completely automatic conversion.
Otherwise, the emulator could generate both the new and old hash, and if an image file containing the old hash is present copy that file but under the name of the new hash. That works, but feels convoluted to me. The easiest thing to do is accept the limitations of 1.0 graphics packs and just add an option in later versions of GBE+ to hash like the old GBE. It'd literally just be an extra if..else... statement or two, so it isn't a big issue.
@Dragon - Kirby's Dreamland is totally doable. In fact, you're not the 1st (plot twist, it was me:
http://m.youtube.com/watch?v=kbd7lGXfRxI) Sorry about the mobile link; blame my phone. As I mentioned, solid colors are inadvisable to edit. To GBE, all instances of that tile are the same whether it's the "sky" or the pause menu background or whatever. There is a way or two around this, but it will be part of GBE+ (GBE's successor, with organized code, a GUI, and GBA support). So for Kirby, I would not touch the sky, but editing the tree's leaves is probably fine. Any tiles that are that same shade and solid (the mountains in the BG in Level 2 for example) will be whatever color you made the tree's leaves in Level 1.
Some good games that you could also try: Megaman I - V, Super Mario Land (Bighead did this), Bomberman GB, Kirby's Dreamland 2 or Block Ball.
I had this awesome idea for a colorization of SaGa using the Genesis palette. I tried to dump the tiles with GBE but they wouldn't... only the sprites would. The tile dumps output blank images.
While I'm not the author of your emulator, I know enough about this stuff to make my own emulator. I won't... seems like a waste of time when you're so far along. But I will tell you that your approach is flawed. You are abstracting your design to the point of obfuscation. I don't know if this is because of training you've received or because you're trying to apply principles from another field with an integrative approach that's isn't what's actually needed.... That said, why not simply monitor data read from ROM? If ever data from a given address makes its way to the VRAM, you flag the address as the start of a tile. Simple, no?
Which version are you using tcaudilllg?
I posted an unofficial build earlier in the thread using of the latest source code before Shonumi moved on to the GBE+ emulator that supports GBA. I tested this build with the English versions of the SaGa series (Final Fantasy Legend) and the original Japanese versions and the BGs dump fine. Here's the link to that post if you're using version 1.0 from the google code site.
http://forums.nesdev.com/viewtopic.php?f=20&t=11216#p137656
tcaudilllg wrote:
I had this awesome idea for a colorization of SaGa using the Genesis palette. I tried to dump the tiles with GBE but they wouldn't... only the sprites would. The tile dumps output blank images.
You're probably using a build of the latest master branch. All of that work is naturally experimental, hence you came across a regression. 1.0 dumps them just fine on my end (as does the latest custom-gfx-hd revision).
tcaudilllg wrote:
But I will tell you that your approach is flawed. You are abstracting your design to the point of obfuscation. I don't know if this is because of training you've received or because you're trying to apply principles from another field with an integrative approach that's isn't what's actually needed....
Okay... It would be nice if you could kindly tell me what parts of the code are obfuscated. If you're simply unfamiliar with the codebase, that's understandable. If you think the code is messy (because it is; that's why I'm working on its successor), that's understandable. If you think there should be more comments, that's understandable too. However, I don't see how this approach is too abstracted. It mirrors what other emulators like Dolphin and Mupen64Plus do: generate hashes for graphics, then replace those graphics based on hash matches.
FYI, I have a B.A. in English. Computer Science is not my field. I just like programming and writing emulators as a hobby.
tcaudilllg wrote:
That said, why not simply monitor data read from ROM? If ever data from a given address makes its way to the VRAM, you flag the address as the start of a tile. Simple, no?
What advantage does that offer over waiting for the data to show up in VRAM and reading it there (the way GBE works now)? Ultimately, the data in ROM is going to mirror what's in VRAM. It won't matter what data GBE uses to get its hashes. There's no point in not waiting for the transfer to finish and looking at VRAM for the graphics. GBE only hashes data at the end of a frame (before VBLANK) so the data will be in VRAM by the time the emulator wants to do anything with it. Reading data from random ROM addresses seems less intuitive than looking at VRAM itself.
I was using 1.0.
I was thinking that you'd run into a problem with the hashes. If you fixed that, then I guess there's no issue. Simply, if you trace from ROM, you don't need hashes.
Anyway I'm glad to see this completed. I think it'll be influential.
Edit: nope. The tiles have not dumped with the unofficial version either. The sprites dumped fine. ...Oh wait... I see now. You have to CLICK on the tiles... they don't just get dumped.
I tested it with 1.0. I can dump BG tiles from SaGa just fine (English version).
Anyway, without hashes, you run into several issues. One of them is that not all of the data comes directly from ROM. A game may compress data (bit-packed fonts for example) and decompress them to RAM, meaning addresses could be arbitrary, and even result in collisions where multiple graphics come from the same few RAM addresses.
Another issue is that the same tile could theoretically exist twice or more in ROM (e.g. copies of a common tile across several memory banks), meaning the emulator would see them as two separate tiles if it does no comparison on the data itself. This would require users to edit the same dump twice (or more) even though visually the tiles are the same.
Lastly, there's tile data that doesn't come from ROM or RAM as direct data, but instead gets generated programatically by the game code (software rendering). If it builds the data one byte at time, it will writing to VRAM from CPU registers. If it builds the whole tile at once, the address will be in RAM, a volatile address.
The point of my approach was to do things in a manner that covered as many bases as possible and would be applicable not only to the GB/GBC but other 2D tile-based systems as well. The problems listed above are only exaggerated on something like the GBA where graphics are commonly decompressed into RAM for example. If you look at VRAM itself, you know exactly what the game is trying to draw. Hash that data sufficiently, and there's no ambiguity or collisions.
I've been meaning to write up the technical theory behind tile replacement via hashing for a while, so that others can implement themselves. I suppose I'll have to finish up the document soon.
I've edited dumped tiles, put them in the load/bg folder, and started up the !load.bat file (after specifying the name of the ROM) but they don't seem to be loading. After the edits didn't show up in-game, I edited the manifest like GGuy said. Still don't show.
What manifest? If you're using the build GGuy made, I can't help you since I haven't used it. If you're talking about the manifest used from the custom-gfx-hd branch, don't use that. It's experimental work intended to handle GBC sprites.
I downloaded the 1.0 tarball from GB Enahnced's Google Code page, compiled that, and was able to dump and load tiles from SaGa. If you're doing anything that's not based on the 1.0 version from the Google Code page, that's likely the cause of the problem.
I went back to the 1.0 version (the binary on Google Code). I dumped the frame tiles at the hero select screen. I opened one of the tiles in MS Paint and colored its interior purple. I saved the file. I copied all the files in the Dump folder to the Load folder. I started with load mode, and now all I get is garbage for both the tiles I dumped and all the sprites.
That's an issue with the format Windows saves their BMPs. You need to save the images as 32-bit ARGB BMPs in 1.0. Older versions of GIMP (2.6 and older as far as I know) did this just fine, but newer ones muck things up. You'll probably have to use an image editor other than MS Paint. Later revisions of GB Enhanced removed 32-bit images because the alpha channel was completely useless, and image editors couldn't agree on how to handle BMPs. Later revisions also added PNG support (loading, not saving).
If you find all of this frustrating, honestly I would encourage you to just wait until GBE+ (
https://github.com/shonumi/gbe-plus) comes out. It does away with a lot of things that are complicated about the process of replacing graphics. I really don't know how people like BigHead (
http://bhemuhelp.co.nf/other/smlc.html) manage to work with GBE 1.0 as it is, especially since it's command-line only. The process will be much smoother when there's a GUI and all images are expected to 24-bit (seriously, the alpha channel is a pain in the rear, both for the end user and GBE). GBE+ is currently in the phase of integrating GB/GBC support; after that comes a GUI, then reimplementing custom graphics.
Shonumi wrote:
If you find all of this frustrating, honestly I would encourage you to just wait until GBE+ (
https://github.com/shonumi/gbe-plus) comes out.
Is it due out before November?
tepples wrote:
Is it due out before November?
I should hope so. The integration process is mainly about porting over the GB/GBC code and tightening things up. The only areas that require non-trivial reworking are the custom graphics and LCD emulation. Everything else just gets cosmetic work and clean-up. That shouldn't take long. GUI programming (Qt specifically) may take a bit more time, since I have no prior experience with it. I don't expect custom graphics for the GBA to make it into the first point release, but definitely for the GBC.
My unofficial build is the last update of the custom-gfx-hd branch complied for people that don't know how to compile. It works fine for me on most regular GB games. GBC games don't work at all for BGs, and sprites for GBC won't load without using the manifest file.
I remember now some games won't load the BG files for whatever reason no matter what. Not sure why. I will compile all the builds and see what works and what doesn't. The 32-bit BMP problem was a huge hassle so that's why I compiled and used the later revisions myself.
edit: Here is revision
ed6790ee337d. I did a quick test and it dumps and loads BG tiles. This is the version before the newer GBC dumping code was added.
Some games won't dump BG tiles (DMG games as well) with the latest custom-gfx-hd revisions either. Plus there are still issues (like the Metroid problem discussed earlier) that I have fixed but have the code to do so. This just means I have to work all that much harder all that much faster