In this post, Dwedit wrote:
The biggest problem with high level languages on the NES is the lack of proper stack support. So local variables don't have a good place to go.
Nobody has written a Stack Flattener that assigns addresses to local variables, depending on the call graph of the program.
Nobody has written a Stack Flattener that assigns addresses to local variables, depending on the call graph of the program.
I seem to remember this idea having been floated before. Years ago, when I was planning to expand Thwaite past a 16384 byte PRG ROM while still letting it share an UNROM multicart with other games, I wrote a tool to make a call graph so that I could find opportunities to encapsulate parts of the program into a separate PRG bank. For example, that big sink near the bottom center of this graph (109K) is a music engine, now called Pently, and I made it a sink because I had planned to allow it to reside in a separate bank.
The source code of this call graph generator could be adapted into a stack flattener for ca65 assembly programs that calculates where in zero page each subroutine's local variables begin so that inner subroutines don't overwrite the variables that outer subroutines expect to keep preserved. But this would need some sort of mark-up for each subroutine to state how many bytes of local variables a function needs. It'd also be a good idea to state how many of these are "caller-saved" bytes, which the inner subroutine is free to overwrite, as opposed to "callee-saved", which the inner subroutine must not modify. This way, the inner subroutine's variables can overlap the outer subroutine's caller-saved variables. Then the processor would make some sort of header file.
What might be a good way to express how many bytes of local variables each subroutine expects to use, as well as how many are callee-saved and how many are caller-saved? Something like Javadoc comments?