Blast processing is amazing. I mean the actual, real deal Blast Processing, the technique described to the Sega marketing guy that he didn't understand at all. It is an extremely technical bug, that people knew about for decades, but was considered flawed until 2012 when primarily 3 coders - Oerg88, Nemesis, and ChillyWilly, developed a method to perfect the technique. I've been trying to write a blast processing demo myself without looking at their source codes for several weeks now and I still can't replicate their steps without cheating. But the process is so insane it's worth talking about.
So, firstly, what is blast processing? Most people assume the term refers to the general speed of the Sega Genesis CPU and how that speed allows the Genesis to do more runtime (read: CPU bound) special effects than the Super NES, thinking "blast" as a term for quickness. Blasting, in fact, is a demoscene term that means sending data to a piece of hardware without an interface. In this specific case, "blast processing" referred to a way the programmers at Sega could "blast" data at the Video Display Processor, the chip that outputs the final image to the television, ignoring hardware data structures and interfaces the chip has built in intended to let people control it, using a method called "Direct Memory Access" to produce strange, even fantastic results way, way beyond what the Sega Genesis hardware could supposedly do.
There is a very good gamehut (head and lead programmer from Traveler's Tales) episode where he discusses stumbling upon the technique back in the 90's and why he, and sega, abandoned it:
That video gives a pretty good broad overview of the concept, but skips a lot of the technical nitty gritty that makes the entire process a lot more amazing. And, more importantly, it skips the recent (relatively) discoveries which now makes the effect viable. In short, to recap what the video above talks about, and go into a bit more detail, the Sega Genesis VDP exposes a few hardware ports to the Sega Genesis CPU and DMA controller which allows software to modify the video hardware. This is intended, for example, to let the CPU change palette colors in the VDP's Color RAM. The problem is that these exposed ports are singular working in parallel, rather than dual ports. Meaning there is only one port, for example, to both read and write to Color Ram in the VDP. When you write to the Color Ram, you are simultaneously using the same ports the VDP uses to display color on the screen. This causes an error known as CRAM dots when you change a color -- the current position where the VDP is drawing to the screen will display a pure RBG value that equates to the 15-bit value being written to CRAM
Now, the Genesis displayable screen is actually much larger than the normal television display, extending far to the left and right (and up and down). As such, most programmers can hide palette changes so they occur when the VDP is drawing offscreen pixels, hiding the artifiact. But some games can't get the timing down and CRAM dots are visible on a real TV screen. What Sega, and Traveler's Tales, and modern demoscene coders realized is that, if every time they changed a palette color, they could produce a 15-bit color pixel on the screen, then they could write loops that would do this over and over again and cover the entire screen in 15-bit color pixels, producing direct color images with a color palette of over 4,000! Way, way, WAY beyond the master palette of 512 the Genesis can produce, and the 61 it normally is only able to display. More importantly, these images can be treated as direct color framebuffers -- where each pixel can be directly accessed and drawn. Old video games systems prior to around the time things like the 32X and 3DO and Jaguar started appearing, used Tiles and cel structures to quantize screen space and data, making it possible to full entire screens with images with only a little bit of memory, at the expense of losing fine pixel granularity. Which is a long way of saying that the Sega Genesis was never designed to let you select and draw single pixels on screen. Instead, the smallest element you are supposed to be able to draw with is an 8x8 tile, which either has to be aligned to a grid (and thus can only move left, right, up, and down in 8 pixel steps), or as one of 20 hardware sprites that can be drawn in any location, but don't provide nearly enough to fill an entire screen. Being able to directly plot pixels is what allows systems to do
3D plotting, forr example.
So this blast processing ability was super, super useful. It gave the Genesis a drawing palette that was 63 times bigger than normal, and let it manipulate the screen directly in enviable ways that wouldn't become the norm for an entire console generation later. Sega knew about it, and knew it was actually a potentially killer technique, but it was never used in a single commercial game. Why? Because the entire method requires extremely strict timing, and the Genesis provides no method of syncing the CPU to the VDP. You have to understand that the CPU is a microprocessor, and the VDP is also a microprocessor, and they run entirely independent of each other. It's almost like they are two separate computers, sitting on a single motherboard, each with it's own RAM and timing. The ports I described earlier are areas of shared ram between the CPU and VDP that let them communicate, but they are asynchronous. The CPU sends data through the port, and then when the VDP is ready, it reads it, and vice versa. It's normally up to the programmer to make sure the read and writes don't collide, and that is done by scheduling these commands liberally, with expectations of large gaps between when one CPU writes and one reads. As such, any methods of synchronization between the two chips is vague. This is compounded because there are very slight timing variations between different models of Sega Genesis. And, unlike other retro systems, the Genesis didn't provide any syncing commands for video horizontal video signals. By contrast, consoles like the C64 and Atari 8-bit computers could provide commands that would make a CPU wait until it got a signal from the television to begin drawing a horizontal line.
These variations in timing means that Traveler's Tales, for example, began experimenting with manual calibration -- letting users use the controller d-pad to set values that slightly altered timing until the picture came in correctly. But they decided that was too cumbersome, and it was essentially impossible to write an application that could discern for itself where to start drawing, to be able to sync up the DMA routines. If your timing is off, what happens is that the image jitters left and right per scanline, and these jitters can be huge depending on how off the timing is. Sometimes like 20 pixels off. That ruins the picture.
Well, in 2012, as mentioned earlier, 3 long time demoscene developers finally figured out a method to sync CPU instructions to VDP instructions based off of VBlank. The entire process of figuring out the precise steps to sync any Megadrive this way is fascinating. Firstly, understand how the Megadrive itself keeps timing of its chips. There is a master clock that cycles at 53.693175 MHz on every megadrive. This value was chosen because it times with the other chips in whole values:
53.693175 / 15 = 3.57 MHz, the speed of the Z80
53.693175 / 7 = 7.67 MHz, the speed of the 68000 cpu
53.693175/10 = 5.36 MHz
This last figure is the speed of a clock inside the VDP called the Pixel Clock. This Pixel Clock controls the master timing for the entire VDP chip. Every tick of the Pixel Clock, a second counter called the Serial Counter is ticked twice. Thus, for every pixel that is processed by the VDP, the serial clock ticks twice. This was discovered by Nemesis using a logic sniffer and lots and lots of notes:
These are his timing notes from sniffing the bus to the VDP over a full frame, noting every VDP operation takes 4 serial clock ticks. The actual logic is very complex, because VDP operations are nested, and actually take 7 serial clock ticks to execute, but for the purposes of timing, only the start of each operation is important, and the first tick of a new operation always happens on the 4th tick of the last operation, making 2 operations work in succession for the final 3 ticks of each cycle. This was discovered by Nemesis mapping the timing access patterns like so:
With a hard reference of how long operations took in the VDP relative to the master clock, a formula of specific commands done in a precise order was discovered that could narrow down execution to precise synchronization. The first step in the process is to give the command to the Genesis CPU to wait for vblank from the television. Despite not having fine sync controls, the Genesis CPU can tell when Vblank occurs on the television. However, it does so vaguely, and gives an error range of 12 or so pixels. Meaning the Genesis can think VBlank begins on any pixel clock count from 0-12. That's a large range that can induce jitter.
The next step in the process is to fill up a number of First-In, First-Out memory queue slots the VDP has. To speed up operations and make scheduling more easy, the VDP doesn't just let you write one command to it at a time, but rather you can buffer 13 commands in a field of memory that is FIFO, and the VDP will execute them in order sequentially, popping them off the queue. The timing of this operation, with the timing of the VDP vs the timing of the master clock just so happens to evoke an effect similar to the
Collatz Conjecture. The Collatz Conjecture is an algorithm with a special ratio that, when repeated over any integer N, will eventually devolve into a finite pattern of 4, 2, 1. The exact ratio of the VDP timing does not produce a finite pattern, but it does eventually devolve into a 2, 1, 2, 1, 2, 1, 2, 1 alignment pattern over several operations (and the ratio eventually desyncs after too many iterations). Thus, if you fill the FIFO memory with enough commands, it'll devolve into that alignment pattern, and the amount of memory the FIFO can hold is too low to desync. So after these two steps, we are now within a 2 pixel clock cycle alignment of our goal.
The final step of the puzzle relates to refresh cycle slots. If you refer to this image:
You see that on Serial clock count 156, it performs once a "refresh cycle." This is because the Sega Genesis uses dynamic ram for its color ram rather than static ram. Dynamic Ram is cheaper than static ram, but it requires a jolt of electricity every few moments to keep the data inside of it alive, vs static ram which keeps the data alive until power is cut entirely. These "refresh cycle" slots are moments when the genesis sends some electricity to the DRAM. This is an extremely high priority process, taking priority over every other command. If you perform a DMA within this Refresh Cycle, it'll wait until a 2 serial clock boundry is hit to perform it because it needs to refresh DRAM, which will eliminate the potential erroneous 2 pixel clock cycle offset (if it exists) that is caused by filling the FIFO method. This is because the math works out to the DMA command is able to be either 1 or 2 serial clock tick bound given our error, meaning those ticks can happen over any 2 serial clock tick moment. But the Refresh Cycle takes a full pixel clock tick, which itself works out to 2 serial clock ticks. Regardless of what our DMA command's alignment is - 1 serial clock tick or 2 - it won't execute until definitely 2 serial clock ticks go by because the Refresh Cycle needs to occur. You have to carefully time your DMA command from your CPU after you fill the FIFO queue to wait specific amount of time to hit this reset cycle reliably, however, and it's been worked out that a specific number of NOP commands (no operation) called by the CPU will align perfectly to the reset cycle. This is currently where I'm stuck, I've been trying to decypher the math to figure out how many NOP commands align to the reset cycle. Chilly Willy, Nemesis, and Oerg88 know the number, of course, but part of the fun is working out the math from the timing on your own. I know the number of NOP commands is circular , occuring again X number of ticks again and again. So if the number of NOP commands is, say, 5, and the cycle is 20, then it'd occur at 5, 25, 45, 65, etc. But again, I haven't worked out the cycle.
All this works out, however, to perfectly timing the VDP to the CPU to the Television screen, without any external input from the player. Once that's done, the effect is unlocked. The Megadrive doing blast processing is really awesome, but sort of useless because it halts the 68000 CPU, you can't do anything while it's drawing to the screen because the CPU is being used to blast the pixels to the VDP. So it's mainly good for static images on the Genesis.
BUT with a Sega CD attached, the proposition becomes entirely different. Since the Sega CD has it's own 68000 CPU that operates independently, it can use the Genesis 68000 CPU as a blast processor to draw a direct color 4096 color image to the screen, while it's own CPU handles the game logic, music, and everything else. This is how things like the Sega CD port of Wolfenstein 3D is done.
There are drawbacks, however. Because there is a refresh cycle of DRAM every several "cels", this manifests on screen by a vertical row of pixels being twice as fat. These vertical rows happen 5 times per frame, so there are 5 rows of "stretched" pixels across the screen. This can be hidden by carefully managing your image output, or by use of noise generation to hide the artifact. Another draw back is that blast processing halves the horizontal resolution, so you're working at a 160x200 resolution. Actually, the effect draws off screen and if you have an HDTV which does not hide overscan, you can actually output a widescreen image from the genesis this way at 180x200. Despite these draw backs, Blast Processing is a super useful video mode, especially when it can be enabled and disabled at will. It's especially great for large, screen filling colorful images.
Blast processing is absolutely amazing stuff. It's a weird instance where Sega knew about it, advertised the hell out of the feature, but totally didn't get it and never once used it. But it WASN'T marketing fluff, and it really could let the Genesis (and especially Sega CD) do some honest to god amazing stuff for the hardware.
This effect was thought impossible to achieve with sync by Nemesis originally. It was Oreg88 who stumbled upon the repeated pattern the VDP timing of the FIFO queue would devolve into after repeated operations, and Chilly Willy with Oreg who figured out the exact number of NOPs needed to sync everything. Really, really great cooperative teamwork there.
You can see this trick all over Overdrive 2 by Titan:
And a quick pic for those who don't want to watch a video:
