The mosaic filter is applied after the BG is rendered and scrolled but before it is clipped, combined with other BGs, pseudo-hiresed, or mathed. Each XxX block of pixels is replaced with the upper-leftmost pixel of the block. The ‘blocks’ are such that the upper-leftmost block is at the left edge of the screen at the scanline where $2106 was written (or the first visible scanline if it was not written this frame).
Modes 5/6 Hires work slightly differently: they use a 2XxX block of half-pixels. Similarly, Modes 5/6 interlaced use a 2Xx2X block of half-pixels. So if you set $2106 to $0F ("1x1" blocks), the even half-pixels will be expanded to cover the odd half-pixels. $1F would cover the next even-and-odd pixel over as well. An example: put a single red pixel at line #1 pixel #0 of Mode 5 BG1, and a single blue pixel in the same place on BG2. Enable BG1 on main and BG2 on sub, you’ll see the blue pixel only. Set $2106=$03, and you’ll suddenly see both the blue and red pixels. Set $2106=$13, and you’ll see "BRBR" on two lines.
Mode 7’s matrix transformations do not affect the mosaic block positions, so BG1 can be mosaiced about as normal. BG2 in EXTBG mode is weird, though: it uses bit 0 of $2106 to control "vertical mosaic" and bit 1 to control "horizontal mosaic". So if $2106 is $F1, BG2 will expand with 1x16 blocks. $F2 will give 16x1 blocks, and only $F3 will give the expected 16x16 blocks. Note that BG1 still uses bit 0 as usual, so you can have BG1 expanded with 16x16 blocks and the high-priority BG2 pixels expanded with 1x16 blocks on top of it. Or you could have BG1 rendered as normal, but with the high-priority pixels from BG2 expanded 16x1 on top of it.
Each main-screen BG (and the color-0 backdrop, and the sprites (although sprites with palettes 0-3 never participate)) may be marked in register $2131 to participate in color math. If the visible pixel is from a layer/OBJ participating in color math, we perform one of 8 operations on the pixel, depending on $2130 bit 1 and $2131 bits 6-7.
0 00: Add the fixed color. R, G, and B are added separately, and clipped to the max. 0 01: Add the fixed color, and divide the result by 2 before clipping (unless the Color Window is clipping colors here). 0 10: Subtract the fixed color from the pixel. For example, if the pixel is (31,31,0) and the fixed color is (0,16,16), the result is (31,15,0). 0 11: Subtract the fixed color, and divide the result by 2 (unless CW etc). 1 00: Add the corresopnding subscreen pixel, or the fixed color if it's the subscreen backdrop. 1 01: Add the subscreen pixel and divide by 2 (unless CW etc), or add the fixed color with no division. 1 10: Subtract the subscreen pixel/fixed color. 1 11: Subtract the subscreen pixel and divide by 2 (unless CW etc), or sub the fixed color with no division.
In hires modes, color math is applied to the visible subscreen pixels as well. Choosing the math operation is simple: look at the previous main-screen pixel (i.e. if we’re at pixel #6 on the 512-pixel screen (which is taken from pixel #4 on the subscreen), we look at pixel #5 (#3 on the main screen)). If no math was applied to that pixel, don’t math this subscreen pixel either. If the fixed color was added/subtracted, add/subtract the fixed color. And if a pixel from the subscreen was added/subtracted, add/subtract that main-screen pixel (the original value before math). What happens to the subscreen pixel at the left edge of the screen is unknown.
This is really important with color subtraction: normally, if you have a block of cyan (#00ffff) on main and a block of magenta (#ff00ff) on sub, subtraction would give a block of green. Hires math will give you a block of alternating green and red, which will probably appear yellow on your TV. If you’ve set $2131 bit 6 and this block is sitting alone in the middle of the backdrop, you’ll have a bright line at the left edge where the fixed color was subtracted from the subscreen pixel and no 1/2 was applied (because the previous main pixel had the fixed color subtracted and no 1/2 applied).
Note that this may be inaccurate.
1) Go down the priority list to find the first BG/OBJ layer that is enabled on main, not clipped, and has a non-transparent pixel here. You’ll always bottom out on the backdrop (color 0) if not before.
2) If the color window clips colors here, set the color of that pixel to 0.
3) If color math is applicable and the color window doesn’t clip math here, do math.
Hires modes (BG modes 5 and 6 or any mode 0-4 with bit 3 of $2133 set) should process the visible subscreen pixels as described above.