The masking windows are pretty simple. The windows can be used to mask off a portion of any BG on the scanline. With HDMA, they can be adjusted per scanline. They can be combined in various ways, per BG. Specifically, the combination logic can be selected to combine windows in a OR, AND, XOR, or XNOR fashion. Each can be used to select either the region of the BG to keep, or the region of the BG to hide, per BG. All that's left is to see the Registers section and the section "Rendering the Screen" for details.
The Color Window
The color window is rather different. The color window itself can be set to clip the colors of pixels to black (before math, so it's almost the same effect you'd get by setting all entries in the palette to black, then fixing them before you do subscreen addition--the only difference is that half math will not occur), and to prevent all color math effects from occurring. These can be applied never, always, inside the "clip" windows specified for the color window, or outside the "clip" window.
Bits 6-7 of register $2130 controls whether the pixel colors (and half-math) will be clipped inside the window, outside the window, never, or always. Bits 4-5 do the same for preventing color math.
Consider the main screen set up so BGs 1 and 2 are visible in an 8x8 checkerboard pattern, with all the BG1 pixels red and all the BG2 pixels blue. The subscreen is filled with a green BG, and color math is enabled on BG 1 only. You'll end up with a yellow and blue checkerboard. Turn on the color window to clip colors, and you'll get a green and black checkerboard since the subscreen is only added (to a black pixel) where BG1 would be visible. If you clip math instead, you'll get the same display you'd get with color math disabled on all BGs.
In hires modes, we use the previous main-screen pixel to determine whether the color window effect should be applied to a subscreen pixel. See "Color Math" on Rendering the Screen for details.