Doing this sort of thing really isn't as hard as you'd think.
Most of the blending types even by Masks are quite simply equasions, particularly blending in this mannor.
Remember 3 years ago myself and GuyS were doing speed experiments to see if R/T Blending was feasible in DarkBASIC Enhanced. Unfortunately not for R/T needs, but for pre-loading .. let's say Lightmaps heh it did the trick will albiet slow.
Generally the Equasion for Blending is a Simple Multiplication.
NewPixel = (Source1 * Source2) / 2
When a mask is involved what happens is only slightly different
fSource1 = (Source1 / 100.0)
fSource2 = (Source2 / 100.0)
NewPixel = ((Source1 * (Mask / 100.0) * (Source2 * ABS(Mask - 255) / 100.0)) / 2
[edit] heh got it wrong, but also that isn't the quickest way.
fSource1 = (Source1 * 0.01)
fSource2 = (Source2 * 0.01)
NewPixel = ((Source1 * (Mask * 0.01) * (Source2 * ABS(Mask - 255) * 0.01)) / 2
that doesn't look right but still multiplying values is far quicker than division.
You can then get really complicated and fun with things. I have a website explaining the mathematics behind the Photoshop Blending Modes if anyones interested.