your running into a complex problem I believe there is not a simple solution to. Rgb vales blend based off a photon flux algorithm I believe based off of the way light behaves. Its not quit as easy as adding or subtracting a color and multiplying times an opacity value.
To blend using alpha channels, you can use these formulas:
r = new Color();
r.A = 1 - (1 - fg.A) * (1 - bg.A);
if (r.A < 1.0e-6) return r; // Fully transparent -- R,G,B not important
r.R = fg.R * fg.A / r.A + bg.R * bg.A * (1 - fg.A) / r.A;
r.G = fg.G * fg.A / r.A + bg.G * bg.A * (1 - fg.A) / r.A;
r.B = fg.B * fg.A / r.A + bg.B * bg.A * (1 - fg.A) / r.A;
fg is the paint color. bg is the background. r is the resulting color. 1.0e-6 is just a really small number, to compensate for rounding errors.
NOTE: All variables used here are in the range [0.0, 1.0]. You have to divide or multiply by 255 if you want to use values in the range [0, 255].
For example, 50% red on top of 50% green:
// background, 50% green
var bg = new Color { R = 0.00, G = 1.00, B = 0.00, A = 0.50 };
// paint, 50% red
var fg = new Color { R = 1.00, G = 0.00, B = 0.00, A = 0.50 };
// The result
var r = new Color();
r.A = 1 - (1 - fg.A) * (1 - bg.A); // 0.75
r.R = fg.R * fg.A / r.A + bg.R * bg.A * (1 - fg.A) / r.A; // 0.67
r.G = fg.G * fg.A / r.A + bg.G * bg.A * (1 - fg.A) / r.A; // 0.33
r.B = fg.B * fg.A / r.A + bg.B * bg.A * (1 - fg.A) / r.A; // 0.00
Resulting color is: (0.67, 0.33, 0.00, 0.75), or 75% brown (or dark orange).
You could also reverse these formulas:
var bg = new Color();
if (1 - fg.A <= 1.0e-6) return null; // No result -- 'fg' is fully opaque
if (r.A - fg.A < -1.0e-6) return null; // No result -- 'fg' can't make the result more transparent
if (r.A - fg.A < 1.0e-6) return bg; // Fully transparent -- R,G,B not important
bg.A = 1 - (1 - r.A) / (1 - fg.A);
bg.R = (r.R * r.A - fg.R * fg.A) / (bg.A * (1 - fg.A));
bg.G = (r.G * r.A - fg.G * fg.A) / (bg.A * (1 - fg.A));
bg.B = (r.B * r.A - fg.B * fg.A) / (bg.A * (1 - fg.A));
or
var fg = new Color();
if (1 - bg.A <= 1.0e-6) return null; // No result -- 'bg' is fully opaque
if (r.A - bg.A < -1.0e-6) return null; // No result -- 'bg' can't make the result more transparent
if (r.A - bg.A < 1.0e-6) return bg; // Fully transparent -- R,G,B not important
fg.A = 1 - (1 - r.A) / (1 - bg.A);
fg.R = (r.R * r.A - bg.R * bg.A * (1 - fg.A)) / fg.A;
fg.G = (r.G * r.A - bg.G * bg.A * (1 - fg.A)) / fg.A;
fg.B = (r.B * r.A - bg.B * bg.A * (1 - fg.A)) / fg.A;
The formulas will calculate that background or paint color would have to be to produce the given resulting color.
If your background is opaque, the result would also be opaque. The foreground color could then take a range of values with different alpha values. For each channel (red, green and blue), you have to check which range of alphas results in valid values (0 - 1).
A child's dream never dies.