That's what I call monkey mast business - a headache.. Would be best if you posted a vid so people could see some changes when you apply more/less downhaul. I use Nolimitz 370/400 on my Naish Choppers, Severne Blades and older Goya sails are they look very good. But they are fairly small sails and I guess compatibility problems diminish the smaller the size..
In your sail (again, my subjective opinion) it looks like the mast is harder bend for what your sail needs. From the pic I can see already pretty shallow pocket and already i see some flapping edge of the sail that extends from the top almost all the way to the boom. Not amazing. Because if you apply more downhaul that entire top to boom flapping region will become even deeper and you will loose the pocket completely. Which means you may loose a huge part of the sail's bottom end and likely stability as well.
With my limited experience (I try to avoid messing with masts as it waists ton of time) soft top mast would have given you very flappy top batten or two and big pocket immediately. Harder top mast for that sail would have required more downhaul. You would get shallow/no pocket (flat sail) and get some flapping between top first/second batten and pretty tight leach... What happens here in my opinion is some Constant Curve mast which is somehow generally harder for what that sail needs..
not sure how to fix it... Grab some softer / different masts from a friend and see what happens. Problem is that it kind of sucks to have a different brand mast for that specific sail which is a proto too.. That mast/sail thing IMHO is scariest thing in windsurfing. Since I don't stick to same sail-mast brand, I never buy new sails or masts to take less $ impact because of high incompatibility risk. Even within same brand it may happen.