Speed sailing is windsurfing at its most accurate.
To go fast you have to be in perfect balance and control.
Anything that upsets your balance will slow you down, either through wobbles, bounces, spin-out or worst of all, a general lack of confidence.
Sideways wobbles could be caused by too big a fin, too wide a board or a lack of mast-foot pressure.
As I have improved over the last few years my favourite fin has gone down from 45cm to 35cm to 32cm for the same 7.0m sail. As you go faster you simply need less fin. As to the width of the board, assuming you can't make it thiner, about the only thing you can do is move your foot-straps to the outside positions, which you should already have done if your keen on going fast.
Now, about the mast foot location. All things being equal (including control), having the mast further back
should
give you more speed, as less board in the water will in general give you less drag. HOWEVER, beyond the balance point it can actually slow you down. Imagine your board is a seesaw, and your sail is a small child sitting on the nose. As you sit on the see-saw it starts to go down (back foot) until you bottom out. When your sailboard bottoms out, the nose rises up, gets air underneath making it wobbly, and throws spray
forwards
slowing you down.
So if you "get your wobble on" try moving the mast base forward 1cm at a time. Small changes can have a profound effect on slalom/speed gear! To continue the analogy, it would be like pushing the child further away from the centre so that their light weight can better balance your heavy weight.
The other trick you can try is moving your back footstraps forward one notch. This makes your back foot pressure lighter, giving a potentially flatter ride.
Experiment a bit until you find what feels right. And don't discount boom height and harness line length as they are all important.
So how do you know you've got it right? Well obviously using your GPS to measure your speed is one way. Racing against your mates is another. When sailing solo, I have come to think you can tell how well balanced (or not) you are by how your board trim handles accidental air time. If your perfectly in balance, the board should sail through the air perfectly horizontal with almost no adjustment needed. I haven't confirmed this with anyone, but a few times I've felt really well balanced I've had some amazingly graceful glides over chop.
Hope this helps.