There's usually only two or three people "swimming" there and about twenty or more more kites and windsurfers. Majority rules, ban the swimmers.

This whole thing stinks. Why do bureaucrats always have to meddle? Can't they just fob off the odd whingers?
Here's another point. Windsurfers tend to launch and go, where as a lot of the kite guys tend to stay close to the beach where the people play in the water.
The maritime regulations say to stay 60 metres from swimmers or reduce speed to six knots. This go slow rule works for windsurfers. I'm not sure about kites though. They can be pretty scary to the average bather even when stationary...
Don't get me wrong I reckon kitesurfing is fantastic. It's just that some of the inexperienced guys can be a danger to themselves and others when in a crowded situation.