Pilchard, I'm in Portland, but not a wavesailor (unless you include my first few attempts over the last couple of weekends - no ability though).
Cape Bridgewater would be your closest (about an hour's drive, through Nelson, turn off toward Portland Airport & follow signs past Bridgewater Lakes & turn right on Bridgewater Rd...or use GPS?). Great on a S - SW, and I've been told NE?!? I've only sailed there a few times recently, but plan on doing more wavesailing this year. Apart from the few local spots, next would be Pt Fairy, over the back of the golf course.
Any windy day you'll find us locals out off Dutton Way (5 mins from Portland toward Warrnambool), where there's a mix of flatwater, chop & breaking reefs. Generally, the most on the water at any time is about 4-5 guys, although 7 on the water on Sunday, & was great to kick back afterwards with a coldie & exaggerated stories of how high we jumped, how good our gybes were & who had the best catapult!

Send me a PM if you're coming over.