It's quite good. Sailed there in a westerly a few times. Westerlies being westerlies it's gusty as can be no matter where you are.
In northerlies/NE'rs the chop seems to reflect off the walls a create havoc a lot more than in westerlies.
I'd go so far as to say the best direction for Rose Bay is actually a westerly.
Above is my usual path, but there's nothing to stop you from venturing upwind a little, and then out into the perhaps cleaner air of the harbour central. Perhaps a wind shadow from Shark Island.
There's free (!!!) 12 hour parking at the Sailing Club car-park. You might be lucky to nab a spot closer to the launch point at the beach which is preferable.
Caledonian Road to the east one street is probably a better rigging and launching spot, but parking can be unobtainable.
Note: This location made me buy booties again as I once cut myself up
badly on the oysters that lurk near the foreshore somewhere.