If you're starting out from Rose Bay then head north to Nielsen Park at Vaucluse, then head north west across to Clifton Gardens on the other side. You're crossing the main ferry route but you've got clear line of sight north and west at that point, and it's the narrowest point across. Once you hit Clifton Gardens, head north to Manly. This also avoids crossing the heads directly, which can be dicey on the best of days. I'd much prefer to do it further west at Middle Head. Swell rebounds a bit off Middle Head so don't travel too close, but you should be fine.
If you plan on paddling back, which is easily managed, I've found that the paddle south is easier than the paddle north - chop through the heads tends to be front corner on when heading south, which I find more comfortable.