I think SUP's are a good board to have in the quiver for when its tiny or flat. So you can still get out & have a bit of fun on the water. I ride short boards & longboards about equally & dust off the SUP in summer to ride some ankle to knee high waves. For me they're a good board to have in your quiver.....however I do not think they are a good board in bigger waves & yes they are very dangerous in crowded conditions no matter how good you think you are. Surfing background or not.
If I get out on the SUP & there are a few people out I tend to end up surf coaching & wave spotting. Then grabbing left over waves or wide ones everyone is going to be too deep for. But there is usually no one else out on account of it being flat in everyone elses eyes.
I don't understand young people or people without some kind of injury preventing them from riding surf craft offering much greater performance opting to ride SUP's in all conditions. Dedicating themselves to this one board type. They might have a greater wave count, but sacrifice performance...the best thing about riding a SUP is how good a surfboard of longboard feels when you get off it.
Why the animosity towards SUP's

a lot of people see it as cheating. SUP's can spot waves much sooner, get into a position of priority sooner & gain momentum earlier. Far too many take advantage of this & it sux seeing a SUP simply trimming across one of the waves of the day. I do have a lot of friends who ride SUP's & a lot of them are pretty good about not hogging waves. Things were pretty bad 5 or 6 years ago when the infestation began. But it does seem to have chilled at my local. Most of the SUP's will go & surf a certain section of reef. Longboarders on another & shortboarders on another. And if SUP's do come over to one of the other spots, if they exercise some etiquette there's no problems. You don't need to catch sets on a SUP, you can have a great time going for waves no one else would have otherwise caught. The best SUP rider I know does this, he catches wave after wave after wave. But no one gives a fark, coz they were waves no one else was gonna catch. He will often even stop paddling for a wave when he see's another person is likely to catch it.
I imagine Manly is very very different to my local being smack bang in the city. But if you utilize the strengths of the SUP, like their ability to ride small waves or waves with sectioning fat sections other boards won't glide through you can have heaps of fun...also if your able. Why not add another board to your quiver which is not a SUP to use on days suited to other surf craft. SUP's suck in wind unless your downwinding & there is usually a lot less crowd. They're also pretty hard to get out the back in bigger days with no channel or a point to paddle wide. The learning curve is a bit steeper, but well worth it