I totally get that. you have to take a risk assessment and make the right call. It's how we survive.
I used to actually like the beatings and basically always went out.. but now my body will more likely splat than bounce, so I pick my battles. luckily on big swells, I can drive an hour or two and find a nice Pointbreak with deeper water and a channel to paddle out in.
however:
last year I drove to surf a great break, as Hurricane Teddy was maxing out.
2 minutes after I got to my favorite lineup spot, the biggest set of the swell came right to me. Holy crap!As I was paddling for the biggest wave I had seen in 40 years, I actually thought "you're 73 years old.when the he!! do you stop doing this $hit?"

I was way deep, way late on a huge wave (triple OH)
I committed and the wave was perfect.. I made the awesome 300 yard wall. The doubt left me.. "Yeah, I got this!"
but that break is not as nasty as my local beach break at that size, so not really a big deal to get whacked. I am sure someone on the crowded beach took a pic the crazy old coot on the SUP, but I haven't found one yet
The surfers were all hooting as I paddled back out. One guy shouted "Nice ride Hoss!"

the rest of the session was just max fun!
I never would have gone out at home, where it breaks in shallow water with no channels out.
the pic below is my buddy Kenyon, the same day at my local beach break. He's a beast!