It comes down to worst case scenarios...and risk, which is a subjective thing. For flatwater sailing, the red smileys below are where I'd consider myself in real danger...one thing in common is the worst case scenarios all occur without a life jacket.
Trapped under sailWith life jacket: It's possible to get out... quick release ProLimit harness, takes less than a second. I don't consider this a big risk, so dont worry about it.

Without life jacket: bit easier to get out
Unable to swim fast enough to catch boardWith life jacket: more likely to happen, however if it does you should be able to get back to shore under your own steam. If you cannot then you've at least bought enough time to raise an alarm, e.g. via phone/EPIRB/passing boat. Very embarrasing, and a last resort, but you're still alive

Without life jacket: less likely to happen, but I think it's still a very real risk. E.g. sail and board sepearate due to failed UV joint in crash. The wind could easily take the board flying. If you are out sailing further than you can swim, there's a fair chance you will drown unless you can get help in time

I'm a reasonable swimmer, but in chop with 30knts...i really wouldn't want to have to try and swim 1-2km.
Knocked unconsciousWith life jacket: maybe somebody will see you floating, maybe you will come round before you drown, maybe you will float on your back. Not a good situation, but there's a chance you might not die today. I wear a helmet for this reason...so with life jacket and helmet, I don't consider it a big enough risk to worry about

Without life jacket: game over in about 5 seconds

unless you're wearing a helmet
Dislocated sholder or ankleWith life jacket: unlikely to drown, you've got time to raise the alarm
Without life jacket: quite possible you will down