Not at all.
The only reason I don't have polyester speed boards anymore is that the mould for the blanks has been destroyed and we can't get suitable blanks anymore.
For a board such as you describe, there are actually advantages. The slightly 'damper' ride of a polyester board eats up chop much better than stiff carbon boards which rattle your fillings out. It is interesting that most Carbon Art brand speed board actually have only a small proportion of carbon fiber in them, and to me that is an advantage.
In a small speed board weight is not an issue and lighter is not faster. We have seen people experimenting with actually adding lead weights to speed boards with favorable results!
up until about 2006 I was regularly using a couple of 47-48cm wide polyester speed boards with great success. I have a 44+ Knots peak on one of them! I was also using a 21" wide slalom and it is still the fastest board on choppy water that I have ever had.
The biggest problem I had with my polyester boards is the long term stability of the rocker shape. I re-faired to rocker of my 47 speed board a couple of times after it developed slight negative tail rocker due to warping. (This can happen to epoxy boards as well though).
I eventually ripped the US type fin box out of the slalom board which ended its life. If it had a Tuttle box it might still be alive.

If you know a source of quality polyester blanks suitable size for speed boards I would be most interested. I recon I could still build a polyester speed board for about a third the cost of an epoxy in about a quarter of the time. I have a few pretty radical ideas I would like to try but I will never do it in Epoxy. Too much trouble and mucking around.