I've gone down from an 8'10 Speeed to a 7'6 Shroom.
Its much more "involving" catching waves on smaller length boards.
You have to paddle quicker and for longer, and be more effective with your feet to release the board.
Is it enjoyable?
Of course it is

Would I recommend it to others? Yes and No.
You can tell a short board rider by thier limp and lack of coherent speech, so it's easy to teach short board riding to these guys because (like me) we don't think too hard at catching waves, we just paddle like loonies to catch em' then cut them up so no one else can use the wave

Basically (IME) you just have to work hard at balance and technical prowess, but that's half the fun isn't it..
Enjoy, do it.