There's a number of different things that could be happening, but there are two ways to steer a windsurfer.
One way is to use the board, by weighting a rail. Basically, if you're planing then the board will turn in the direction of the weighted rail, and this is referred to as "foot steering". There's an exception to this, which has to do with what they call "railing up" which is a result of fin pressure, but we're getting into advanced territory with that, and I don't think you're there yet :)
If, however, you're not planing (were you planing?) then the board will turn in the direction opposite to the weighted rail. I might be corrected on this though as I know a centerboard also impacts this, but I know you're not using a centerboard.
The other way to steer is using your sail. Tilting the mast forward or aft will point you downwind or upwind, respectively.
Windsurfing is a fluid sport, like most others. So you can't just take your hands off the boom and expect to remain on exactly the same course. In theory, yes, but not in actuality. If you take your hands off the boom then the sail balance will likely change and even your weight on the board will likely change. It might only be subtle, but it will impact the direction in which you're traveling.
I would think that if you took your hands off the boom and you rounded up into the wind, then your sail was falling aft to some degree, in which case you wouldn't have been letting go of it for very long.
I suggest you checkout Guy Cribbs harness line article:
www.guycribb.com/userfiles/documents/The%20Truth%20About%20Harness%20Lines.pdf Guy Cribb has plenty of good infor for beginners right through to advanced sailors.
Good luck.