On Sunday I had a nice session but I did not seem to be able to get to the speeds I thought I should. A few other sailors that were out were pulling away a little and although I am not as experienced as them I thought I would have been close.
I experimented with fin size and the smaller fin was faster but still it wasnt feeling right. As the winds werent overly strong, I released a little downhaul and it made a little difference but not a lot. I also slackened the outhaul but that seemed to make no difference.

Today I was reading a mag and the guy was saying that he has less downhaul but increases outhaul to flatten the sail to get more speed. Is this right? If so then maybe I should have been increasing outhaul? Anyone have any ideas as to what setting would increase speed?

Depends on angle of sailing. If you are going broad for max speed. Rig overpowered with plenty of downhaul and little if not negative outhaul (assuming cammed sail).
Upwind an adjustable outhaul will allow you to increase outhaul and flatten the sail to point higher. If underpowered a little less downhaul will increase power but decrease range. Board trim(mast foot) is also important along with strap position.
www.seabreeze.com.au/forums/topic.asp?TOPIC_ID=42231
There are some other threads discussing these ideas and as they say, change one thing at a time and see what works for yourself. If you can get someone to take some pics/ video and post so we can see your stance and set up that can help with specifics.
Good luck