I think you're about right with what you're describing.
Most of the time, I use about 1.6 degree rake (1mm shim with my foil and board) with the IQfoil. More when it's windy and choppy or doing slalom and less when it's light and doing course racing.
To sum up, it's a trade of between having the nose too low, crashing hard if you touch and having the nose too high with difficulties to get on the foil in light wind and losing a tiny bit of control (because you're rig is higher up) in very strong wind.
I tried 4 degrees in very light wind and at first I could easily go on the foil by pumping harder but after a couple of races, I screwed up a tack and was too worn out to pump hard enough to take off again. You don't have that problem in strong wind.
My setting (which could be different with your foil) is:
Course racing:
Light wind: no shim or 0.5mm if choppy.
Medium wind: 1mm shim or 1.5mm if choppy.
Strong wind: 2 mm or 2.5mm if choppy.
Slalom: 2mm in light wind and 3mm medium /strong.
Like you, I prefer foiling without the shim and with the nose as low as possible but I started to have enough of cracking my board's nose with any touch down
You just have to try the different shims and find the setting where you're confident you can push without doing a catapult if you touch.