Are you using anything to pull with or using bare hands? Just to state the obvious, you can use the harness hook with a couple of wraps of the line to pull the downhaul. I use that when on the water to spare my hands when tweaking downhaul, especially on the big 9.0 4 cam.
On the beach I use a clam cleat, which is just a piece of metal with a cleat that's easier to pull the rope with. Other people use winches but wanted to make you aware of the harness option if you weren't already.
On smaller sails (6ish and down) I have a cyclops extension that has a protected plastic pull handle that makes it really easy to adjust downhaul, but bigger than that it would not be easy as it only uses a couple pulleys while the bigger sails use 3 or 4.