My experience is get a motor that will get you around in your home waters. I have a 2.3 hp long shaft and it gets me around just fine (I sail in a lake -flat water). I do think anything over 3.5hp will get heavy taking on and off the stern. Stick with the long shaft and you will be happy you did. Reason being if you sail in chop/walk around the boat under power or the weather kicks up, your prop will be coming out of the water all the time. Also with a shorter shaft you risk the chance of getting water splashing in the motor (since it will have to be closer to the water) and that creates more problems.
I had an air cooled short shaft sears 5hp, it was ok, but had to keep crew in the back until we got up to speed. Couldn't keep the prop in the water solo. If it wasn't air cooled we would have cooked it on account of no cooling water. Ran a 3 or 4hp 2 stroke long shaft 2cyl that worked best, except it was hard to start and was a loaner. Switched to the 2.3hp which was the cheapest new I could find. It's a little weak going out against 4ft waves and 15kt breeze, can only do 1kt into that and definitely needs main assistance in those conditions. In flat calm it can do 3.5 kts, it is a little slower than the others we've tried which could get hull speed. Bogs down a little towing another J24 in flat calm. If I had it to do over I should have spent a little more, waited and got the 3hp online, but I went with the 2.3hp because it was local and cheaper and I didn't have a motor for that week's racing.
Short shaft outboard on a 24 isn't worth your time. It becomes obnoxious trying to keep the prop in the water without two people on the transom.
If your water doesn't move, you can go below 4hp, but it does get tricky to get a motor of the required minimum class rules weight.
Class stalwart motor was the OMC Johnson/Mercury 4hp 2 stroke, external tank, or for lightweight guys the Tohatsu 4hp top loader. Both of those older but very reliable designs were killed by the 2 stroke carb regulations around 2000-2001. Used examples in excellent condition should be readily available.
Guys who buck the tide often or deal with places where the water moves a lot prefer the Mercury 5hp. They typically win the race to the hoist. We're also starting to see a few early adopters of the propane driven 5hp alternative now available, but I can't speak to longevity yet.