Providing your steering effort at idle speed is "normal", then what you more-than-likely are experiencing is "prop-torque". Ideally, optimum set-up enables you to achieve max speed with perhaps 1-2 degrees of positive trim. Max efficiency is had when prop is parallel to the water. The less the motor has to hold the bow "up", the more horsepower is available to push it forward. Experiment with weight distribution and prop height. The more trim you use, the more difficult it is to turn.
Does your g'case have a built in torque tab? Identifiable by a "wedge" cast into the starboard skeg.....