Remove it and try it. You will know soon enough if you can live with the difference. It will, however, alter your handling and increase the risk of loss of control if you do something like make a sudden lane change at speed. If you do this with a load in the back, it only gets worse.
There is more involved than simple body roll. Steering geometry changes when one side of your suspension is extended and one compressed. The changes will be reflected in the direction your tires deflect, and you may find yourself over correcting or getting into a cycle of over correction that can lead to loss of control.
The problem with removing a sway bar is that all of the improvement you seek will be found on the trail. If you only use it that way a fraction of the time, you have to live with the difference most of the time. Try it and make sure it is something you want to do. It may be that it is more comfortable to put up with the sway bar on the trail than without it on the highway.
