Wranglers, including the newest JK/JKU, are an irrational purchase for most everyone as a daily driver. Those of us that own them would probably agree but include a laundry list of justifications (some for ourselves, mostly for our wives). Despite a million reasons not to buy one, nearly all of us would do it again and again. If you aren't interested in removing the top, then I would recommend anything but a Wrangler. Why compromise so much for an option you won't use?
With your budget I'd vote for a ZJ or WJ since you are interested in towing more than a small trailer. It doesn't take much to get it "trail ready" and you won't suffer from analysis paralysis since the aftermarket is paltry compared to the Wrangler. There's options, just not nearly so many. It is also a much more comfortable ride. A $5k ZJ or WJ is going to feel like you spent a lot more on it.
A XJ is nice, but I believe the tow rating and overall quality fall short of the ZJ and WJ. If you don't care that the interior and paint are crap, this could be a winner if you are wanting something closer to Wrangler off-road capability. A $5k XJ is going to feel like a $5k or less vehicle.