The real issue with a trailer off road is the extra stress on the tow vehicle. Pulling a trailer up a powerline trail strewn with scree is likely to cause a bit of wheelspin and what lurks beneath can be deadly on tires, ask how I know. My trailer weighs 1500# fully loaded and my Jeep is almost empty on trips, everything rides in the Jeep but one tire spun, dug into the scree and found a sharp rock. I doubt it would have happened without the trailer.
This was one of those places I was crawling, 4L, 2nd gear, not much more than an idle. But dragging 1500# made a difference.
I've never felt a loss of mobility with the trailer, actually after 20 years with a F250 Crewcab, the Wrangler and trailer are capable of driving around obstacles the F250 would need a chainsaw to get past. But weight is the enemy. Regardless of what you overland in if you don't need it, don't pack it.