I drove a 1st gen and a 2nd gen for work. Both trucks stock.
Both are awesome. I always felt like the 1st gen was perfect for 2 people + gear. Back seat was marginal for carrying others. That truck was a V6 with manual trans. Very enjoyable to drive but I always felt like the engine had "ample" power. It had enough to get the truck moving but if you hooked up 2,000lb trailer to it, it had trouble.
Off road it was great. I thoroughly enjoyed it (geology work). Rear limited slip did its job.
The 2nd gen brings the win. Bigger truck, more room, more power. Not as nimble as the 1st gen but better all around. The engine was straight ballsy and the 6 speed shifted like a dream. I wasn't a fan of the low range gears as it didn't feel low enough to me. However the low range gears are probably fine if you go with an auto. Towing was better and it handled loads a lot easier.
If the truck is going to be a 2nd vehicle and you want to do a bunch of trips and not carry a bunch of crap, get a 1st gen. Do a little lift, rebuild the front suspension, do some other maintenance, throw a rear locker in and tires, go. The truck is small enough to be nimble on the trails which is nice. However for a daily driver, hauling the kids + gear or high altitude driving, get the 2nd gen.
Just my uneducated opinion. I do feel like the current model Xterra is probably the best SUV in its class for actual off road use. I'd rather wheel one than a jeep. Simpler and cheaper than a rover. Everything else in the class doesn't even have low range. I told my dad to get one to replace his aging Tahoe.