Hmmm - Interesting 'dilemma'
I own 2 Xterras (2010 SE, 2013 Pro4X) and have spent a fair bit of time, both highway and trail, in a GX470 (as well as several other versions of the Toyota/Lexus family). Not exactly what you asked, but pretty close?
As much as I love my Xterras, to be honest you can't go wrong with the Lexus. Comfort and capability right out of the box is as good/better than anything else you're likely to find.
What are your backseat aspirations? My kids always found the backseat of our Xterra just fine for long trips, and liked sitting up higher, but access is tight - next time you drive one notice how your pant legs brush the vehicle getting in and out. Who cares during a sunny test drive turns into "daddy I ruined my dress" when you get to the piano recital after driving through a spring snow storm.
How tall is your wife? I'm 6', my wife is 5'3" - she has no problem driving the Xterra, but the manual seats need considerable adjusting. I feel like everything in the Lexus is probably power/automatic/magic. The Lexus is an amazing daily/long distance driver.
But for me - the Xterra fits me perfectly. It just feels "right" when I get in it -that's why I bought the second one - the kids ended up with the first one, and when my oldest went to college and my youngest started driving, i realized it was going to be a few years before I got it back, so I bought my own.
Creature comforts - the Lexus is going to win this one. Both of mine have leather, but Xterra leather is not Lexus leather. Not even the same family of cow. I have seat heaters in the older one, and have them on the upgrade list for the newer one, for my purposes that's all I need. Opinions vary, but I find the audio quality in the Rockford Fosgate system in the Xterras to be great, but I'm sure the Mark Levinson system in the Lexus is 67 internet points better. Honestly I like the whole system in the older one better than the newer one.
Hard plastic "way back" area of the Xterra is a benefit for my uses. Dogs, Ski boots, mountain bike tires, camping gear, etc.... Who cares? I can clean even the worst messes out of the Xterra with a damp cloth. I don't know what the Lexus has back there, but I'm sure it's very very nice and very hard to clean.
I've only owned my "new" Xterra since last March, and I considered switching over to the Lexus family at the time. At the end of the day the lower price, my comfort with my first one and the relative "uniqueness" of the Xterra got me to buy the second one. Uniqueness is relative though, I live near Boulder, and there are 2 Xterras and 3 Outbacks (2 of them with rooftop tents) at every stoplight. The Xterra's will only get more unique though, as they aren't making any more but depending on where you live it can feel like "everybody has one". The only thing that will get me out of my Xterra is possibly switching to a truck, now that I don't have a truck bed I realize how much I miss it.