I can't comment on the Toyota--never owned one. However, I can vouch for the 4.0 XJ reliability. We have a '97 Cherokee Country that we bought in '04 with 50k on the odometer. I lifted it 4 inches and installed 31" BFGs within a couple months of buying it. We went back to 30" BFGs around 110k. This was my wife's daily driver and the family transportation until '10 at around 180k. During that time the only non-routine maintenance items were a cracked exhaust header, a power steering pump, and the switch on right side of the tranny that senses shifter position. We're in central Indiana, deep in the rust belt. We replaced the XJ as my wife's daily driver because rear sub-frame rust had me concerned about the strength of the uni-body. At that point, I didn't think the body would last long and all but stopped doing routine maitenance--there was likely a 25k span between oil changes at least once. It became my beater & winter driver.
At about 235k, the unibody wasn't showing any sign of weakness in spite of the severe rust. I replaced a cracked steering box and started doing routine maintenance again to see how far it will go. I also replaced front axle u-joints and front wheel bearings for the first time. It's currently showing about 244k on the odometer. Corrected for tire size over the years, that's about 265k. While it hasn't been wheeled heavily, we live in the country on very rough roads. It's always been driven much faster than it should be on those roads--never been pamper pampered. These days I'm driving it daily to keep miles & salt of my LJ & Miata. It's still getting over 18 mpg. Total cost of ownership, including fuel and insurance has been about 25 cents/mile so far.
I'll echo the comments about rear seat space. When our daughter hit her teenage years, the rear seat got cramped. That was another major contributor to finding a new family vehicle.
When you look at XJs, pay close attention to the reinforcing channel directly over the rear axle. I looked for another XJ before buying the LJ, and that spot seems to rust first. For some reason they seem to be worse on the passenger's side--maybe exhaust heat is a contributor. Once there's a sign of rust in this area, holes open up fast.
I'd also avoid vehicles with sun-roofs. The controller lost it's mind several years ago and started opening randomly & moving in random directions when you push the close button. The controller is above the head liner and the factory service manual doesn't cover it. I recently had to disconnect some of the linkages to get it closed and then caulk it shut.
As much as I prefer a manual transmission, the Aisin automatic has been trouble free except for the external switch mentioned previously. I had an '89 MJ with the Peugeot 5-speed which was also trouble free until the truck was hit & totalled with about 75k. With a manual tranny, the MJ got 21+ mpg with the same 30" tires as the XJ.