I've owned 3 XJs, 2 ZJs, 1 WJ, and 1 WKII and I'd not have any problem building any of them - except the WKII (too much fanciness to go wrong on the trail). My current off roader is an 01 XJ, and I love it, but its not as comfortable to drive long distance as the other models. My '96 ZJ has been my most reliable vehicle to date having owned it 17 years and done just over 200,00 miles personally on it. However, my '95 ZJ, which was a 4x4 with the 5.2L V8 was a much more powerful truck and more fun to drive. I wasn't as fond of our '04 WJ as I was either of the ZJs, but I the advantage for overlanding is the size of the interior and the headroom (important for a 6'6" driver). Since my trail rig is an XJ it goes without saying (even though I just said it) that I spend a lot of time in the junk yard. I've noticed a SEVERE drop off on the number of available XJs in the yards and the ENORMOUS increase in WJs.
If I had to start over I think I'd look for a 1998 ZJ 5.9, a 96-98 ZJ 5.2. I'm not convinced I'd build another XJ - and this coming from a guy on the board of directors of the North American XJ Association.