You live in Idaho and you have a
2wd Xterra?
Must be the only one in the state.

(just busting your chops - there was a guy on the 4runner forums who had a 2wd 4runner in CO and people there used to give him a lot of grief, too.)
Anyway, are you sure the Taco's you're seeing are 4 doors? Only reason I ask is that from what I've seen, 4 door 4x4 Tacos are priced higher on the market than similar year/mileage 4runners, probably because there are more 4runners. If you can find a DC taco that's in your price range, that would probably be a good way to go, though.
Assuming you're buying used, one nice thing about 4runners is that they are typically "soccer mom vehicles", i.e, have rarely or never been off-road. The biggest downside to the 4runner is that the factory suspension and tires are typically "soccer mom" type, as well (that is, a low, soft suspension with road-oriented tires.) However, these shortcomings can be fixed by upgrading both.
Upside to the Taco is if you can find a short bed TRD I believe they all have the factory locking diff. I had a locker on my '99 4runner and loved it. Downside is that Tacos are often bought by off-roaders or people who haul dirtbikes, mountain bikes, etc, and are sometimes more abused (this is something I've noticed with pickups in general.)