So when I bought my Frontier (later we got an Xterra), we test drove both the Gen1 and Gen2.
We test drove the 2004 with the Nismo package, crew cab, 2wd, 3.3L V6... The salesman, wife, infant daughter, and I drove up this big hill near Renton, WA. The trucked dogged out and really lost power up the hill.
When we got back to the dealership, we got in a 2005 SE crew cab, 4wd, 4.0L V6... We went up the same hill... We POWERED UP THE HILL and it wasn't even a challenge for the vehicle.
So I chose what I could afford at the time, an SE 2005 for slightly more $$$ than the 2004 Nismo model.
I would suggest looking for a 2005+ Xterra vs a 2004- Xterra... Just my opinion, and you know those are just like **************, every one has one.