~4k would be my starting budget, and then I could slowly work from there when I am able to get more.
I'm 26 actually, and my current ride is IS300 that was my fathers before he decided he wanted something else. I like the car a lot, but I would like a 4x4 for overlanding and seeing parts of the country less traveled, something not offered in the Lexus. I would keep that as like the DD.
When buying a used vehicle, you need to account for the routine maintenance that should be performed before embarking on any extended trips. Such items would include servicing the engine, transmission, t-case, and both diffs. In addition, it would probably be a good idea to flush and fill the radiator, replace all belts and hoses, possibly replace brakes and repack or replace wheel bearings. You may need to replace the timing chain or belt as well; a $4k vehicle is likely going to have some serious miles on the clock.
All of those items are necessary to prevent any catastrophic failures and should be considered a minimum for safe travel off the beaten path. As you can imagine, doing all of that maintenance will take a considerable chunk out of your $4k budget, hence the reason I asked if you're willing to do some work yourself. Even if you do perform all of the labor, this will still take a significant amount of money. This is something you really should factor into the budget regardless of vehicle you plan to start with.
The reason I asked about the 4+ trail capability is most of your travel goals could possibly be done in a soft-roader (Subaru Forester, Honda CR-V, etc). These vehicles should allow you to average the fuel economy you desire, but don't have a low range for serious off road use. (A dual-range transmission can be sourced for the Forester, but that involves importing one from the EU, Australia, Africa, or somewhere else.)
Any modded 4WD vehicle is going to have a difficult time meeting the 20mpg requirement. Weight and drag play a significant factor. Since I swapped out the tires on my Tacoma, I have difficulty getting better than 18mpg where as 20 wasn't an issue with the lighter weight highway tires that came installed on it. Add some serious weight for travels, etc, it will continue to drop.