No rock crawling, no Rubicon
I'm leaning toward the Sport. I've owned a '91 Trooper on 32s, full sized Dodge 1/2 ton pickup, a fully locked CJ-7 on full width Chevy axles set up for rock crawling, a '08 JK Unlimited Sport w/ a 2.5" BB, and now back to a '01 Cherokee on 33's w/ one rear ARB locker. There really is a difference between rock crawling locked up with a 4:1 transfer case where you're front tires have to crawl up vertical rock ledges 4 feet tall vs a "trail run", or exploration run. The former is all about driving ability where scenery is a slightly noticed bonus and damage is more likely the center of attention. The latter is about the discovery process and overcoming some obstacles with the prize being successfully experiencing the scenery and unknown.
I only took my '08 JK Sport on a couple trips in the Colorado mountains. We did no real rock crawling but I was impressed at the traction control's performance - even with stock tires. For driving dirt roads with a few basket ball sized rocks, the Sport will do just fine. However, you WILL and I mean that emphatically, need to replace the weak stock shocks and springs. The stock set up bottoms out continually and it starts with immediate shock fade.
One other point to consider. The Rubi's 4:1 transfer case is a gem, but you better be the tail gunner in a large group as the crawl ratio will have you lagging behind the other rigs if they have more sain crawl ratios (this assumes the Toyotas in your group are running dual cases). The 6 speed manual's first gear is surprisingly low anyway.
Also, the perception is the JK is much larger than the TJs and XJs, but that's mostly the external demensons. I'm more comfortable in my XJ than I was in the JK. I've taken my CJ-7 and my 1/2 Dodge on the full Mohave Road (about 130 miles). I was alot more comfortable in the Dodge and loved the extra room for goodies. The CJ wasn't even remotely challenged by the terrain and did the deepest sand by the lava tubes in 4hi. It was challenged when trying to keep up with longer wheelbase rigs at higher speeds on the dirt roads as it loved to fish tail. Oh, I also had no passenger seat in the CJ to make extra room.
I'd get the Sport, then spend your savings on real springs and shocks, and maybe (maybe) a new front axle. Gears are a given though. If you're going to "graduate" to technical rock crawling where you really need the 4:1, then go used Rubi, and upgrade. If you stay below 35 inch tires, the Dana 30 will be fine.
One last thing. If you're a tall person - beware of the older JK's center dash board. They stick out and will beat your right knee to a pulp. Also, get power windows!! The manual crank handle will beat your left knee silly. I haven't seen a '12 but hope Fiat fixed this design flaw.
Happy hunting.