I disconnected my sway bar years ago because I was wheeling a lot, and it helped tremendously on the trails. I got used to the feel of driving on the road with it off pretty fast. I have 650 lbs springs up front (softer than most guys with 1st gen Tundras), stiffer rear springs, and my rear shocks have adjustable dampening. With the dampening increased in the rear the truck is more stable, close to if I had the sway bar attached. I personally like the additional body roll without the sway bar, so I typically leave the dampening soft on the rear. I've driven it this way for years and am used to it.
However, in an emergency situation the lack of sway bar can become an issue. My wife is not used to the feel of no sway bar, and when she drove my truck to the airport a year ago she had to quickly swerve when someone jumped into her lane and cut her off (no time for brake). Within a matter of seconds she smashed the driver's side into the concrete barrier, then got thrown the opposite direction into a semi trailer. The bed was crushed and I have a very expensive plate bumper that is now bent. Not to mention the rear of the frame rails are tweaked. But I'm just glad it wasn't worse. She could have easily rolled or caused injury to someone else.
Having no sway bar is great for wheeling. But it is not without risk. Just understand that if you choose to run with no sway bar, you are increasing your risk. The ideal situation is to have quick disconnects on the sway bar, so you can easily disconnect when you are on the trails. Some trucks make this easy, some do not. I have "quick" disconnects for my Tundra, but they are not quick. I still have to lay in the dirt and remove several nuts. Hence the reason I mostly leave it disconnected. But if my wife, or anyone else, is going to drive the truck I reconnect them.