There are still plenty of nomadic people's in the world who 'camp' all the time, so there is no too young. Just keep them bundled up so they don't get cold. They'll succumb before a larger child or an adult, purely based on body mass.
It amazes me how people forget where we came from. I almost never take a tent camping and most people are astounded at the fact of just sleeping on the ground.
I take my kids camping all the time. Our first trip with my youngest was when he was 3 months old.