My wife and I used 2 basic rules before we said "no" to a young one- is it a safety issue or is it a moral issue? If the answer to both was no, then we usually let stuff go. As a result , we have precious pictures of kids hiking in oneseys on Angels Landing, camping meals consisting of marshmellow peanut butter omelets, and memories of epic hours' long air soft gun battles in Nevada ghost towns. Many conversations now start with "remember the time you let me ....."
The point, enjoy EVERY MOMENT with them, and choose your battles carefully, because most of the stuff we make issues of dont not really matter.