I think it should be fine if your battery and charging system is in good shape. I used to take out my fridge after every trip and clean it out. But I found I was using it almost every weekend for errands, so I started leaving it in the car and just plugging it in on the weekends. Eventually, I got tired of packing and unpacking everything and decided to leave it plugged into my car full time now. I drive about an hour or so each day just commuting and running errands, and haven't had a problem since i started doing it a few months ago. This is with and edgestar fridge which isn't the most efficient, and during the summer when we were having 100+ degree days. I think I tripped the low voltage alarm one time, but it may have been because if I'm working on the car I tend to leave the interior lights and radio on for hours at a time, and a few times i may have left the interior light on overnight. Still never had a problem starting the car though.
I did read that the alternator probably won't give the batteries a full charge on short daily trips like that, so I've just started to plug in a battery charger to top off the battery over the weekend. This is the first time I've ever done it, so I can't really say if it was needed or not, but I'll probably do that once a month just for safety.