Assuming you don't exceed the safe operating angle of the u-joints when your rear axle (I assume we are talking about the rear...) is at full droop (thus avoiding the u-joints from destroying themselves), the two concerns are vibration and overall length.
If you are able to run down the freeway without any vibrations caused by the operating angle of the u-joints, count yourself as one of the lucky ones. People seem to run into problems around 3" of lift. At 3.5", I was unable to run the stock driveshaft due to vibrations. Others have been OK.
Also, check the length at full droop. Be sure your slip-joint isn't on the ragged edge of coming apart...cause that would suck.
The actual problems would be the physical annoyance of vibrations + the increased wear on u-joints and tail shaft bearings. I have not seen a tailshaft bearing fail, but have heard of people complaining of it...take that for what it is worth (not much). If the driveshaft is to short, the result could be your driveshaft coming apart at the slip joint. If this happened at any sort of speed, it could result in the destruction of the driveshaft, possibly damage to the rear differential (when the back half digs into the pavement and delivers a huge shock load to the pinion), and possible transfer case damage (caused by having the front half of the driveshaft swinging around, uncontrolled, on the tailshaft).
Of course, this is assuming that you are using a fixed yolk (think: rubicon).
