The high speed test sets load and psi, and varies speed (increasing). Temperature is usually the limiting factor at ~200F as the rubber compounds deteriorate in performance above that.
The main thing generating heat is hysteresis in the sidewalls and tread where it meets the ground. So you should be safe on the road at low psi so long as the tire is no more squashed than it was at the max load.
When airing down offroad you're typically nowhere near the speed rating of your tire, so the heat generated (which is a little above linear with speed) will also be less. The cooling component due to airflow will also be less (varies less than linearly with speed), but the conduction to the ground (doesn't vary with speed) will be greater, due to more surface contact. Since I can't find any reliable analysis or testing of this... I dunno...
If you're concerned about it, measure your tire temperature with a IR deal.