Just my .02, but you really can't have only one layer... so there is no one material to rely on.
For pure insulation rating, down wins.
For a breathable waterproof, goretex wins.
For base layers, anything that wicks wins. (Merino wool and silk are my choices).
For wear and tear, canvas or leather or cordura (depending on type of use).
You simply need to mix and match layers to gain the best advantages of all of these materials. Saying wool is best is like saying vitamin C is the best vitamin... you need all of them. Not just one.