I work out in it all the time. I was wearing a hoody and a t shirt my son had on two hoodies and a jacket and was freezing I was fine
My fashion thermometer for the winter:
above 50 - shorts and Tshirt
50 - long pants and heavy Tshirt
40 - sweat pants and tops
30 - add gloves and a hat.
About the only time we're in the "cold" is walking to and from the car, usually less than 30 seconds. Add another minute to unload the car. Doesn't seem worthwhile to be bundled up and restricted with coats inside the car for the short time we're out of it.