The unexpected twist in the climate have marked the retailers' long rely on sales of high-margin winter coats and boots to boost annual profits, but with the season becoming shorter and warmer, U.S. fund managers are shedding department store and apparel stocks, convinced the industry is becoming a victim of climate change.

The industry's long supply lines are one reason why apparel companies and department stores struggle to adjust to warmer winters, Haverty said. The industry is also still wedded to the idea that buyers want winter coats - which tend to have margins of around 40 percent - in early October, when summer weather often lingers.

The 2015-2016 winter was the warmest on record, prompting many U.S. fund managers to abandon apparel stocks altogether. Approximately 48 percent of U.S. actively managed equity funds have zero exposure to apparel companies, up from the 38 percent of funds that shunned apparel stocks at this time last year, according to Morningstar data.

There are signs that this season will continue to be hard on apparel makers. The average temperature in October, often seen as the kick-off to winter clothing sales, was nearly 58 degrees. That was the third-warmest October on record and the warmest since 1963, according to the National Centers for Environmental Information.

The average October temperature has warmed by 0.6 degree per decade over the last 30 years, with only September having a greater increase in average warmth, the Centers said. Fifteen of the 16 warmest years on record have been in the 21st Century.

Warm weather in late November helped push retail stocks in the S&P 500 down 2 percent over the past 5 days, while the broad market was mostly flat. Express Inc cited warm weather as hurting its sales and margins in its quarterly results announced on Thursday, and its shares skidded more than 21 percent.

Arun Daniel, a portfolio manager at J.O. Hambro Capital Management, said he has reduced his holdings of apparel companies as the cold-weather selling window becomes "tighter and tighter."

While an extremely cold winter or surprise snowstorm could lead to high sales on pent-up demand, he said, the trend of warmer winters will mean "there is going to be a price to pay for companies who are exposed to outerwear" like Columbia Sportswear Co and Burlington Stores Inc.

Companies are starting to roll out more warm-weather products. Decker's Outdoor Corp , the maker of Ugg boots, said it will expand its non-shearling styles next year to extend the brand's seasonal relevance. The company's shares are down 6.7 percent over the last 5 days, in large part because of warm-weather concerns, according to a note from Stifel Financial.

Chris Terry, a portfolio manager at Hodges Funds, said he focuses on companies that rely less on cold weather clothing. He has been taking advantage of the sell-off in shares of American Eagle Outfitters Inc to add to his position in the company, he said. The retailer's shares fell nearly 12 percent on Wednesday after it blamed declining mall traffic for its weak holiday sales forecast.

Terry says the company looks attractive over the long run because it sells mostly denim and flannel clothing, making it less reliant on cold weather markets.