The storied investing empire of American billionaire Edward S. Lampert is shrinking, the New York Times said in a report. 

According to the report, entertainment mogul David Geffen got out in 2007. Members of the billionaire household Ziff family cashed out more recently, between 2011 and earlier this year.

The NY Times said that more investors are heading for the exits, discouraged by the declining fortunes of Lampert's signature stake in Sears Holdings Corp.

Lampert's hedge fund, ESL Investments, managed over USD15 billion near its peak in 2006. It still managed USD10 billion at the end of 2011. Last year, however, its disclosed total was less than USD6 billion, the report said. 

ESL Investments on Tuesday announced that for the second consecutive year, it had reduced the size of its stake in Sears Holdings to meet investor redemptions. Its stake dropped below 50% for the first time since 2008, NY Times said. 

The report said that Lampert isn't selling any of his personal stake in the company. Although his firm announced that it had distributed 7.4 million shares worth USD411 million to investors who exited in 2013. The distribution cut his fund's stake to 48% from 55%. The remaining Sears stake of 51.6 million shares worth USD2.9 billion remains ESL's largest holding.

The action is the latest sign of investor disenchantment with Lampert. Lampert created Sears Holding in 2005 by combing Sears, Roebuck & Company and Kmart, NY Times said.