A major Dutch investor lost $60 million when a Carlyle Group fund collapsed in March 2008. Now, he is paying for others to sue the U.S. private-equity firm, a high-stakes gamble that could make him hundreds of millions of dollars.

Louis Reijtenbagh, a 70-year-old former family doctor who became one of the Netherlands’s richest men by investing in art, distressed debt and other assets, has already reportedly put tens of millions of dollars behind the case. Meanwhile, Carlyle is suing him in Delaware for providing the funding for the litigation.

The situation recalls tech billionaire Peter Thiel’s backing of a successful suit this year by wrestler Hulk Hogan against Gawker Media, bankrupting the media company that had outed Mr. Thiel as gay in a 2007 blog. It is also just one of several lawsuits Mr. Reijtenbagh has funded in recent years, including in Australia where he stands to make more than $400 million from a $1.25 billion settlement between a failed conglomerate, Bell Group, and its banks.

LITIGATION FUNDING - or the practice of paying for lawsuits in exchange for a share of any settlement or award - has become an asset class in its own right, attracting money from pension funds, investment managers and family offices. But it typically involves groups of investors spreading their capital across portfolios of lawsuits rather than a single individual funding the action. Returns can be high but critics say litigation funding can lead to overzealous legal action, or to billionaires trying to silence their enemies or tie them up in costly court proceedings for years. Proponents of litigation funding say it helps cash-strapped plaintiffs get cases off the ground.

In any event, the Carlyle case, in a court on the English Channel island of Guernsey, hinges on whether Carlyle Group and several of its executives met their duties to a short-lived mortgage-bond fund called Carlyle Capital Corp. CCC failed in March 2008 when banks pulled the short-term loans it needed to buy mortgages.