The most expensive performance bug in history
· 8 min
On August 1, 2012, Knight Capital Group, one of the largest market makers on the New York Stock Exchange at the time, deployed new trading software shortly before the market opened. Forty-five minutes later, the company nearly went bankrupt.
A deploy to 8 servers, but only 7 got the new code
The deploy went out to 8 production servers, but one of them didn't actually receive the update, it kept running old code tied to a feature called "Power Peg" that had been disabled and unused for roughly eight years.
A reused flag woke the dead code back up
The new software reused a flag that the old, dormant code on that eighth server still interpreted as a trigger for its own obsolete logic. As soon as the market opened and real order flow started arriving, that one server began executing the old routine against live orders.
45 minutes with no safeguard catching it in time
The dead code repeatedly bought and sold shares of roughly 150 companies in rapid, erratic succession, and it took about 45 minutes for the team to identify the source and stop it, no automated safeguard in Knight's own systems caught the problem fast enough.
The cost: close to $440 million pre-tax
Knight Capital reported pre-tax losses of roughly $440 million from the incident, enough to put the firm's solvency at risk and lead to its acquisition within months, ending its run as an independent company.
The lesson that outlives the specific case
The failure wasn't a logic bug in the new code, it was a deploy process that allowed inconsistent state across a fleet (one server out of eight running different code than the rest), with no verification step catching that mismatch before it touched real traffic. That class of failure has nothing to do with financial markets specifically, and everything to do with how deploys get verified.