Black Friday was first associated with the collapse of the gold market on September 24, 1869. Two men conspired to cheat Wall Street investors by buying up as much gold as possible to drive up the price and then sell to net a huge profit. The plan fell through when President Ulysses S. Grant got wind of their interest in the gold market and sold $4 million worth of government gold. This caused the price of gold to plummet and investors sold off their holdings.
Source: Library of Congress