The digital transformation of the US stock market

During the 1980s, financial markets in the United States changed dramatically in the direction of a growing computerization of trading information and execution—a trend already initiated during the 1970s. As early as 1971, the National Association of Securities Dealers (NASD) created the National Association of Securities Dealers Automated Quotation (NASDAQ), a system which allowed market makers in over-the-counter stocks to post prices electronically and brokers to access such prices nationwide (Ingebretsen, 2002). Investors would still place orders via phone until the early 1980s when the small execution order (SOE) system was introduced to allow automatic order execution for traders with orders less than or equal to 1,000 shares. The launch of NASDAQ set American financial markets “on a path toward a national, integrated system, one whose base infrastructure was digital” (Cortada, 2006, page 169).

In 1976, NYSE launched the designated order turnaround (DOT) that allowed brokers to submit an order electronically from their computers directly to the specialists—hence disintermediating the role of floor brokers who took the order on the telephones and walked to the specialists’ booths to bid against each other. Other regional exchanges such as the Pacific Stock Exchange and the Philadelphia Stock Exchange also introduced similar technologies. Thus, by the mid-1980s, the computerization of clearing and settlements, market information, and trade execution was in full swing. In this context, several individual and organizations played a key role in pushing forward the computerization of financial markets. Financial journalists and academics have focused their attention on individuals such as Josh Levine, a libertarian whiz-kid who created the share-trading platform Island and sparked a sea of electronic innovations in American financial markets (MacKenzie and Pardo-Guerra 2014; Patterson 2012). Others have highlighted the opportunistic strategies used by Thomas Peterffy—the founder of the brokerage firm Interactive Brokers—to automate trading execution (Steiner, 2012).

During the 1987 ‘Black Monday’ stock market crash (October 19), some NASDAQ broker-dealer stopped processing sell orders coming through the SOE system. In response to this, the SEC forced NASDAQ to execute small buy and sell orders (up to one thousand shares) automatically. The new SOE system went live in June 1988. Unintentionally, the fact that small orders were prioritized and given near instant execution enabled the SOE system to become the hotbed of day-trading arbitrage. Traders from small brokerage firms like All-Tech Investment Securities and Datek Securities exploited the SOE system to arbitrage between buy and sell offers posted by different NASDAQ market-makers. These traders and their firms came to be known as ‘SOES bandits’ (Patterson, 2012, page 81).

The story of Island

Josh Levine began to work on Datek’s trading system during the late 1980s. In 1990, Levine created the Watcher, an automated portfolio management system keeping track of trader’s orders and positions and calculating profits and losses. The Watcher was created through a program that interpreted the data from the printer cable of NASDAQ’s Level II Workstation and transferred it to a PC. Gradually, Levine would transform the PC receiving the information from NASDAQ Workstation into a front-end trading platform that allowed Datek’s traders to buy and sell stocks directly through the Watcher, monitor multiple stocks as well as the changes in bids and offers. The Watcher outpaced NASDAQ’s clunky Level II Workstation through which traders had until then placed their orders manually. In this regard, Levine programmed in the Watcher a powerful—albeit primitive—algorithm called Monster Key that enabled traders to exploit the ‘price-time priority’ protocol, according to which orders with better prices would jump the order queue and still get the best price (Patterson, 2012, pages 89–90, 93–95).

 By the 1990s, Datek’s traders were constantly hitting the limit of orders that could be placed through the SOE system. Hence, they had to submit their orders to NASDAQ SelectNet, a system whereby orders could be submitted through Level II Workstations. However, such orders were not automated and NASDAQ’s market makers could back away from trades. Levine and Datek’s executives sought to find a new market where to trade. Besides the NYSE—which was out of touch—a potential pool was Instinet. Founded in 1967 as Institutional Network and later renamed Instinet in the 1980s when Reuters bought it, Instinet was a dark pool where trading was completely anonymous and allowed institutional investors to trade non-NYSE stocks—although originally the purpose of Institutional Network was to trade NYSE-listed stocks that institutional investors would have liked to buy or sell without the high-fee intermediation of NYSE specialists. Instinet worked through dedicated computers where investors could place their orders. However, the matching process was not automated and Instinet traders would match orders through phone calls with institutional investors. Unfortunately, Instinet’s executives turned down Datek’s offer to direct the orders of Watcher users to the Instinet pool.

In the mid-1990s, the number of Watcher users outside of Datek was growing. In general, day trading was skyrocketing as part of the tech-oriented stock market boom. In this context, Levine noticed how traders using the Watcher were often exchanging the same stocks for the same prices. Such phenomenon encouraged Levine to cut out the middleman and develop a program through which traders to execute trades directly through the Watcher. Such trades would be reported to the national consolidated tape automatically. Launched in November 1995, the system was called Jump Trades. Within a year Jump Trades would develop into Island, the first share-trading venue that facilitated fully automated trading (MacKenzie and Pardo-Guerra, 2014).

Island was officially launched on February 16, 1996, as a computer program that matched buy and sell orders. Island’s electronic feed called ITCH disseminated all the trading-related information and the order book was available in a computer-readable way to allow algorithms to track orders and react instantly. Another protocol called OUCH offered investor a fast way to connect to Island and execute orders (Patterson, 2012, page 119). Trading on Island was cheaper compared to NASDAQ. Island charged $1 per trade while NASDAQ charged $2.50. Those orders that were not matched on Island, where sometimes executed on Nasdaq SelectNet. Soon after Island started to operate, Automated Trading Desk—a pioneering algorithmic-trading company specializing in high-frequency trading (MacKenzie, 2017)—began to send its buy and sell orders on Island. Successively Renaissance Technologies and other algorithmic trading outfits like Timber Hill and Getco joined the Island platform too (Patterson, 2012, pages 117–124). Crucially, during these years, together with Island, a group of individuals and organizations were playing a key role in transforming American financial markets into a space for trading algorithms. These market players were “as adept writing complex code, soldering semiconductor chips, and employing math as they were negotiating labyrinthine market structures” (Steiner, 2012, page 17).

Electronic communications networks (ECN) and the rise of algorithmic trading

A series of regulatory changes supported the transformation of American financial markets towards more automation. Beginning with Regulation OHR in 1997, the SEC introduced rules that allowed investors to access information about best prices. Following the Justice Department’s investigation into NASDAQ market makers colluding in fixing stock prices (Dutta and Madhavan, 1997), SEC proposed a set of ‘order-handling’ rules in which NASDAQ market makers were forced to post quotes from competing firms alongside their quotes—hence, bringing more competition and transparency to the market. The SEC designated also the trading entity called electronic communications network (ECN), which is a computerized system for matching limit orders outside of traditional stock exchanges and charging very small transaction fees (SEC, 1996). Bids and offers that did not match internally on ECNs would appear on the NASDAQ system. As a result, Instinet was classified as an ECN and therefore its quotes would be available to all investors and not only to those who owned Instinet terminals. Island met the requirements and became an ECN too, together with Bloomberg Tradebook and a new Chicago-based firm called Archipelago (Patterson, 2012, pages 138, 144).

The rise of ECN transformed the NASDAQ marketplace into an electronic system increasingly open to algorithmic trading. As Patterson (2012, pages 128–129) summarizes, two factors played a key role in this transformation. First, traders had been using technology for years, but the market infrastructure for algorithmic trading was not in place yet. Firms like Hull Trading, Renaissance Technologies and ATD used AI to develop trading models, but AI was not involved in actual trading activities. Without human market makers in the middle, ECN now provided the market plumbing for AI algorithms to change trading strategies instantly depending on market movements. Second, ECNs like Island offered a large amount of market data that was coded in a way that computers could read it. Computers could now crunch all this information and use it to develop automated trading strategies.

Rebate trading

Besides state-of-the-art technology such as the use of ‘distributed computing’ to organize its servers across several systems (Sanburn, 2002), Island ECN brought to the marketplace a unique microeconomic feature that altered the American stock market in the years to come by encouraging the rise of ‘rebate traders’: the maker-taker model. To incentivize big players to use Island, the latter paid firms 1 cent for every hundred-share order. In other words, Island paid the ‘take’ fee, while large investors earned the ‘make’ fee (Patterson, 2012, pages 157–159). Moreover, Island supported policymakers’ and regulators’ call for the decimalization of trading. In early July 2000, Island became the first marketplace in the United States to trade stocks in 1-cent increments. Decimalization reduced the bid-ask spread compared to the fractional trading system which was previously in place. Accordingly, it drove many market makers out of business (Patterson, 2012, pages 175–177).

By the early 2000s, electronic trading was in full swing. ECNs offered the market infrastructure to trade NASDAQ’s stocks faster, cheap and in an automated manner without human market makers. A range of algorithmic trading firms were scouting markets for profitable trading opportunities. Firms like Renaissance Technologies, D. E. Shaw, Timber Hill, and Automated Trading Desk were at the forefront of designing automated trading programs. Other algorithmic trading firms were also emerging that specialized in high-frequency trading blasting out orders into the market every second. Three firms stood out in particular: Texas-based RGM Advisors, Chicago’s Getco, and Kansas City’s Tradebot. As these firms flooded ECNs with orders, Island courted them to use its platform while at the same time also trying to divert flows of ‘dumb money’ from retail brokerage firms such as E*Trade, Ameritrade and Charles Schwab through the practice called ‘payment for order flow’ (Patterson, 2012, pages 182–187).

High-frequency trading

 In the early 2000s, high-frequency trading firms like Getco and Tradebot became the ‘new market makers of the digital age’ (Patterson, 2012, pages 182–187). They flooded Island and other ECNs with orders. For example, Dave Cummings’ automated strategy at Tradebot was to post a two-side quote (bid and offer) for stocks and exchange-traded funds, while monitoring S&P 500 futures for clues about general market moves. The bid-offer spread was small but profitable if hundreds of thousands of orders were submitted per day—this is far more than the first-generation of automated trading firms did. ECNs—especially Island as the only platform that could handle massive trading volumes—loved this high flow of trades because it provided liquidity and fees (Patterson, 2012, page 193). By 2002, Tradebot and Getco accounted for about 10 per cent of trading in NASDAQ stocks. They went in and out stocks in less than a second. Spreads for the most heavily traded stocks narrowed to pennies (e.g., a single cent), putting human market makers on NASDAQ out of business. In a word, high-frequency trading firms with their high-octane automated computer-based strategies and the fast-automated electronic infrastructure provided by ECNs ‘altered the very structure of the market’ (Patterson, 2012, page 195).

Colocation

In different ways depending on the organizational cultures, all automated trading firms recruited AI programmers skilled in machine learning techniques. These programs monitored reams of market data and adapt trading strategies dynamically based on a range of circumstances (Patterson, 2012, page 196). As these firms pushed their high-frequency trading strategies to the limits, they hit the physical barrier of the speed of light. Cummings noticed how Tradebot was outbid by Automated Trading Desk because the latter had placed its computers close to Island’s racks. Tradebot was instead based a thousand miles away in Kansas City. Speed became the key concern for high-frequency traders and any possible technological trick could be exploited in order to reduce the latency at which traders would receive and send their order information.

In this regard, Island and other ECNs developed the practice of ‘colocation’ whereby traders would put their own servers as closest as possible to the servers of ECNs, which charged a fee for such service (Patterson, 2012, pages 199–200). The marketplace evolved into a context where ECNs and exchanges catered more and more to the needs of high-frequency traders and automated trading firms. As Patterson (2012, pages 204–205) aptly described, ‘bots were taking control, pushing their favorite trading networks for more capacity, more speed, more creative ways to make money […] high-speed firms worked hand in hand with the trading networks to create exotic order types that would behave in very specific ways […] Archipelago proved to be a master at this game, which kept the speedsters coming back for more.’

 By the mid-2005, Tradebot had developed a ‘latency arbitrage’ strategy between the price of a stock in the‘lit’ markets and dark pools. Dark pools priced stocks based on an electronic feed called the Securities Information Processor (SIP), which was slow compared to, say, Tradebot’s co-located direct line into Island’s ITCH feed. Hence, when the price of a stock changed in the lit market, Tradebot obtained that information faster than others. The firm could then exploit this advantage in the dark pool where the new price of that given stock updated a bit later due to the slow SIP feed. Of course, these trades were done thousands of times in order for the narrow spreads to add up significantly. Regulators saw these trades as ‘making the market more efficient’ (Patterson, 2012, pages 202–203).

High-frequency traders as the new entrenched middlemen

By the mid-2000s, four automated trading firms—Automated Trading Desk, Renaissance Technologies, Tradebot, and Getco—accounted for about 25 per cent of all stock trading in the United States. Three of these (Tradebot, Getco, Automated Trading Desk) specialized in high-frequency trading. They co-located their servers close to those of ECNs and worked with ECNs to customize the market infrastructure the specific plumbing of their systems in line with their trading needs. ECNs complied with these demands to route liquidity to their systems and earn fees. New high-frequency trading firms were emerging such as Lime Brokerage, Hudson River Trading and Sun Trading. Traditional investment banks (e.g., Goldman Sachs, Deutsche Bank, JPMorgan) also got into the game. A new paradigm of trading had emerged, one which was based on automated AI techniques and fast trading execution, which by the end of the 2000s was measured in micro-seconds—in other words, a race to zero in information latency close to the speed of light (Patterson, 2012, pages 205–206). ‘A new Wall Street elite was emerging, a wealthy Technorati that owned the massive computers and superfast cables that shaved precious microseconds off orders. Levine’s dream of a market without middlemen had been turned on its head. Technology had created a new entrenched middleman: the high-frequency traders (Patterson, 2012, page 207).

Some questionable practices began to emerge too, like ‘spoofing’ phantom orders that high-frequency traders cancelled well before execution so that could create the illusion of buy or sell momentum in a stock. Furthermore, dark pools of liquidity like Liquidnet, Credit Suisse’s Crossfinder, and Goldman Sachs’ Sigma X were emerging as key electronic markets where large institutional investors could trade big blocks of shares without being anticipated by other faster traders. However, high-frequency traders found their ways into dark pools too through practices such as ‘pinging’ with orders that were quickly cancelled and also by using AI pattern-recognition methods (Patterson, 2012, pages 207–208).

Modernizing NASDAQ and NYSE

As the marketplace changed, high-speed traders never targeted the actual NASDAQ and NYSE. In the main, the presence of human market makers made high-frequency trading impossible. However, both exchanges did not want to lose relevance in the new game. In particular, NASDAQ noticed how ECNs reported back huge volumes of trades (e.g., Island did that) but those trades were not actually happening on the actual system—hence, NASDAQ was losing money (Patterson, 2012, page 212). In this context, both markets initiated a controversial process of digital transformation. NASDAQ tried to launch an ‘ECN killer’ project called SuperMontage, an electronic ‘super pool’ aggregating all best bids and offers across the market, including Instinet and more recent ECNs like Island and Archipelago. SuperMontage would however give preference to NASDAQ quotes (Patterson, 2012, page 213). The project would eventually be scrapped when NASDAQ would agree to buy Island’s matching engine from Instinet in 2005 (Patterson, 2012, page 238).

 Island nonetheless prepared to operate outside SuperMontage by initially trying to join forces with NYSE. Matt Andresen (Island’s CEO) tried to print its trades on NYSE instead of NASDAQ, but NYSE’s CEO Dick Grasso was not interested. Thus, Island struck a deal with the Cincinnati Stock Exchange to report its trades there. Cincinnati Stock Exchange had been the first regional exchange to go entirely electronic but was on life support by the early 2000s (Patterson, 2012, page 218). Between March and June 2002, venture capital investors that were majority owners of Island decided to sell. Although Archipelago would have been a strategically fit partner to compete in an ECN market that was becoming tough, Island ultimately signed a merger deal with Instinet (Patterson, 2012, pages 221–222).

NYSE’s digital transformation occurred differently and through the Archipelago platform. Archipelago went public in 2004. A year later, in April 2005, NYSE announced its plans to reorganize as a publicly traded company and at the same time merge with Archipelago. NYSE Group stocks began trading on March 8, 2006. In the NYSE-Archipelago deal, Goldman Sachs was an adviser to both parties. The investment bank also owned 15 per cent stake in Archipelago and a block of seats at NYSE through ownership of the specialist firm Spear, Leeds & Kellogg (Patterson, 2012, pages 234–237). Merging the two technological systems of Archipelago and NYSE became a hard task as the NYSE’s one was antiquated. For example, orders that flowed through the DOT system were still matched frenetically by human typists. Above all, the NYSE’s old guard resisted to the electronic trading transformation (Patterson, 2012, pages 241–242).

Thus, the new structure of American financial markets consolidated with NYSE and NASDAQ taking over Archipelago and Instinet’s computer system (previously Island). In the meantime, regulators introduced Regulation National Market System in 2007, according to which any buy and sell order had to be routed to the venue that has the best price. Crucially, Reg NMS mandated that ECNs could bypass exchanges where humans executed trades in about 20 seconds. This meant that investors could now buy and sell NYSE-listed stocks also through other exchanges and ECNs. High-speed traders could now finally access the most heavily traded shares of blue-chip companies such as IBM and General Electric (Patterson, 2012, page 245).

After the 2010 Flash Crash

Following the May 6, 2010 Flash Crash (Borch, 2016), little was done to address the problem that caused it. Trading was fragmented across more than fifty venues, many of which are dark pools. Liquidity was maintained by automated trading algorithms and high-frequency traders. Novel methods were created to turbocharge microprocessors, such as the ‘overclocking’ practice of using nitrogen-cooling systems—a technology invented for top-end videogame players. Furthermore, exchanges were establishing data centers across the world with state-of-the-art computers matching orders and offering co-location services to automated trading firms. In 2010, NYSE erected its data center in Mahwah, New Jersey; The Chicago Mercantile Exchange established its data center in Aurora, Illinois. Numerous other data centers were established nearby London, in Kowloon, China, Mumbai, Singapore, Melbourne, São Paulo and elsewhere (Patterson, 2012, pages 281–283). In other words, automated trading expanded globally with trading firms and exchanges (e.g., BATS Chi-X Europe) internationalizing. Investments in new communication technologies (e.g., new fiber-optic cables, microwaves plates) reduced information latency between trading data centers down to ten milliseconds. At the same time, SEC planned to launch the Consolidated Audit Trail (CAT), a computer capable of capturing and analyzing market orders to avoid manipulations and crashes like the May 2010 Flash Crash (Patterson, 2012, pages 287–291).

High-frequency traders as the new designated market makers

The market changed to the point where high-frequency traders replaced NYSE’s designated market makers (DMM, previously known as specialists). Getco became one of the largest specialists at the NYSE together with Goldman Sachs and Knight Capital. Only four specialist firms existed compared to thirty-five firms that run the floor a decade earlier. Currently, there are only three: Citadel Securities, GTS (Global Trading Systems) Securities, and Virtu Financial, IMC Financial Markets, Jane Street. Getco combined state-of-the-art programming tools (including the visual programming language called Kodu used to develop Xbox video games), advanced computing technology (e.g., Nvidia graphics processing units), deep knowledge of market plumbing and cutting-edge fast communication access (e.g., InfiniBand). Getco had offices all over the world and growing lobbying power in Washington (Patterson, 2012, pages 284–285). Getco and Knight Capital merged and formed KCG Holdings in 2013, which was then acquired by Virtu in 2017.