By SCOTT PATTERSON and JENNY STRASBURG Haim Bodek was a Wall Street insider at Goldman Sachs GS -0.31% and UBS UBSN.VX -0.25% before launching his own trading firm. Now he is taking on the financial establishment that spawned him. Mr. Bodek approached the Securities and Exchange Commission last year alleging that stock exchanges, in a race for more revenue, had worked with rapid-fire trading firms to give them an unfair edge over everyday investors. He became convinced exchanges were providing such an edge after he says he was offered one himself when he ran a high-speed trading firmâa way to place orders that can be filled ahead of others placed earlier. The key: a kind of order called "Hide Not Slide." Jesse Neider for The Wall Street Journal Haim Bodek told the SEC what he had learned about order types. His sword was a gift from a friend who said he would need it in business. The encounter set off an odyssey for Mr. Bodek that has fueled a sweeping SEC inquiry into the activities of sophisticated trading firms and stock-exchange operatorsâincluding Nasdaq OMX Group Inc., NDAQ +0.97% NYSE Euronext, NYX +0.65% Direct Edge Holdings LLC and BATS Global Marketsâaccording to exchange and other officials, and lawyers with knowledge of the inquiry. At issue is whether exchanges sometimes allow high-speed trading firms to trade ahead of less-sophisticated investors, potentially disadvantaging them and violating regulatory rules. The inclusion of Nasdaq and NYSE in this high-profile probe, and the role Mr. Bodek has played in it, haven't previously been reported. High-frequency traders use computer algorithms to place blizzards of buy and sell ordersâmany of which they instantly cancelâin an effort to detect and exploit the tiniest shifts in demand for stocks. This kind of trading now generates some two-thirds of all share volume on U.S. markets. Critics worry that the flood of orders driven not by investors evaluating stocks but simply by computers makes markets increasingly vulnerable to incidents like the May 2010 "flash crash," when stocks plunged hundreds of points for no evident reason, and this summer's mess at Knight Capital Group Inc., KCG +1.07% when a trade-coding problem caused a $440 million loss for the brokerage firm. The upshot, say critics, is a stock market grown so complex and opaque that many investors have little idea how it operates, and are made wary by the repeated glitches. Any notion that the computer-armed traders are being given an unfair edge could only heighten investor concerns. Just last week, the SEC fined NYSE Euronext for providing data slightly faster to certain traders than to a public database, in allegations the NYSE settled without admitting or denying them. The ongoing probe comes at a sensitive time for Nasdaq, which has acknowledged that technical problems bungled the Facebook Inc. FB +0.27% initial public offering in May. The sloppy IPO appears to have worsened a continuing exodus of cash from stock mutual funds. Representatives of Nasdaq and the other stock exchanges all declined to comment on the inquiry. Exchange officials don't deny making available certain advantages, like data feeds with detailed information about trades, that the high-frequency traders can use. The exchanges' position is that these are fully disclosed; they can be used by anyone with the right hardware and technical savvy; and they ultimately benefit all investors because by pulling in a higher volume of orders, they make it possible to buy and sell more easily and at better prices. Many high-frequency trading firms similarly say that their activity in constantly buying stocks from investors who want to sell, and selling to investors who want to buy, helps keep the markets running smoothly. But it involves risk, and advantages such as detailed streams of trade data help compensate them for the risk, the firms and the stock exchanges say. In papers filed with the SEC, Mr. Bodek took aim not at the data streams but at the way orders from high-frequency traders work. He focused on "order types"âprogrammed commands traders use to tell exchanges how to handle their bids and their offers to sell. Hundreds of order-type options are available, which translate to thousands of variations because they behave differently depending on how an investor's trading programs are coded. One thing investigators want to know is whether stock exchanges have worked with programmers or officials at high-speed trading firms to create order types that, when used with certain trading algorithms, can be unfair to less-savvy investors, people familiar with the probe say. Looking closely at order types "is the first step for regulators to understand the fundamental interactions in the marketplace, which have literally gone unchecked and untested" in recent years, says Christopher Nagy, a consultant who formerly handled the routing of orders for brokerage firm TD Ameritrade Holding Corp., AMTD -2.04% which serves individual investors. A stock exchange that wants to offer a new order type must get it approved by the SEC, in a process that entails public notice and comment. That makes the probe tricky, because the SEC is examining the effects of procedures the agency itself approved. Among questions the regulators want to answer is whether exchanges have at times misled them in seeking approval for certain order types or mischaracterized to investors how the orders work, those familiar with the probe say. A spokesman for the SEC said its staff "is in daily contact with market participants on a variety of market structure issues, including new ways exchange order types may be used by sophisticated firms in their evolving trading strategies. If the staff discovers that exchange order types are being used in ways that present concerns under the Exchange Act, it will develop an appropriate regulatory response or recommend enforcement action, as warranted." The spokesman didn't comment on Mr. Bodek's involvement in its inquiry. Mr. Bodek could gain from passing along information. The issues he raised were taken up by the SEC under a new whistleblower program that became effective last year. If the SEC were to collect a fine in an enforcement case growing out of the inquiryâthere aren't any cases at this pointâMr. Bodek might snare up to 30% of it. A lawyer for Mr. Bodek described him as a Wall Street veteran who is "sounding an alarm." Investors, said the lawyer, Shayne Stevenson of Seattle law firm Hagens Berman LLP, "needed to know the allegations he was bringing forward."