Nasdaq OMX Group (NDAQ.O) said on Friday its experiment in lowering exchange fees and rebates in 14 stocks has so far led to lower market share in those names on its exchange as many electronic market making firms sought higher rebates elsewhere.

The experiment, which began on Feb. 2 and will run for at least four months, includes 14 stocks that had their associated fees lowered on Nasdaq's exchange to 5 cents per 100 shares from 30 cents per 100 shares, and their rebates cut to 4 cents per 100 shares from 29 cents per 100 shares.

Stock exchanges pay rebates as incentives to market makers that post resting orders on both sides of the best bid and offer in order to help get trades done for other market participants. When a broker sends an order to the exchange to be immediately executed and it hits a resting order, the broker is charged a fee that subsidizes the rebate.

But many retail brokers, fund companies and institutional investors have complained that exchange fees are too high, forcing them to send their orders to private, broker-run trading venues that charge less than public exchanges.

Nasdaq said in a report detailing the first month of its pricing experiment that it lost market share in all but two of the 14 stocks as market makers provided 16.8 percent fewer resting orders in those names overall on Nasdaq's exchange. The shares also trade on other exchanges, such as BATS Global Markets and Intercontinental Exchange's (ICE.N) NYSE unit, which are not part of the Nasdaq initiative.

When the rebate-seeking firms pulled back from Nasdaq, Investment Technology Group (ITG.N), a brokerage that caters to institutional investors, said it began receiving better order executions at Nasdaq and it actually increased the amount of orders for the test stocks it sent to the exchange.

ITG said its order routing algorithms look for the markets that fill their orders the fastest and most completely, and do not take fees and rebates into account.

"The brokerage community, if interested in improving performance on behalf of their clients, should pay heed," ITG said in a note to clients on Friday.

Nasdaq plans to release data each month during the experiment to show how market participants are adapting to the change in pricing.