Real estate brokerage firms are adapting to fierce agent competition, lower brokerage valuation and a changing M&A landscape.