In multiuser multiple-input multiple-output (MIMO) downlink systems, orthogonal space division multiplexing (OSDM) techniques have been studied to approach the optimum performance of dirty paper coding with low complexity. Assuming a large number of users in the multiuser system, a proper user scheduling is needed to utilize the OSDM. In addition, the performance of the chosen users can be maximized by properly configuring the numbers of data streams, or the transmission modes. In this paper, we propose a joint user and mode selection algorithm which approaches the performance of the exhaustive search with significantly lower complexity. To this end we first decompose the sum rate into the sum of the individual rates, and derive an upper bound of each individual rate. By utilizing the obtained upper bound, the user and mode subset is determined to maximize the sum rate. Comparing with other conventional low-complexity schemes, the proposed scheme requires much lower complexity especially when the number of users is large. For example, our method achieves a complexity reduction of 92% compared to the conventional scheme when the number of users is 25. Simulation results show that our method achieves more than 95% of the sum rate of the exhaustive search.