In this paper, we propose a general framework to investigate the average bit error rate (BER) performance of minimum mean square error (MMSE) based transceiver designs in amplify-and-forward relaying systems where all nodes are equipped with multiple antennas. Especially, we consider spatial multiplexing relaying schemes which transmit independent data streams simultaneously, which include the single stream beamforming design as a special case. Due to difficulty in finding a closed form expression of the average BER, we focus on the high signal-to-noise-ratio (SNR) analysis which generates simple analytical expressions. Then we derive new closed form expressions for the high-SNR performance of relay schemes under different design criteria, quantifying the performance in terms of a diversity gain and a coding gain. Monte-Carlo simulations show that our analytic work predicts accurately the diversity and coding gain.