Construction of brain atlases is generally carried out using a two-step procedure involving registering a population of images to a common space and then fusing the aligned images to form an atlas. In practice, image registration is not perfect and simple averaging of the images will blur structures and cause artifacts. In diffusion MRI, this is further complicated by the possibility of within-voxel fiber misalignment due to natural inter-subject orientation dispersion. In this paper, we propose a method to improve the construction of diffusion atlases in light of inter-subject fiber dispersion. Our method involves a novel q-space (i.e., wavevector space) patch matching mechanism that is incorporated in a mean shift algorithm to seek the most probable signal at each point in q-space. Our method relies on the fact that the mean shift algorithm is a mode seeking algorithm that converges to the mode of a distribution and is hence robustness to outliers. Our method is therefore in effect seeking the most probable signal profile at each voxel given a distribution of profiles. Experimental results confirm that our method yields cleaner fiber orientation distribution functions with less artifacts caused by dispersion.