Precise quantification of brain tissue micro-architecture using diffusion MRI is hampered by the conflation of diffusion-attenuated signals from micro-environments that can be orientationally heterogeneous due to complex fiber configurations, such as crossing, fanning, and bending, and compartmentally heterogeneous due to variability in tissue organization. In this paper, we introduce a method, called Spherical Mean Spectrum Imaging (SMSI), for quantification of tissue microstructure. SMSI does not assume a fixed number of compartments, but characterizes the signal as a spectrum of fine- to coarse-scale diffusion processes. Using SMSI, multiple orientation distribution invariant indices can be computed, allowing for example the quantification of neurite density, microscopic fractional anisotropy per-axon axial/radial diffusivity, and free/restricted isotropic diffusivity. We show that SMSI is fast, accurate, and can overcome biases in state-of-the-art microstructure models. We demonstrate its application in probing microstructural changes in the baby brain during the first two years of life.