The human brain develops very rapidly in the first years of life, resulting in significant changes in water diffusion anisotropy. Developmental changes pose significant challenges to longitudinally consistent white matter tractography. In this paper, we will introduce a method to harmonize infant diffusion MRI data longitudinally across time. Specifically, we harmonize diffusion MRI data collected at an earlier time point to data collected at a later time point. This will promote longitudinal consistency and allow sharpening of fiber orientation distribution functions (ODFs) based on information available at the later time point. For this purpose, we will introduce an approach that is based on the method of moments, which allows harmonization to be performed directly on the diffusion-attenuated signal without the need to fit any diffusion models to the data. Given two diffusion MRI datasets, our method harmonizes them voxel-wise using well-behaving mapping functions (i.e., monotonic, diffeomorphic, etc.), parameters of which are determined by matching the spherical moments (i.e., mean, variance, skewness, etc.) of signal measurements on each shell. The mapping functions we use is isotropic and does not introduce new orientations that are not already in the original data. Our analysis indicates that longitudinal harmonization sharpens ODFs and improves tractography in infant diffusion MRI.