Neuroimaging-based infant age prediction is important for brain development analysis but often suffers insufficient data. To address this challenge, we introduce label distribution learning (LDL), a popular machine learning paradigm focusing on the small sample problem, for infant age prediction. As directly applying LDL yields dramatically increased number of day-to-day age labels and also extremely scarce data describing each label, we propose a new strategy, called granular label distribution (GLD). Particularly, by assembling the adjacent labels to granules and designing granular distributions, GLD makes each brain MRI contribute to not only its own age but also its neighboring ages at a granule scale, which effectively keeps the information augmentation superiority of LDL and reduces the number of labels. Furthermore, to extremely augment the information supplied by the small data, we propose a novel method named granular feature distribution (GFD). GFD leverages the variability of the brain images at the same age, thus significantly increases the learning effectiveness. Moreover, deep neural network is exploited to approximate the GLD. These strategies constitute a new model: deep granular feature-label distribution learning (DGFLDL). By taking 8 types of cortical morphometric features from structural MRI as predictors, the proposed DGFLDL is validated on infant age prediction using 384 brain MRI scans from 35 to 848 days after birth. Our proposed method, approaching the mean absolute error as 36.1 days, significantly outperforms the baseline methods. Besides, the permutation importance analysis of features based on our method reveals important biomarkers of infant brain development.