Altered brain development is a common feature of the neurological sequelae of complex congenital heart disease (CHD). These alterations include abnormalities in brain size and growth that begin prenatally and persist postnatally. However, the longitudinal trajectory of changes in brain volume from the prenatal to postnatal environment have not been investigated. We aimed to evaluate the trajectory of brain growth in a cohort of patients with complex CHD (n = 16) and healthy controls (n = 15) to test the hypothesis that patients with complex CHD would have smaller total brain volume (TBV) prenatally, which would become increasingly prominent by three months of age. Participants underwent fetal magnetic resonance imaging (MRI) at a mean of 32 weeks gestation, a preoperative/neonatal MRI shortly after birth, a postoperative MRI (CHD only), and a 3-month MRI to evaluate the trajectory of brain growth. Three-dimensional volumetric analysis was applied to the MRI data to measure TBV, as well as tissue-specific volumes of the cortical gray matter (CGM), white matter (WM), subcortical (deep nuclear) gray matter (SCGM), cerebellum, and cerebrospinal fluid (CSF). A random coefficients model was used to investigate longitudinal changes in TBV and demonstrated an altered trajectory of brain growth in the CHD population. The estimated slope for TBV from fetal to 3-month MRI was 11.5 cm per week for CHD infants compared to 16.7 cm per week for controls (p = 0.0002). Brain growth followed a similar trajectory for the CGM (p