High-redshift galaxies with strong star formation activity are likely sources of cosmic rays. This is due to their abundance of charged seed particles coupled with widespread conditions required to accelerate them to high-energies. The cosmic ray particles interact with the matter density and radiation fields in and around the galactic environment via hadronuclear, photo-pair and photo-pion processes to produce charged and neutral pions, neutrons and protons. The effect of these interactions is for the cosmic rays to deposit energy along their propagations through their host's interstellar medium (ISM) and circum-galactic medium, and then beyond into the intergalactic medium (IGM). In this talk, we show how the energy transport and deposition by ultra high-energy cosmic rays is regulated by the evolution of the host galaxy, in particular by the development of the galactic magnetic field. We demonstrate how the magnetic field can act to contain cosmic rays and, for Supernova-driven magnetic field models, we estimate the short timescales over which this containment can occur. Despite this, we argue that cosmic rays may still readily escape from high-redshift starbursts into the intergalactic medium. This can occur both in the window before the galactic magnetic field has sufficiently developed, and also at later times due to outflows driven by star formation which can transport cosmic rays into the IGM by advection. Such cosmic rays, if unconfined, would transport substantial amounts of energy into the IGM and this could have significant consequences on the energy budget at the onset of cosmic reionization.