In the three-dimensional sl(N) Chern-Simons higher-spin theory, we prove that the conical surplus and the black hole solution are related by the S-transformation of the modulus of the boundary torus. Then applying the modular group on a given conical surplus solution, we generate a 'SL(2,Z)' family of smooth constant solutions. We then show how these solutions are mapped into one another by coordinate transformations that act non-trivially on the homology of the boundary torus. After deriving a thermodynamics that applies to all the solutions in the 'SL(2,Z)' family, we compute their entropies and free energies, and determine how the latter transform under the modular transformations. Summing over all the modular images of the conical surplus, we write down a (tree-level) modular invariant partition function.