Ground-state coherence versus orientation: competing mechanisms for light-induced magnetic self-organization in cold atoms

We investigate the interplay between two mechanisms for magnetic self-organization in a cloud of cold rubidium atoms subjected to a retroreflected laser beam. The transition between two different phases, one linked to a spontaneous spatial modulation of the Δm=2 ground-state coherence and the other to that of the magnetic orientation (spin), can be induced by tuning either a weak transverse magnetic field or the laser intensity. We observe both first- and second-order transitions depending on the presence of the magnetic field. The experimental observations are successfully compared to extended numerical simulations based on a spin-1 model.