Can a Quantum Nondemolition Measurement Improve the Sensitivity of an Atomic Magnetometer?

We consider the limitations due to noise (e.g., quantum projection noise and photon shot-noise) on the sensitivity of an idealized atomic magnetometer that utilizes spin squeezing induced by a continuous quantum nondemolition measurement. Such a magnetometer measures spin precession of N atomic spins by detecting optical rotation of far-detuned light. We show that for very short measurement times, the optimal sensitivity scales as N^-3/4; if strongly squeezed probe light is used, the Heisenberg limit of N^-1 scaling can be achieved. However, if the measurement time exceeds τ_rel/N^1/2 in the former case, or τ_rel/N in the latter, where τ_rel is the spin relaxation time, the scaling becomes N^-1/2, as for a standard shot-noise-limited magnetometer.