Sensing the scattered fields of single metallic nanostructures is a crucial step towards the applications of isolated plasmonic antennas, such as for the sensing of single molecules or nanoparticles. In the past, both near- and far-field spectroscopy methods have been applied to monitor single plasmonic resonances. So far, however, these spectral-domain techniques do not yet provide the femtosecond time resolution that is needed to probe the dynamics of plasmonic fields in the time domain. Here, we introduce a time-domain technique that combines broadband Fourier-transform spectroscopy and spatial modulation spectroscopy (FT-SMS) to quantitatively measure the extinction spectra of the isolated gold nanorods with a nominal footprint of 41×10 nm². Using a phase-stable pulse pair for excitation, the technique is capable of rejecting off-resonant stray fields and providing absolute measurements of the extinction cross section. Our results indicate that the method is well suited for measuring the optical response of strongly coupled hybrid systems with high signal-to-noise ratio. It may form the basis for new approaches towards time-domain spectroscopy of single nanoantennas with few-cycle time resolution.