摘要:Selective detection of signal over noise is essential to measurement and signal processing. Time-frequency filtering has been the standard approach for the optimal detection of non-stationary signals. However, there is a fundamental tradeoff between the signal detection efficiency and the amount of undesirable noise detected simultaneously, which restricts its uses under weak signal yet strong noise conditions. Here, we demonstrate quantum parametric mode sorting based on nonlinear optics at the edge of phase matching to improve the tradeoff. By tailoring the nonlinear process in a commercial lithium-niobate waveguide through optical arbitrary waveform generation, we demonstrate highly selective detection of picosecond signals overlapping temporally and spectrally but in orthogonal time-frequency modes as well as against broadband noise, with performance well exceeding the theoretical limit of the optimized time-frequency filtering. We also verify that our device does not introduce any significant quantum noise to the detected signal and demonstrate faithful detection of pico-second single photons. Together, these results point to unexplored opportunities in measurement and signal processing under challenging conditions, such as photon-starving quantum applications.