Chaotic systems combined with Gauss mixture model to achieve synchronic control detection is often used in traditional network attack detection methods, and the effect of detection is good when the attack signal to be detected has Gauss linear features. As the cyber-attack signals develop toward nonlinear random sequence, traditional detection models cannot achieve effective attack detection. A potential mining algorithm of average mutual information feature based on Rossle chaotic model is proposed, and according to the nonlinearity feature solution of the mutual information, the realization of effective detection of cyber-attack signals with stochastic nonlinear characteristics is obtained. On the basis of the foundation model of Rossle chaotic system, design an adaptive cascade notch filter which can remove multiple known interference frequency with the minimum mean square error criterion, realize the filtering pretreatment of the attack signals, extracting of Rossle nonlinear mutual information feature of the network data flow to be detected, and accomplish feature mining and detection of cyber-attack signals. Simulation results show that the detection performance is improved obviously, and the probability of detection reaches 98.7%, which appears superior performance of detection and value of network security defense.