Rain and snow bring poor visibility at outdoor vision systems. The common used image processing methods may be not suitable for a degraded image. In this paper, a guidance image method is proposed to remove rain and snow in a single image. To removal rain and snow only using one image, a guidance image is derived from the imaging model of a raindrop or a snowflake when it is passing through an element on the CCD of the camera. Since only using this guidance image may lose some detailed information, in this paper, a refined guidance image is proposed. This refined guidance image has similar contour with the un-degraded image and also maintains the detailed information which may be lost at the guidance image. Then a removal procedure is given by the use of the refined guidance image. Some comparison results are made between different methods using the guidance image and the refined guidance image. The refined guidance image can be used to get a better removal result. Our results show that this proposed method has both good performance in rain removal and snow removal.