In order to narrow the "semantic gap" problem between the image low-level features and high-level semantic features, this paper proposed a FSRM algorithm based on the learning theory. To compress the dimension of FSRM, the algorithm divided the image database into "related" and "irrelevant" two classes by retrieval of low-level features image. Then, adjust the weights in FSRM based on user feedback. Finally, after a finite time feedback, adjust the weights in FSRM using the learning theory FSRM algorithm, and returned the new retrieval results to the user. The experiment shows that this algorithm can express the semantics contained in the image, also can be a good description of the semantic similarity between images, therefore, the proposed algorithm has certain robustness.