In order to resolve the "semantic gap" problem between the image low-level features and high-level semantic features, this paper proposed a FSRM algorithm based on the learning theory. To compress the dimension of FSRM, the algorithm divided the image database into "related" and "irrelevant" two classes by retrieval of low-level features image. Then, the weights were adjusted in FSRM based on user feedback. Finally, after a finite time feedback, the weights were adjusted in FSRM using the learning theory FSRM algorithm, and the new retrieval results were returned to the user. The experiment shows that this algorithm can express the semantics contained in the image and can present a good description of the semantic similarity between images; therefore, the proposed algorithm has certain significance.