A 128(H) × 64(V) × RGB CMOS imager is integrated with region-of-interest selection, RGB-to-HSI transformation, HSI-based pixel segmentation, (36bins × 12bits)-HSI histogramming, and sum-of-absolute-difference (SAD) template matching. Thirty-two learned color templates are stored and compared to each image. The chip captures the R, G, and B images using in-pixel storage before passing the pixel content to a multiplying digital-to-analog converter (DAC) for white balancing. The DAC can also be used to pipe in images for a PC. The color processing uses a biologically inspired color opponent representation and an analog lookup table to determine the Hue (H) of each pixel. Saturation (S) is computed using a loser-take-all circuit. Intensity (I) is given by the sum of the color components. A histogram of the segments of the image, constructed by counting the number of pixels falling into 36 Hue intervals of 10 degrees, is stored on a chip and compared against the histograms of new segments using SAD comparisons. We demonstrate color-based image segmentation and object recognition with this chip. Running at 30 fps, it uses 1 mW. To our knowledge, this is the first chip that integrates imaging, color segmentation, and color-based object recognition at the focal plane.