To satisfy various requirement of embedded video encoder, an embedded video coding system based on TMS320C64xx DSP is designed in this paper. TMS320C64xx DSP (Digital signal processor) is the core of the coding hardware system, and FPGA (Field program gate array) as the co-processor of DSP, which can convert video data into the specified format. For real-time application, we optimize the encoder at three levels: Firstly, at algorithm level, develop some fast algorithm fitting for DSP; secondly, at system level, adjust the architecture of encoder and optimization data transfer with EDMA (Enhance Direct Memory Access); thirdly, at code level, we use linear assembly rewrite key code. In order to avoid overflow and under flow state of bit rate, software FIFO (First Input First Output) is designed, and by the status of FIFO, we can know the state of underflow and overflow of buffer. Experimental results show that the embedded video encoder can compress four channel CIF (352×288) video data real-time.