In recent years, with the increasing demand for security, intelligent video surveillance system has become more and more widely used in national security, intelligent transportation, social life and other fields Among them, the detection and tracking of moving targets are the key points and difficult parts in the intelligent video surveillance system, which mainly involves in the related knowledge of pattern recognition, image processing and intelligent control and other fields, and is also the basis of subsequent steps such as target analysis and processing. In the detection and tracking of moving targets, the existing algorithms can not adapt to the changes of the external environment. Therefore, it is necessary to study an algorithm, which is robust and not afraid of complex background environment. In this paper, the detection and tracking algorithms of moving objects in complex scenes are studied comprehensively and in detail, and the existing algorithms are improved to solve the tracking problem and make some progress.