Shot boundary detection (SBD) is the first and prerequisite step for content-based video indexing and retrieval. A novel shot boundary detection algorithmbased on modified SVMmodel which is improved by simulated annealing algorithm and culture algorithm is proposed. The formation of classifying model adopts culture algorithm for its strong evolutional ability and simulated annealing algorithm embedded into culture framework. Simulated annealing algorithmis employed to carry on the local search process in which individual solution changes towards neighborhood, addressing the optimal solution to belief space of culture algorithm. After updating the belief space, it directs the individual evolution, further improves the SVM model parameters. The feature indicators of brightness means, brightness variances, edge changing edges, block histogramandDCcoefficients are extracted frompixel domain and compressed domain, and then the sliding window is used to organize the features into feature vectors. Finally the video frames are classified into the cuts, gradual changes, and non-changes by SVM classifying model. The experimentswith sample video data demonstrate the algorithm is effective and robust.