To detect and classify vehicles in omnidirectional videos, an approach based on the shape (silhouette) of the moving object obtained by background subtraction is proposed. Different from other shape based classification techniques, the information available in multiple frames of the video is exploited. Two different approaches were investigated for this purpose. One is combining silhouettes extracted from a sequence of frames to create an average silhouette, the other is making individual decisions for all frames and use consensus of these decisions. Using multiple frames eliminates most of the wrong decisions which are caused by a poorly extracted silhouette from a single video frame. The vehicle types which are classified are motorcycle, car (sedan) and van (minibus). The features extracted from the silhouettes are convexity, elongation, rectangularity, and Hu moments. Three separate methods of classification is applied. The first one is a flowchart based (i.e. rule based) method, the second one is K nearest neighbor classification, and the third one is using a Deep Neural Network. 60% of the samples in the dataset are used for training. To ensure randomization, the procedure is repeated three times with the whole dataset split each time differently into training and testing samples (i.e. three-fold cross validation). The results indicate that using silhouettes in multiple frames performs better than using single frame silhouettes.