In this study, we carried out social touch gesture classification on two publicly available datasets, Corpus of Social Touch (CoST) and Human-Animal Affective Robot Touch (HAART), and our demo dataset. In order to classify touch gesture datasets, four different models are proposed: 3-dimensional convolutional neural network (3D-CNN), 3-dimensional convolutional-long term short term memory neural network (3D-CNNLSTM), 3-dimensional convolutional-bidirectional long term short term memory neural network (3D-CNN-BiLSTM) + and 3-dimensional convolutional transformers network (3D-CNN-Transformer). The fundamental layer of the proposed deep neural network architectures is 3-dimensional convolution layer that enables to extract spatio-temporal features of touch gestures. In this regard, with the use of spatio-temporal features of touch gestures, generalization performance of proposed four models have been improved using data augmentation techniques by applying randomly shift and rotation, and ensemble learning. Additionally, We also found out that Stochastic Gradient Descent (SGD) optimization algorithm has better generalization performance than Adaptive Moment Estimation (ADAM), which is used more frequently in deep learning. The accuracy of classification results of three dataset is investigated in terms of proposed model. The results showed that the proposed methods, especially ensemble classifier and the ensemble classifier with data augmentation, are beneficial for obtaining more generalizable learning algorithms. The scripts of deep neural network architecture are available upon request.