In the studies of semantics, the main aim is to address meaning. In a computational manner, this goal is accomplished through the encoding of language constructs. These encodings are in the form of information-theoretic measures and vector representations. We have focused on the representation of words. In word representations, the earlier approaches depend on counting the statistics between word and its accompanied words, whereas the current methods are based on learning approaches. At this point, we have investigated the relation between these two approaches. We have realized that both approaches use context as the normalization factor. We support our idea by evaluating word representations on some Natural Language Processing (NLP) tasks. Furthermore, we have studied the polysemous words which carry more than one meaning. The word representation of the polysemous word provides a representation that covers more than one meaning. To overcome this issue, we provide a method to create a representation for each sense of polysemous word.