In the previous note on the Probability and Statistical Inference, we have seen the the important concept of probability and statistics which are as follows: At the very first, we have seen the basic theory of probability and how to model a random phenomenon by satisfying the axioms of probability. Further, we explore the random variables …
22,424 total views
In the previous note on the Probability and Statistical Inference, we have learned expectations, moments, skewness, and kurtosis to measure the central tendency, dispersion, symmetry, and peakedness of probability curve or distribution, respectively. Introduction We define quantiles in terms of the distribution function. The value for which the cumulative distribution function is: is called the p-quantile. Here, is a value …
21,123 total views
In the previous note on the Probability and Statistical Inference, we have learned expectations and moments for the probability distribution of a random variable which gives central tendency and variability of the values of a random variable, respectively. In this note, we will further extend the concept of moments and study the other characteristics precisely the shape and peakedness, of the probability …
214 total views
In the previous note on the Probability and Statistical Inference, we started a new topic that characterizes the probability distribution of a random variable to get the hidden statistical information about the probability distribution. One of the statistical tools is the expectation of random variables or expectation of the probability distribution of a random variable. We have seen …
22,187 total views
In the earlier note on langid of this note series automatic language identification, we introduced how to detect language using the langid tool, which uses a naive Bayes classifier with a multinomial event model over a mixture of character n-grams and trained over 97 languages. It provided additional tools for model building, training, tokenization, etc., that are helpful …
Automatic Language Identification in Texts – Polyglot Read More »
32,361 total views
In the earlier note on sparknlp of this note series automatic language identification, we introduced how to detect language using the sparknlp library, which uses pre-trained deep learning models generated using CNN architectures in TensorFlow/Keras. Currently, they have published pre-trained models that can detect 375 languages, which is significantly higher than any other open-source library. Introduction In this …
Automatic Language Identification in Texts – LangId Read More »
21,779 total views
In the earlier note on langdetect of this note series automatic language identification, we had introduced how to detect language using the langdetect library, which uses a Naive Bayes classifier with character n-gram to detect language. In this note, we introduce another language identification library, which is a part of the sparknlp package. They had designed and developed Deep Learning models …
Automatic Language Identification in Texts – Sparknlp Read More »
27,970 total views
In the note series of automatic language identification, we had introduced how to detect language using the gcld3 library. Moreover, it is designed to run in the Chrome browser, written in the C++ programming language, based on a neural network model, and supports over 100 languages/scripts. In this note, we introduce another language identification library called LangDetect. …
Automatic Language Identification in Texts – Langdetect Read More »
32,357 total views
In the previous note on automatic language identification, we had introduced how to detect language using fasttext. Fasttext is a library created by Facebook’s AI research lab for efficient learning of text representations and classification. In this note, we introduce another language identification library called Google Compact Language Detector v3 (GCLD3). GCLD3 is designed to …
Automatic Language Identification in Texts – GCLD3 Read More »
12,246 total views
Language detection is vital in Natural Langauge Processing (NLP), as different NLP tasks or activities are language-dependent. Moreover, finding the best language detector that can support most natural language, short text, and multilingual texts is difficult. However, the Fasttext library performs well compared to other automatic language identification libraries such as gcld3, langdetect, langid, nltk_textcat, …
Automatic Language Identification in Texts – Fasttext Read More »
31,392 total views
