Python - Frequency Distribution





Counting the frequency of occurrence of a word in a body of text is often needed during text processing. This can be achieved by applying the word_tokenize() function and appending the result to a list to keep count of the words as shown in the below program.

from nltk.tokenize import word_tokenize
from nltk.corpus import gutenberg

sample = gutenberg.raw("blake-poems.txt")

token = word_tokenize(sample)
wlist = []

for i in range(50):
    wlist.append(token[i])

wordfreq = [wlist.count(w) for w in wlist]
print("Pairs\n" + str(zip(token, wordfreq)))

When we run the above program, we get the following output −

[([', 1), (Poems', 1), (by', 1), (William', 1), (Blake', 1), (1789', 1), (]', 1), (SONGS', 2), (OF', 3), (INNOCENCE', 2), (AND', 1), (OF', 3), (EXPERIENCE', 1), (and', 1), (THE', 1), (BOOK', 1), (of', 2), (THEL', 1), (SONGS', 2), (OF', 3), (INNOCENCE', 2), (INTRODUCTION', 1), (Piping', 2), (down', 1), (the', 1), (valleys', 1), (wild', 1), (,', 3), (Piping', 2), (songs', 1), (of', 2), (pleasant', 1), (glee', 1), (,', 3), (On', 1), (a', 2), (cloud', 1), (I', 1), (saw', 1), (a', 2), (child', 1), (,', 3), (And', 1), (he', 1), (laughing', 1), (said', 1), (to', 1), (me', 1), (:', 1), (``', 1)]

Conditional Frequency Distribution

Conditional Frequency Distribution is used when we want to count words meeting specific crteria satisfying a set of text.

import nltk
#from nltk.tokenize import word_tokenize
from nltk.corpus import brown

cfd = nltk.ConditionalFreqDist(
          (genre, word)
          for genre in brown.categories()
          for word in brown.words(categories=genre))
categories = ['hobbies', 'romance','humor']
searchwords = [ 'may', 'might', 'must', 'will']
cfd.tabulate(conditions=categories, samples=searchwords)

When we run the above program, we get the following output −

          may might  must  will 
hobbies   131    22    83   264 
romance    11    51    45    43 
  humor     8     8     9    13 


Frequently Asked Questions

+
Ans: Python Text Processing Tutorial for Beginners - Learn Python Text Processing in simple and easy steps starting from basic to advanced concepts with examples including Text Processing,Text Processing Environment,String Immutability,Sorting Lines,Reformatting Paragraphs,Counting Token in Paragraphs ,Convert Binary to ASCII,Convert ASCII to Binary,Strings as Files,Backward File Reading,Filter Duplicate Words,Extract Emails from Text,Extract URL from Text,Pretty Print Numbers,Text Processing State Machine,Capitalize and Translate,Tokenization,Remove Stopwords,Synonyms and Antonyms,Text Translation,Word Replacement,Spelling Check,WordNet Interface,Corpora Access,Tagging Words,Chunks and Chinks,Chunk Classification,Text Classification,Bigrams,Process PDF,Process Word Document,Reading RSS feed,Sentiment Analysis,Search and Match,Text Munging,Text wrapping,Frequency Distribution,Text Summarization,Stemming Algorithms,Constrained Search. view more..
+
Ans: Python Text Processing Tutorial for Beginners - Learn Python Text Processing in simple and easy steps starting from basic to advanced concepts with examples including Text Processing,Text Processing Environment,String Immutability,Sorting Lines,Reformatting Paragraphs,Counting Token in Paragraphs ,Convert Binary to ASCII,Convert ASCII to Binary,Strings as Files,Backward File Reading,Filter Duplicate Words,Extract Emails from Text,Extract URL from Text,Pretty Print Numbers,Text Processing State Machine,Capitalize and Translate,Tokenization,Remove Stopwords,Synonyms and Antonyms,Text Translation,Word Replacement,Spelling Check,WordNet Interface,Corpora Access,Tagging Words,Chunks and Chinks,Chunk Classification,Text Classification,Bigrams,Process PDF,Process Word Document,Reading RSS feed,Sentiment Analysis,Search and Match,Text Munging,Text wrapping,Frequency Distribution,Text Summarization,Stemming Algorithms,Constrained Search. view more..
+
Ans: Python Text Processing Tutorial for Beginners - Learn Python Text Processing in simple and easy steps starting from basic to advanced concepts with examples including Text Processing,Text Processing Environment,String Immutability,Sorting Lines,Reformatting Paragraphs,Counting Token in Paragraphs ,Convert Binary to ASCII,Convert ASCII to Binary,Strings as Files,Backward File Reading,Filter Duplicate Words,Extract Emails from Text,Extract URL from Text,Pretty Print Numbers,Text Processing State Machine,Capitalize and Translate,Tokenization,Remove Stopwords,Synonyms and Antonyms,Text Translation,Word Replacement,Spelling Check,WordNet Interface,Corpora Access,Tagging Words,Chunks and Chinks,Chunk Classification,Text Classification,Bigrams,Process PDF,Process Word Document,Reading RSS feed,Sentiment Analysis,Search and Match,Text Munging,Text wrapping,Frequency Distribution,Text Summarization,Stemming Algorithms,Constrained Search. view more..
+
Ans: Python Text Processing Tutorial for Beginners - Learn Python Text Processing in simple and easy steps starting from basic to advanced concepts with examples including Text Processing,Text Processing Environment,String Immutability,Sorting Lines,Reformatting Paragraphs,Counting Token in Paragraphs ,Convert Binary to ASCII,Convert ASCII to Binary,Strings as Files,Backward File Reading,Filter Duplicate Words,Extract Emails from Text,Extract URL from Text,Pretty Print Numbers,Text Processing State Machine,Capitalize and Translate,Tokenization,Remove Stopwords,Synonyms and Antonyms,Text Translation,Word Replacement,Spelling Check,WordNet Interface,Corpora Access,Tagging Words,Chunks and Chinks,Chunk Classification,Text Classification,Bigrams,Process PDF,Process Word Document,Reading RSS feed,Sentiment Analysis,Search and Match,Text Munging,Text wrapping,Frequency Distribution,Text Summarization,Stemming Algorithms,Constrained Search. view more..
+
Ans: Python Text Processing Tutorial for Beginners - Learn Python Text Processing in simple and easy steps starting from basic to advanced concepts with examples including Text Processing,Text Processing Environment,String Immutability,Sorting Lines,Reformatting Paragraphs,Counting Token in Paragraphs ,Convert Binary to ASCII,Convert ASCII to Binary,Strings as Files,Backward File Reading,Filter Duplicate Words,Extract Emails from Text,Extract URL from Text,Pretty Print Numbers,Text Processing State Machine,Capitalize and Translate,Tokenization,Remove Stopwords,Synonyms and Antonyms,Text Translation,Word Replacement,Spelling Check,WordNet Interface,Corpora Access,Tagging Words,Chunks and Chinks,Chunk Classification,Text Classification,Bigrams,Process PDF,Process Word Document,Reading RSS feed,Sentiment Analysis,Search and Match,Text Munging,Text wrapping,Frequency Distribution,Text Summarization,Stemming Algorithms,Constrained Search. view more..
+
Ans: Python Text Processing Tutorial for Beginners - Learn Python Text Processing in simple and easy steps starting from basic to advanced concepts with examples including Text Processing,Text Processing Environment,String Immutability,Sorting Lines,Reformatting Paragraphs,Counting Token in Paragraphs ,Convert Binary to ASCII,Convert ASCII to Binary,Strings as Files,Backward File Reading,Filter Duplicate Words,Extract Emails from Text,Extract URL from Text,Pretty Print Numbers,Text Processing State Machine,Capitalize and Translate,Tokenization,Remove Stopwords,Synonyms and Antonyms,Text Translation,Word Replacement,Spelling Check,WordNet Interface,Corpora Access,Tagging Words,Chunks and Chinks,Chunk Classification,Text Classification,Bigrams,Process PDF,Process Word Document,Reading RSS feed,Sentiment Analysis,Search and Match,Text Munging,Text wrapping,Frequency Distribution,Text Summarization,Stemming Algorithms,Constrained Search. view more..
+
Ans: Python Text Processing Tutorial for Beginners - Learn Python Text Processing in simple and easy steps starting from basic to advanced concepts with examples including Text Processing,Text Processing Environment,String Immutability,Sorting Lines,Reformatting Paragraphs,Counting Token in Paragraphs ,Convert Binary to ASCII,Convert ASCII to Binary,Strings as Files,Backward File Reading,Filter Duplicate Words,Extract Emails from Text,Extract URL from Text,Pretty Print Numbers,Text Processing State Machine,Capitalize and Translate,Tokenization,Remove Stopwords,Synonyms and Antonyms,Text Translation,Word Replacement,Spelling Check,WordNet Interface,Corpora Access,Tagging Words,Chunks and Chinks,Chunk Classification,Text Classification,Bigrams,Process PDF,Process Word Document,Reading RSS feed,Sentiment Analysis,Search and Match,Text Munging,Text wrapping,Frequency Distribution,Text Summarization,Stemming Algorithms,Constrained Search. view more..
+
Ans: Python Tutorial for Beginners - Learn Python in simple and easy steps starting from basic to advanced concepts with examples including Python Syntax Object Oriented Language, Methods, Tuples, Tools/Utilities, Exceptions Handling, Sockets, GUI, Extentions, XML Programming. view more..
+
Ans: Python Overview - Learn Python in simple and easy steps starting from basic to advanced concepts with examples including Python Syntax Object Oriented Language, Methods, Tuples, Tools/Utilities, Exceptions Handling, Sockets, GUI, Extentions, XML Programming. view more..
+
Ans: Python Environment Setup - Learn Python in simple and easy steps starting from basic to advanced concepts with examples including Python 3 Syntax Object Oriented Language, Methods, Tuples, Tools/Utilities, Exceptions Handling, Sockets, GUI, Extentions, XML Programming. view more..
+
Ans: Python Basic Syntax - Learn Python in simple and easy steps starting from basic to advanced concepts with examples including Python Syntax Object Oriented Language, Methods, Tuples, Tools/Utilities, Exceptions Handling, Sockets, GUI, Extentions, XML Programming. view more..
+
Ans: Python Variable Types - Learn Python in simple and easy steps starting from basic to advanced concepts with examples including Python Syntax Object Oriented Language, Methods, Tuples, Tools/Utilities, Exceptions Handling, Sockets, GUI, Extentions, XML Programming. view more..
+
Ans: Python Basic Operators - Learn Python in simple and easy steps starting from basic to advanced concepts with examples including Python Syntax Object Oriented Language, Methods, Tuples, Tools/Utilities, Exceptions Handling, Sockets, GUI, Extentions, XML Programming. view more..
+
Ans: Python Decision Making - Learn Python in simple and easy steps starting from basic to advanced concepts with examples including Python Syntax Object Oriented Language, Methods, Tuples, Tools/Utilities, Exceptions Handling, Sockets, GUI, Extentions, XML Programming. view more..
+
Ans: Python Loops - Learn Python in simple and easy steps starting from basic to advanced concepts with examples including Python Syntax Object Oriented Language, Methods, Tuples, Tools/Utilities, Exceptions Handling, Sockets, GUI, Extentions, XML Programming. view more..
+
Ans: Python Numbers - Learn Python in simple and easy steps starting from basic to advanced concepts with examples including Python Syntax Object Oriented Language, Methods, Tuples, Tools/Utilities, Exceptions Handling, Sockets, GUI, Extentions, XML Programming. view more..
+
Ans: Python Strings - Learn Python in simple and easy steps starting from basic to advanced concepts with examples including Python Syntax Object Oriented Language, Methods, Tuples, Tools/Utilities, Exceptions Handling, Sockets, GUI, Extentions, XML Programming. view more..
+
Ans: Python Lists - Learn Python in simple and easy steps starting from basic to advanced concepts with examples including Python Syntax Object Oriented Language, Methods, Tuples, Tools/Utilities, Exceptions Handling, Sockets, GUI, Extentions, XML Programming. view more..




Rating - NAN/5
470 views

Advertisements