Sentiment Analysis Project: Code Walkthrough

by Pedro Alvarez 45 views

Hey guys! Today, we're diving deep into a sentiment analysis project, breaking down the code line by line. This project uses Python and popular libraries like pandas, scikit-learn, and NLTK to analyze the sentiment of text reviews. Whether you're a beginner or an experienced coder, this detailed explanation will help you understand each part of the project. Let's get started!

1. Main Imports in sentiment_analysis.py

import pandas as pd              # Data manipulation library
import numpy as np              # Numerical operations library
from sklearn.model_selection import train_test_split    # For splitting dataset
from sklearn.feature_extraction.text import TfidfVectorizer  # Text vectorization
from sklearn.linear_model import LogisticRegression    # Primary classifier
from sklearn.naive_bayes import MultinomialNB         # Bonus classifier
from sklearn.metrics import accuracy_score, classification_report  # Evaluation metrics
import nltk                     # Natural Language Toolkit
from nltk.corpus import stopwords    # For removing common words
from nltk.tokenize import word_tokenize  # For splitting text into words
import re                      # Regular expressions for text cleaning
import matplotlib.pyplot as plt  # For plotting
import seaborn as sns          # Enhanced plotting

So, let's break down these imports. These imports are the foundation of our project. We're bringing in a bunch of powerful libraries to help us handle data, process text, build machine learning models, and visualize our results. Think of it like gathering all the tools you need before starting a big project. Each library has its own set of functions and classes that we'll use to perform specific tasks. Let’s go through each one to understand what it does.

  • pandas: This is our go-to library for data manipulation. Think of it as Excel but on steroids. We use it to load, clean, and organize our data into tables called DataFrames. If you’re working with structured data, pandas is your best friend. It makes handling datasets a breeze.
  • numpy: Short for Numerical Python, numpy is essential for numerical operations. It provides support for large, multi-dimensional arrays and matrices, along with a collection of mathematical functions to operate on these arrays. We'll use it for various calculations and data transformations.
  • sklearn.model_selection.train_test_split: This function is crucial for splitting our dataset into training and testing sets. We train our models on the training set and then evaluate their performance on the testing set. This helps us ensure that our model can generalize to unseen data.
  • sklearn.feature_extraction.text.TfidfVectorizer: Text data needs to be converted into numerical form before we can feed it into our machine learning models. TfidfVectorizer does just that. It converts text into numerical features using the TF-IDF (Term Frequency-Inverse Document Frequency) method, which weighs words based on their importance in the document and the entire corpus.
  • sklearn.linear_model.LogisticRegression: This is our primary classifier. Logistic Regression is a powerful algorithm for binary classification problems (like sentiment analysis, where we classify text as either positive or negative). It’s simple, efficient, and often provides excellent results.
  • sklearn.naive_bayes.MultinomialNB: We're also including Multinomial Naive Bayes as a bonus classifier. Naive Bayes algorithms are particularly well-suited for text classification tasks. They are based on Bayes’ theorem and assume that the features are conditionally independent given the class label.
  • sklearn.metrics.accuracy_score and sklearn.metrics.classification_report: These are our evaluation metrics. accuracy_score calculates the overall accuracy of our model, while classification_report provides a detailed breakdown of precision, recall, and F1-score for each class. These metrics help us understand how well our model is performing.
  • nltk: The Natural Language Toolkit (nltk) is a powerhouse for text processing. It provides a wide range of tools for tasks like tokenization, stemming, tagging, parsing, and more. We’ll be using it for cleaning and preparing our text data.
  • nltk.corpus.stopwords: Stopwords are common words (like “the,” “a,” “is”) that don’t carry much meaning in the context of text analysis. We use this to remove stopwords from our text to reduce noise and improve the performance of our models.
  • nltk.tokenize.word_tokenize: This function splits text into individual words, a process known as tokenization. It’s a crucial step in text preprocessing, as it allows us to work with individual words rather than entire sentences or paragraphs.
  • re: The re module provides support for regular expressions, which are powerful tools for pattern matching and text manipulation. We’ll use it to clean our text by removing special characters and other unwanted elements.
  • matplotlib.pyplot: This is a fundamental library for plotting graphs and charts. We’ll use it to visualize our data and the results of our analysis.
  • seaborn: Built on top of matplotlib, seaborn provides a higher-level interface for creating more visually appealing and informative plots. It’s great for statistical data visualization. Together, these libraries provide a comprehensive toolkit for our sentiment analysis project.

Each import serves a specific purpose:

  • pandas and numpy: Core data handling.
  • sklearn components: Machine learning functionality.
  • nltk: Text processing.
  • matplotlib and seaborn: Visualization.

2. TextPreprocessor Class (text_preprocessor.py)

class TextPreprocessor:
    def __init__(self):
        self.stop_words = set(stopwords.words('english'))  # Initialize English stopwords

Okay, let's move on to the TextPreprocessor class. This class is all about cleaning up our text data. Think of it as the janitor of our project, making sure everything is spick and span before we start analyzing it. The main goal here is to remove any noise or irrelevant information from the text so that our sentiment analysis model can focus on the actual sentiment-bearing words. This is a crucial step because raw text data often contains a lot of clutter, like special characters, numbers, and common words that don't add much to the sentiment.

The TextPreprocessor class starts with an __init__ method, which is the constructor. This method is called when we create a new instance of the class. Inside the constructor, we initialize a set of English stopwords. Stopwords are common words like “the,” “a,” “is,” etc., that don't typically carry much sentiment information. We want to remove them because they can clutter our analysis and make it harder to identify the words that truly express sentiment. Initializing stopwords as a set is efficient because checking for membership in a set is much faster than in a list.

  • Constructor initializes stopwords set for efficient lookup.
    def preprocess_text(self, text):
        # Convert to lowercase
        text = text.lower()  # Standardize text case
        
        # Remove special characters and numbers
        text = re.sub(r'[^a-zA-Z\s]', '', text)  # Keep only letters and spaces
        
        # Tokenization
        tokens = word_tokenize(text)  # Split into individual words
        
        # Remove stopwords
        tokens = [token for token in tokens if token not in self.stop_words]
        
        # Join tokens back to text
        return ' '.join(tokens)  # Reconstruct cleaned text

This is where the magic happens! The preprocess_text method is the heart of our text cleaning process. It takes a piece of text as input and performs several operations to clean it up. Let's walk through each step:

  1. Lowercase Conversion: The first step is to convert the input text to lowercase using text.lower(). This is important because it ensures consistency. For example, “Good” and “good” would be treated as the same word. This standardization helps our analysis by reducing the variability in the text.
  2. Special Character Removal: Next, we remove special characters and numbers using regular expressions. The line text = re.sub(r'[^a-zA-Z\s]', '', text) does this. Let’s break it down:
    • re.sub() is a function from the re module that substitutes patterns in a string.
    • The first argument, r'[^a-zA-Z\s]', is a regular expression pattern. [^a-zA-Z\s] means “match any character that is not an uppercase letter, a lowercase letter, or a whitespace character.”
    • The second argument, '', is the replacement string (an empty string), which means we're replacing the matched characters with nothing, effectively removing them.
    • The third argument, text, is the input string. This step ensures that we only keep letters and spaces in our text, removing any noise from punctuation, symbols, or numbers.
  3. Tokenization: Now, we need to break the text into individual words. This is done using the word_tokenize() function from nltk.tokenize. The line tokens = word_tokenize(text) splits the text into a list of tokens (words). Tokenization is crucial because it allows us to work with individual words, which is essential for many text analysis tasks.
  4. Stopword Removal: We've already initialized our set of stopwords, so now we can use it to remove stopwords from our list of tokens. The line tokens = [token for token in tokens if token not in self.stop_words] does this using a list comprehension. It iterates through the tokens and keeps only those that are not in our stop_words set. This step reduces the noise in our data by removing common words that don't carry much sentiment.
  5. Joining Tokens: Finally, we need to reconstruct the cleaned text by joining the remaining tokens back into a single string. The line return ' '.join(tokens) does this. We use ' '.join(tokens) to join the tokens with a space in between, creating a cleaned version of the original text.

Each step serves a purpose:

  1. Lowercase conversion ensures consistency.
  2. Special character removal cleans the text.
  3. Tokenization breaks text into words.
  4. Stopword removal eliminates common words.
  5. Joining tokens recreates cleaned text.

3. SentimentAnalyzer Class (sentiment_analyzer.py)

class SentimentAnalyzer:
    def __init__(self):
        self.preprocessor = TextPreprocessor()  # Text cleaning component
        self.vectorizer = TfidfVectorizer(max_features=5000)  # Convert text to numbers
        self.classifier = LogisticRegression(max_iter=1000)   # Main classifier
        self.naive_bayes = MultinomialNB()  # Bonus classifier

Alright, let's dive into the SentimentAnalyzer class. This class is the brains of our operation, bringing together all the pieces we've discussed so far to analyze sentiment. It's responsible for preparing the data, training the model, and making predictions. Think of it as the conductor of an orchestra, coordinating all the different instruments to create a beautiful symphony of sentiment analysis.

The SentimentAnalyzer class starts with an __init__ method, which initializes the components we need for sentiment analysis. Let's break down each one:

  • self.preprocessor = TextPreprocessor(): Here, we create an instance of our TextPreprocessor class. This is the text cleaning component we discussed earlier. We'll use it to preprocess our text data before feeding it into our machine learning models.

  • self.vectorizer = TfidfVectorizer(max_features=5000): This is where we initialize our TfidfVectorizer. Remember, machine learning models can't work directly with text; they need numerical data. TfidfVectorizer converts text into numerical features using the TF-IDF (Term Frequency-Inverse Document Frequency) method. The max_features=5000 parameter limits the number of features (words) to the top 5000 most frequent ones. This helps reduce the dimensionality of our data and can improve performance.

  • self.classifier = LogisticRegression(max_iter=1000): This is our main classifier. We're using Logistic Regression, a powerful algorithm for binary classification problems like sentiment analysis. The max_iter=1000 parameter sets the maximum number of iterations for the solver to converge.

  • self.naive_bayes = MultinomialNB(): We're also including a Multinomial Naive Bayes classifier as a bonus. Naive Bayes algorithms are particularly well-suited for text classification tasks due to their simplicity and efficiency.

  • Initialize all components needed for analysis.

    def prepare_data(self, df):
        # Preprocess all reviews
        df['processed_text'] = df['review'].apply(self.preprocessor.preprocess_text)
        
        # Split into training and testing sets
        X_train, X_test, y_train, y_test = train_test_split(
            df['processed_text'], 
            df['sentiment'], 
            test_size=0.2,   # 80% training, 20% testing
            random_state=42  # For reproducibility
        )
        
        # Vectorize the text
        X_train_vectorized = self.vectorizer.fit_transform(X_train)
        X_test_vectorized = self.vectorizer.transform(X_test)
        
        return X_train_vectorized, X_test_vectorized, y_train, y_test

The prepare_data method is where we get our data ready for analysis. It takes a DataFrame df as input and performs three main steps:

  1. Preprocess all reviews: We apply our TextPreprocessor to each review in the DataFrame. The line df['processed_text'] = df['review'].apply(self.preprocessor.preprocess_text) does this. We're creating a new column called processed_text in our DataFrame, which contains the cleaned versions of the reviews. The apply method applies the preprocess_text function to each value in the review column.
  2. Split into training and testing sets: We split our data into training and testing sets using the train_test_split function from scikit-learn. The line X_train, X_test, y_train, y_test = train_test_split(df['processed_text'], df['sentiment'], test_size=0.2, random_state=42) does this. Let's break it down:
    • df['processed_text'] is our input data (the cleaned reviews).
    • df['sentiment'] is our target variable (the sentiment labels).
    • test_size=0.2 means we're using 20% of the data for testing and 80% for training.
    • random_state=42 sets a seed for the random number generator, which ensures that our splits are reproducible. This is important for consistency in our experiments.
  3. Vectorize the text: We convert our text data into numerical features using our TfidfVectorizer. We do this separately for the training and testing sets. The lines:
    X_train_vectorized = self.vectorizer.fit_transform(X_train)
X_test_vectorized = self.vectorizer.transform(X_test)
    do the following:
    • fit_transform is used on the training data (X_train). This method both fits the vectorizer to the training data (learns the vocabulary and IDF weights) and transforms the training data into a TF-IDF matrix.
    • transform is used on the testing data (X_test). This method transforms the testing data using the vocabulary and IDF weights learned from the training data. It's crucial to use transform here to avoid data leakage.

Finally, we return the vectorized training data, vectorized testing data, training labels, and testing labels.

  • Data preparation pipeline:
    1. Clean all review texts.
    2. Split data into training/testing sets.
    3. Convert text to numerical features.

4. WordFrequencyAnalyzer Class (word_frequency_analyzer.py)

class WordFrequencyAnalyzer:
    def __init__(self):
        self.positive_words = Counter()  # Count positive words
        self.negative_words = Counter()  # Count negative words

Now, let's explore the WordFrequencyAnalyzer class. This class is all about understanding which words are most frequently associated with positive and negative sentiments. Think of it as our detective, uncovering the clues hidden in the words themselves. By analyzing word frequencies, we can gain valuable insights into the language used to express different sentiments.

The WordFrequencyAnalyzer class starts with an __init__ method, which initializes two Counter objects: self.positive_words and self.negative_words. A Counter is a special type of dictionary that counts the occurrences of items. In this case, we'll use it to count the occurrences of words in positive and negative reviews separately. This will help us identify which words are most strongly associated with each sentiment.

  • Initialize counters for word frequency tracking.
    def analyze_word_frequencies(self, df, processed_text_column, sentiment_column):
        # Separate positive and negative reviews
        positive_texts = ' '.join(df[df[sentiment_column] == 1][processed_text_column])
        negative_texts = ' '.join(df[df[sentiment_column] == 0][processed_text_column])
        
        # Count frequencies
        self.positive_words = Counter(positive_texts.split())
        self.negative_words = Counter(negative_texts.split())

The analyze_word_frequencies method is where the word frequency analysis happens. It takes a DataFrame df, the name of the column containing the processed text (processed_text_column), and the name of the column containing the sentiment labels (sentiment_column) as inputs. Let's break down the steps:

  1. Separate positive and negative reviews: We first separate the reviews into positive and negative categories. The lines:
    positive_texts = ' '.join(df[df[sentiment_column] == 1][processed_text_column])
negative_texts = ' '.join(df[df[sentiment_column] == 0][processed_text_column])
    do this. Let's break it down further:
    • df[df[sentiment_column] == 1] selects the rows where the sentiment is positive (assuming 1 represents positive sentiment).
    • [processed_text_column] selects the column containing the processed text from these rows.
    • ' '.join(...) joins all the texts in this column into a single string, with spaces in between. We do the same for negative reviews (sentiment label 0).
  2. Count frequencies: Next, we count the frequencies of words in the positive and negative texts using the Counter objects we initialized earlier. The lines:
    self.positive_words = Counter(positive_texts.split())
self.negative_words = Counter(negative_texts.split())
    do this. positive_texts.split() splits the string of positive texts into a list of words, and Counter(...) counts the occurrences of each word in this list. We do the same for negative texts.
  • Word frequency analysis:
    1. Separate reviews by sentiment.
    2. Join all text for each sentiment.
    3. Count word occurrences.

5. ModelComparison Class (model_comparison.py)

class ModelComparison:
    def __init__(self):
        self.models = {
            'Logistic Regression': LogisticRegression(max_iter=1000),
            'Naive Bayes': MultinomialNB()
        }
        self.results = {}

Let's move on to the ModelComparison class. This class is designed to train and evaluate multiple machine learning models, allowing us to compare their performance. Think of it as a head-to-head competition, where different models battle it out to see who can best analyze sentiment. By comparing models, we can choose the one that performs best for our specific task.

The ModelComparison class starts with an __init__ method, which initializes two main components:

  • self.models: This is a dictionary that stores the models we want to compare. The keys are the names of the models (strings), and the values are the model objects themselves. In this case, we're comparing Logistic Regression and Naive Bayes.

  • self.results: This is an empty dictionary that we'll use to store the evaluation results for each model. We'll store metrics like accuracy, confusion matrix, and classification report for each model in this dictionary.

  • Initialize models and results storage.

    def train_and_evaluate(self, X_train, X_test, y_train, y_test):
        for name, model in self.models.items():
            # Train model
            model.fit(X_train, y_train)
            
            # Make predictions
            y_pred = model.predict(X_test)
            
            # Store results
            self.results[name] = {
                'accuracy': accuracy_score(y_test, y_pred),
                'confusion_matrix': confusion_matrix(y_test, y_pred),
                'classification_report': classification_report(y_test, y_pred)
            }

The train_and_evaluate method is where the model training and evaluation happen. It takes the training data (X_train, y_train) and testing data (X_test, y_test) as inputs. Let's break down the steps:

  1. Iterate through models: We iterate through the models in our self.models dictionary using a for loop. For each model, we get its name (name) and the model object itself (model).
  2. Train model: We train the model on the training data using the fit method. The line model.fit(X_train, y_train) does this. This step is where the model learns the relationship between the input features and the target variable.
  3. Make predictions: We make predictions on the testing data using the predict method. The line y_pred = model.predict(X_test) does this. The model uses what it learned during training to predict the sentiment labels for the test data.
  4. Store results: We store the evaluation results for the model in our self.results dictionary. We create a new entry in the dictionary with the model name as the key. The value is another dictionary containing the following metrics:
    • accuracy: Calculated using accuracy_score(y_test, y_pred). This metric measures the overall accuracy of the model.
    • confusion_matrix: Calculated using confusion_matrix(y_test, y_pred). This matrix provides a detailed breakdown of the model's performance, showing the counts of true positives, true negatives, false positives, and false negatives.
    • classification_report: Calculated using classification_report(y_test, y_pred). This report provides a comprehensive set of metrics, including precision, recall, F1-score, and support for each class.

By storing these metrics for each model, we can easily compare their performance and choose the best one for our task.

  • Model evaluation process:
    1. Train each model.
    2. Make predictions.
    3. Calculate and store metrics.

Key Concepts Used:

  1. Text Processing:
    • Tokenization: Breaking text into words.
    • Stopword removal: Eliminating common words.
    • Case normalization: Converting to lowercase.
  2. Feature Engineering:
    • TF-IDF: Converting text to numerical features.
    • Vectorization: Creating feature matrices.
  3. Machine Learning:
    • Model training: Fitting to training data.
    • Prediction: Testing on new data.
    • Evaluation: Calculating performance metrics.
  4. Visualization:
    • Word frequency plots.
    • Confusion matrices.
    • Performance comparisons.

Usage Instructions:

  1. Data Preparation:

    analyzer = SentimentAnalyzer()
df = pd.read_csv('reviews.csv')
X_train, X_test, y_train, y_test = analyzer.prepare_data(df)

  2. Model Training:

    comparison = ModelComparison()
comparison.train_and_evaluate(X_train, X_test, y_train, y_test)

  3. Visualization:

    word_analyzer = WordFrequencyAnalyzer()
word_analyzer.analyze_word_frequencies(df, 'processed_text', 'sentiment')
word_analyzer.plot_word_frequencies()

Conclusion

Alright, guys, we've made it through a detailed exploration of our sentiment analysis project! We've covered everything from the initial imports to the final evaluation, and hopefully, you now have a solid understanding of how each piece fits into the puzzle. We started by setting up our environment with the necessary libraries, then dove into the TextPreprocessor to clean our data. From there, we explored the SentimentAnalyzer, which prepped the data and set up our classifiers. The WordFrequencyAnalyzer gave us insights into the words driving sentiment, and finally, the ModelComparison helped us evaluate our models. Each class and method plays a crucial role in the overall process. By understanding these components, you're well-equipped to tackle your own sentiment analysis projects or even improve upon this one. Whether you're looking to analyze customer feedback, social media trends, or any other text data, the principles and techniques we've discussed here will serve as a strong foundation. Keep experimenting, keep learning, and you'll be amazed at what you can achieve! If you have any questions or want to discuss further improvements, feel free to dive into the comments below. Happy analyzing!