Sentiment Detection using the IBM watsonx.ai

Photo by Brock Wegner on Unsplash

Sentiment analysis is a powerful technique used to determine the emotional tone behind a piece of text. It has many applications, from understanding customer feedback to analyzing social media trends.

On July 11th IBM announced the general availability of their watsonx platform. In this blog, I have used the watsonx.ai platform, which includes large language models (LLMs) that help organizations build customized applications to solve their business challenges.

In this blog, I tried to demo how a customer can leverage the IBM LLM model to determine the sentiment of the comment. We will create a sample feedback form portal to submit the text for sentiment analysis by the IBM LLM model. After analysis, the comment and sentiment will be stored in the PostgreSQL database table for further analysis.

The following architectural design will help you understand how the application works.

Sentiment Analysis Architectural Diagram

1. The user will open a feedback form, fill out details and click submit.
2. The feedback application will parse the user comments and pass them to watsonx.ai using the API to get sentiment details.
3. Watsonx.ai returns sentiment details.
4. The Python code will write the customer-provided details from the feedback form and sentiment details to the PostgreSQL table.
5. The Cognos dashboard will refresh automatically once the table is updated with new details.
6. The Cloud Pak for Data will store the cognos dashboard and make it available for future changes.

Assumptions:

To ensure a smooth implementation, let’s establish the following assumptions:

• You have a basic understanding of Python programming.
• You can access an IBM cloud account, Watson Machine learning service, Cloud Pak for data, and PostgreSQL server.
• Create IBM cloud free tier account by following this link.
• Python 3.x and pip package manager is installed on your machine.

Pre-Requisite:

• Use this link to create API keys in the IBM cloud and store them in a safe place.
• Login to the watsonx platform using this link, and If you log in the first time, it will ask you to create a project. Click create sandbox project.
• If no Watson Machine learning service is present, you will get the following error message. Click Create a Project instead.

• Please give it a name on the next screen, associate the storage bucket, and click Create.
• Now you will be on the project screen. Click manage and copy the project id. You will need to update the project id in the Python code below.
• Click Assets, Click New Task, and click “Experiment with foundation models and build prompts.

• Associate WML service to the prompt lab

• If no WML services are found, then it will ask you to create one. Click New Service on the right side corner.

• Click Watson Machine Learning service and select the lite plan and click Create.
• Now go back to assets, click on prompt lab, and check all the boxes on the screen for your agreement. Click skip tour.
• Click on the Sample prompt on the left side panel, click sentiment detection, and fill out the details shown in the screenshot below. Click Generate.

• Click on the model in the top right corner and click “view all foundations model.” On the next screen, select “MPT-7B-Instruct2” and click Select model.
• Collect the following info from your PostgreSQL server. Ensure you have permission to create a table in the database and insert/update records to it.

DB_NAME = ‘your_database_name’
DB_USER = 'your_username'
DB_PASSWORD = 'your_password'
DB_HOST = 'your_host'
DB_PORT = 'your_port'

• Add a cognos dashboard service from IBM Cloud Pak for the Data service catalog. Follow this link for step-by-step instructions.

Step-by-step Instructions:

• Install Flask and psycopg2 on your local machine using the following command.

Pip install Flask psycopg2

For Mac or Linux run 

pip install Flask psycopg2-binary

• Create an app.py file and paste the following code to it. Update the parameters as instructed in the code. Save the file.

from flask import Flask, render_template, request, jsonify
import traceback
import psycopg2
import requests
import json

app = Flask(__name__)

api_key='Insert your IBM Cloud API key here'
PROJECT_ID='Insert your watsonx project id here'

def generate_bearer_token(api_key):
    url = 'https://iam.cloud.ibm.com/identity/token'
    headers = {
        'Content-Type': 'application/x-www-form-urlencoded'
    }
    data = {
        'grant_type': 'urn:ibm:params:oauth:grant-type:apikey',
        'apikey': api_key
    }
    response = requests.post(url, headers=headers, data=data)
    response_json = response.json()
    bearer_token = response_json.get('access_token')
    return bearer_token

bearer_token = generate_bearer_token(api_key)

# Replace the following database credentials with your own
DB_NAME = 'your_database_name'
DB_USER = 'your_username'
DB_PASSWORD = 'your_password'
DB_HOST = 'your_host'
DB_PORT = 'your_port'

def create_table():
    conn = psycopg2.connect(
        dbname=DB_NAME,
        user=DB_USER,
        password=DB_PASSWORD,
        host=DB_HOST,
        port=DB_PORT
    )
    cursor = conn.cursor()
    cursor.execute(
    '''
    CREATE TABLE IF NOT EXISTS comments (
        id SERIAL PRIMARY KEY,
        first_name TEXT,
        last_name TEXT,
        gender TEXT,
        country TEXT,
        comment_text TEXT,
        sentiment TEXT
    )
    '''
)

    conn.commit()
    cursor.close()
    conn.close()

@app.route('/', methods=['GET', 'POST'])
def index():
    if request.method == 'POST':
        try:
            first_name = request.form['first_name']
            last_name = request.form['last_name']
            gender = request.form['gender']
            country = request.form['country']
            comment_text = request.form['comment']
            insert_comment(first_name, last_name, gender, country, comment_text)
        except Exception as e:
            traceback.print_exc()
            return jsonify({'error': str(e)})
    return render_template('index.html')


def get_sentiment(review):
    url = "https://us-south.ml.cloud.ibm.com/ml/v1-beta/generation/text?version=2023-05-29"
    headers = {
        'Content-Type': 'application/json',
        'Accept': 'application/json',
        'Authorization': f'Bearer {bearer_token}'
    }
    data = {
        "model_id": "ibm/mpt-7b-instruct2",
        "input": f"Classify this review as positive or negative.\\n\\nReview:\\n{review}\\n\\nClassification:\\n",
        "parameters": {
            "decoding_method": "greedy",
            "max_new_tokens": 6,
            "min_new_tokens": 0,
            "stop_sequences": [],
            "repetition_penalty": 1,
            "time_limit": 60
        },
        "project_id": PROJECT_ID
    }
    response = requests.post(url, headers=headers, json=data)
    response_json = response.json()
    generated_text = response_json.get('results')[0].get('generated_text')
    sentiment = None
    if generated_text:
        sentiment = generated_text.strip(' "')
    if sentiment.lower() in ["positive", "negative", "null"]:
        return sentiment.lower()
    return None


def insert_comment(first_name, last_name, gender, country, comment_text):
    sentiment = get_sentiment(comment_text)
    conn = psycopg2.connect(
        dbname=DB_NAME,
        user=DB_USER,
        password=DB_PASSWORD,
        host=DB_HOST,
        port=DB_PORT
    )
    cursor = conn.cursor()
    cursor.execute(
        '''
        INSERT INTO comments (first_name, last_name, gender, country, comment_text, sentiment)
        VALUES (%s, %s, %s, %s, %s, %s)
        ''',
        (first_name, last_name, gender, country, comment_text, sentiment)
    )
    conn.commit()
    cursor.close()
    conn.close()

if __name__ == '__main__':
    create_table()
    app.run()

• Create a folder named “templates” in the same directory as your Python file.
• Inside the templates folder, create a file named index.html and copy the following content into the index.html file and save it.

<!DOCTYPE html>
<html>
<head>
    <title>Comment Form</title>
</head>
<body>
    <h1>Feedback Form</h1>
    <form method="POST" action="/">
        <label for="first_name">First Name:</label>
        <input type="text" name="first_name" id="first_name"><br>

        <label for="last_name">Last Name:</label>
        <input type="text" name="last_name" id="last_name"><br>

        <label for="gender">Gender:</label>
        <input type="text" name="gender" id="gender"><br>

        <label for="country">Country:</label>
        <input type="text" name="country" id="country"><br>

        <label for="comment">Comment:</label>
        <textarea name="comment" rows="4" cols="50"></textarea><br>
        <input type="submit" value="Submit">
    </form>
</body>
</html>

• From your computer shell, go to the directory where the app.py file resides. Once done, run the following command to execute the Python code.

Python app.py

• If you see the following output, open the browser, type localhost:5000, and press enter.

• In your browser, you will see the following screen.

• Fill out the details in the feedback form and click submit.
• When you click submit for the first time, it will create a table with “public” as the schema name and “comment” as the table name.
• Access the table and run the following SQL query.

SELECT * FROM public.comments

• You should see your record with the data you provided in the form and sentiment info. Such as “Positive,” “Negative,” or “Null”

• Now let’s create a dashboard in IBM Cloud Pak for Data using Cognos Dashboard service.
• Log in to your IBM Cloud Pak for Data account using this link to create a dashboard.
• Use the instruction provided in the link to create a platform connection.
• Now create a project and click +New Asset to add a connection to the project.
• Click +New Asset again and click Dashboard editor. Please provide a name to a dashboard and associate it with the Cognos Dashboard.
• Add the Comment table from PostgreSQL DB to the dashboard and add graphs. Here is the link to get familiarized with Cognos Dashboard.
• Try to build something like the one below.

Additional Resources:

• watsonx.ai
• watsonx.data
• watsonx.governance