Check our Ultimate Chatbot Guide
How to check chatbots’ effectiveness – Metrics, Statistics and Analytics

How to check chatbots’ effectiveness – Metrics, Statistics and Analytics

KODA Bots 30.07.2020
KODA Bots

Implementing a chatbot is one thing, but the key is to assess whether their analytics is working. That’s why we continuously develop our statistics module to let you check, for example, whether users use the solution, what content or offers they find interesting or to draw your attention to unexpected – or expected – events in your chatbot. This allows you to learn and take rights steps to make your chatbot work best for your users. And also to improve the products or services offered. 

 

Imagine that you’ve organized a campaign to attract new chatbot users and you want to analyze whether it has succeeded. Or that you’ve sent out a push notification with product updates and links to the e-store and want to check which products are most popular. Today you’ll see the platform’s statistics module and specific quantitative metrics relating to users, messages, and effects, as well as qualitative metrics that provide inspiring insights.

 

1. User statistics 

The bot users data allows you to analyze how popular the solution is and whether users actually use it.

Number of chatbot users

Here’s the total number of people who have interacted with your bot. You can follow how the number changes from the launch. This lets you assess whether your campaigns have brought the desired effect and whether your user base is gradually growing. 

Daily chatbot activity

How many people talked to your chatbot that day? How many have just started using it? The “daily users activity” metrics will answer these questions. The daily activity dashboard is organized into new and active users. It gives you an overview of how many new users have reached out to the chatbot on a particular day and how many have interacted with it: pressed any button, sent a message or file. 

And also how many of them have previously used the tool and got back to it.

Sources of new users

The metrics allows you to examine where your chatbot got the most traffic from. This is where you check how people got there. The statistics on user acquisition sources are especially useful if we’ve used additional ways make the redirecting to chatbot easier: customer chat plugin, QR code option or auto-response to a comment. 

 

2. Bot message statistics

The users statistics reflect the general traffic trend in your bot. In order to evaluate the tool, however, you need more detailed information about the interactions. The “message metrics” are helpful data. They are used to check the scale and length of the conversations. 

Sessions

The “sessions” metrics will tell you how many separate visits to the chatbot were generated by the users. Comparing this value to the number of users, you will see how many sessions on average correspond to an individual user. 

You can also check here the average duration of a session, i.e. how long users have talked to the bot and what is the average number of messages exchanged per session. This is an important factor, especially if your aim is to engage your audience

Interactions

The “interaction” dashboard shows you the number of messages exchanged between the user and the bot. You can also convert it into the number of interactions per person. This will give you a general idea of how long the users calls last.  

Usually, there are 2-3 times more messages from the chatbot, since it often sends several items in a row, e.g. an introductory message, a carousel or messages and graphics.

3. Additional chatbot action metrics

You can carry out additional actions in the chatbot and set up notifications about events that interest you. Whether it’s adding a contest entry, a product inquiry or booking a service in chatbot. In all cases, the additional statistics available in the platform’s analytics panel will prove to be useful. 

Events in chatbot

In statistical events we collect information about people who have reached a particular point in conversation. This allows us to adjust the client’s path to their needs on an ongoing basis and optimize the solution to be as user friendly and useful as possible. 

Some events are evaluated automatically:

  • Sending specific elements to the chatbot, such as swear words or pictures. The metrics help to qualitatively assess the user path and optimize it.
  • Enabling Moderator Mode – this informs you how many users of your chatbot have been looking for help from a CS specialist. The statistics are useful when assessing the tool effectiveness.

 

When we determine with the client the chatbot objective and the point of conversion, we establish a statistical event specially tailored to this. This provides us with additional analytical information to help us tailor the solution specifically for them. 


For example, we can measure:

  • The number of bookings made by the bot
  • The number of products ordered by your bot

Contest entries

If you’re organizing, or planning to organize, an engaging chatbot contest, make use of this statistics item. In the contest entry statistics, you can check how the entries submission was distributed throughout the contest duration and on which day participants sent the highest number of entries. This helps you keep track of the effectiveness of marketing actions and the entire contest.

 

Popularity of bot elements

It shows you how many times a given “block” or chatbot element has been displayed to users. In other words, you can see which of them are the most popular. This is yet another statistics item that allows you to improve the content and offer presented in the chatbot, so that users are more likely to use it.

 

Outgoing links

You can direct your chatbot users to external websites. In the “outgoing links” section you will see how many times the link has been followed. The statistics item comes in handy, especially if you want your chatbot to support sales or direct users to a specific site, e.g. a blog, information page or product.

 

NLP statistics

The “NLP statistics” section reveals how chatbot recognizes users’ intentions. It also saves phrases that haven’t been identified. This not only helps us to evaluate whether the chatbot understands the users, but also to enable NLP training. 


4. Chatbot performance or commercial statistics

Together with our clients we’ve developed effectiveness definitions, so that they are reliable and determine to what extent a given chatbot helps to achieve specific goals. And also to what extent automating communication with users has been useful. 

When working with clients, we often create new metrics that depend on the chatbot objective. Below you can find the most commonly used metrics.

Number of quotations or services inquiries

This is one of the most important indicators as far as chatbots used in sales are concerned. They should produce more or better inquiries from prospects. In the case of chatbots used in customer service, the metric is of lesser importance, however, some of them may also identify user problems and cross/add-on selling opportunities.

 

Return on investment

Return on investment (ROI) is crucial for many projects. It makes it possible to analyze bot expenses and the benefits it generates. ROI is measured by companies, taking into account the chosen objective and checking how the chatbot has translated into specific results: time saving for employees, fewer queries to customer service departments, product sales or engagement in marketing campaigns.

 

Autonomy of the solution

In the context of customer service, we’ve assumed that the effectiveness refers to the interactions and users percentage, in which there was no the need for a moderator intervention. 

Hence, we calculate, for example on a monthly basis, how many conversations were held with a moderator as compared to the number of all calls. And similarly for users: the number of users who talked in the chatbot with a moderator compared to all users who interacted with the chatbot at the given time. The moderator could be called by the user themselves or by the moderator. Their participation could also result from the adopted conversation scenario.

While carrying out the analysis, one should note that the chatbot only deals with a part of the interaction by taking over specific processes and often the CS specialist participation is a natural part of the conversation.

 

Qualitative chatbot analytics

In chatbots you have access not only to hard data, e.g. how many users use the solution or what links they follow the most, but also soft data: what the purchase path looks like, what the most common problems are, and what motivates users. It takes place not only on a “declarative” level, but also on the behavioral one. Many insights come from choices and naturally emerge from questions. 

Bots help to “listen” more closely to people’s needs and they support you in personalizing and improving products or services. Therefore, as Iwona writes in her text, “advanced analysis should never end with quantitative data.” We can draw a great deal of qualitative knowledge from chatbots that will allow us to get to know the organization’s audience better: check their preferences, their way of thinking, decision-making processes, as well as understand the language of their communication. Based on this, we can optimize the solutions and content created to make them work as best as possible.