Today, the KPI is applied to any work unit, and it doesn’t have to be a person. KPI is measured differently in different areas of work: for a freight handler, it’s the number of kilograms of cargo per day, for a salesman, it would be sales per month. One could say that chatbots are employee s of a new generation, and the same method of evaluating efficiency is applicable to them. In this article, we will tell you how to measure the KPI of a chatbot using the example of one of our clients’ chatbots. Review Period – November 2018.

Users

https://nanosemantics.ai/wp-content/uploads/2019/12/пользователи-1024x184.png

The total number of users for a specified period of time can be tracked and compared with the prev ious period. The total number is comprised of both new and return users, and the latter is an especially useful metric. It shows how many users return to the chatbot for help compared to the previous period. This indicator is especially important for those companies that have a large flow of dail y visitors on the website or in the app. If this metric is increasing, it means that users were satisfied with the answer of the bot earlier and willingly choose this particular communication channel.

Dialogues

https://nan osemantics.ai/wp-content/uploads/2019/12/диалоги_вх_исх-1024x180.png

You can track the number of dialogs, incoming and outgoing messages for a certain period and compare it with the previous one. The aver age number of phrases in a single dialogue reflects its depth. In the example above, we see that, on average, one user writes 3.79 phrases in the course of one conversation. But quantity doesn’t always indicate quality. You can compare this with the pageview metric of your website; it is taken into account, but in itself doesn’t provide an understanding of the quality of your product. Therefore, the following indicator comes into play.

Dialogue Assessment

https://nanosemantics.ai/wp-content/uploads/2019/12/о ценка_реплики-1024x188.png

At the end of the conversation, the chatbot prompts the user to evaluate it. For example, by asking, “Were you satisfied with the answer?” The bot offers a choice from several options: “I asked another question”, “I did not understand the answer”, “The answer did not solve the problem”. Based on this feedback, you can identify the cause of negative assessment by analyzing the history of this conversation. Also, the degree of user engagement reflects the time spent on the dialogue.

Unrecognized Requests

Sometimes chatbots get confused, and that's all right. This most often happens with non-typical user requests, the answers for which aren’t included in the bot's knowledge base. There are two solutions for t hese situations.

It’s easy for the bot to tell users that it doesn’t understand the question by saying,

“Sorry, I didn’t understand you. Can you ask a different question?”

This usually works. But if the misunderstanding persists, the dialogue can be calibrated. A chatbot might say, “Sorry, I still don’t understand you. I remind you that I can help you with the following issues…”

The ironclad fallback is, of course, a customer service officer, who is essential when the conversation steers out of the chatbot’s competence. We tend to make the switch to a customer service specialist seamless—practically unnoticeable for the user.

Similar situations also need to be monitored and gradually eliminated by equipping your chatbot with new knowledge based on the history logs.

Identifying and measuring chatbot KPIs by SAP. Read article.

Internal Link Clicks

An analysis of this indicator shows which webpages suggested by the ch atbot are visited most often. This way, you can study the trend of the users’ demands for specific services or products and offer them in announcements and notifications.

Notifications

https://nanosemantics.ai/wp-content /uploads/2019/12/нотификации_метрика.pnghttps://nanosemantics.ai/wp-content/uploads/2019/12/нотификация_пример.png

The chatbot can display specific notifications. For example, if there is a sale taking place on your website, the chatbot will greet the user and offer them to find out more about the sale by clicking on the link. Notification efficiency is measured by the number of clicks.

Dialogue Topics

https://nanosemantics.ai/wp-content/uploads/2019/12/темы_диалогов-1024x502.png

The chatbot’s phrases are categorized by topic. Analysis of the quantitative distribution of messages allows you to track the most popular topics and improve future conversations. For example, if there is a sharp increase in the topic ‘Problems’, this is a signal that something is wrong.

Chatbot KPI = NPS

A chatbot is a convenient tool for conducting surveys and calculating the NPS. NPS, or the Net Promoter Score, is an assessment of user loyalty to the product and/or company.

Typically, such a survey is conducted via emails, calls, pop-ups, etc. With a chatbot to perform such an assessment, the process will become faster and more convenient. The bot will autonomously make calculations according to the following formula:

NPS = (sum of ratings from 9 to 10) – (sum of grades from 0 to 6) ÷ the number of all responders × 100

In the personal user accounts of Nanosemantics’ chatbots, for example, it is reflected like this:

https://nanosemantics.ai/wp-content/uploads/2019/12/индекс-1-1024x125.png

In conclusion, all of the above chatbot KPIs should be considered even at the stage of early development. For the chatbot to work as well as a real employee, you will have to set concrete business goals and prescribe ways to achieve them.

Read more about our Personal User Account.

Today, the KPI is applied to any work unit, and it doesn’t have to be a person. KPI is measured differently in different areas of work: for a freight handler, it’s the number of kilograms of cargo per day, for a salesman, it would be sales per month.