Wise customers contact our customer support when they encounter issues or difficulties using our product, or to remove any confusion and get a clearer understanding of how everything works. This serves as one of our most valuable forms of customer communication, and provides us with direct feedback about our product - highlighting where it is confusing, where problems exist, and what our customers actually need.  The more we grow - which has been substantial over the past 10 years - the more this happens, it’s a fact of life for us. So much so that one might be inclined to rewrite the age-old adage “nothing is certain except death and contacts”. Understanding the problems customers encounter during these interactions is a crucial step not only in providing better customer service, but also in improving our product overall.

In order to serve our customers we offer the ability to contact us via multiple channels - email, phone and chat. We refer to the last two as our live channels, allowing our agents to help customers in real-time and we see higher rates of resolution on these channels. For the past 2-3 years we have been working on automated ways to assist the customers who contact us via email and chat, when we believe it makes sense to do so.

With every contact the first step is understanding the problem the customer is contacting us about - this is what our expert agents do too, on whom we model our automation system. This post will describe the approach we used to build our automation system’s core “problem understanding” component.

Defining problems

To begin with, we had to create a well structured definition of what a customer problem is. We ultimately arrived at a definition which broke down the problems customers contact us about into two attributes:

Customer problem = customer intent + additional information (optional)

Another way to frame this:

"i need your help with A,B,C and x,y,z is the related information that you need to know about my situation" is the description of the customer problem that they are contacting us about

“i need your help with A,B,C” is the customer intent, and “x,y,z is the related information that you need to know about my situation” is the additional information.

Both components of the customer problem (customer intent and additional info) are recorded from the customer’s perspective, and how they describe their problem.

Because we have the luxury of hindsight when it comes to historical contacts, we can use a clever trick to prompt the LLM to provide us with the full description of the problem that a customer would ideally supply to us up-front in the description box. We do this by providing the following scenario along with the historical conversation transcript:

“Imagine you are a customer who has just had this conversation with a Wise CS agent {insert conversation}, but your session was disconnected and you have just been reconnected to a new agent. Provide a full description of your problem with all the necessary additional information you were asked for (if any) from the previous conversation. Frame your problem in two distinct (mutually exclusive) parts: {customer intent, and additional_information}”

This is essentially the prompting strategy we use to have the LLM extract the relevant information from the conversation transcript in a structured format. Chat transcripts have an inherently less standardized structure to email transcripts so introducing this structure was important for us. We couldn’t rely on the issue being represented well within an agent’s first reply (as we could with emails). In order to create the necessary structure to identify related groups (clusters) of customer problems we needed to create some standardization. We were confident that LLMs offered us our best chance of achieving the desired structure in such an unstructured setting.

When requesting the LLM to parse the chat, we did not impose any restrictions. We did not provide it with any prior knowledge of previously extracted intents and was not limited by a pre-defined list of intents. This enabled it to freely generate descriptions for any of the fields it was asked to populate. 

Managing the process

Imposing structure on our map:

Once we have assessed all the groups, our intent map is complete.

When adding an intent to the map, we implemented a deduplication step. As noted in the first section of this document, we unify the names of entities after initial parsing by selecting the first entity encountered to represent the group. However, this is often not the most suitable name. Therefore, we evaluate each group of names and choose the most frequent one instead of the first encountered. This selected name will be used in the map.

After changing the name, it is essential to verify that it does not become too similar to other entities. We repeat the deduplication step by comparing the embedding representations of the new names, ensuring that the distance to others remains above the established threshold of 0.4 for cosine distance. If the new name is deemed too similar, we will skip this intent and refrain from adding it to the map.

At this point we have a system that is capable of processing new chat transcripts and uncovering any new customer intents that appear, essentially the capability to learn the problem space and apply suitable labels to each contact. This allows us to organize our contacts based on the customer intent, and also track changes in the incidence rate of the various contacts we receive. It also allows us to provide our automation system with an understanding of the different types of problems that our customers contact us about, so that it can determine which one it’s dealing with in a live setting.