article by Grant Ronald, July 2020
First published at https://www.linkedin.com/pulse/training-your-nlp-model-best-practices-writing-grant-ronald/
src: pixabay.com
Image by Manfred Steger from https://pixabay.com
Stop the press – an NLP model is only as good as the data you train it with. I think we can all generally agree on that statement. Also, when it comes to training intents with utterances, ultimately the best data is real data. However, you have to start somewhere and to do so you need to bootstrap your NLP model by creating your own utterances.
Even with the best intentions in the world, synthetic utterances can introduce bias, often because we all have our own myopic view of the world. It is also fair to ask the question “what makes a good utterances or training corpus?” So how might you kick-start training your NLP model?
Remember The Basics
You don’t have to actually know much, or anything, about NLP so long as you follow some general guidelines. From my experiences, here are the guidelines I try to follow:
- Focus on the essence of the intent and what makes it unique: remember the goal is not to “understand”, but to classify. You must ensure intents are designed to be as distinct as possible and the utterances will define what makes those intents distinct. Think of the rings in the Olympic flag; ideally each intent should have no overlap (or as little as possible).
- As you create utterances, focus on what it is that makes this intent distinct from any other – that is the essence you want to capture in your utterances.
- Consider the different approach to triggering the intent – for example, “I want to reset my password” vs. “I can’t log into my email” are essentially requests for the same intent but one states the solution and one states the problem. People can arrive at the intent in different ways, so ensure the utterances cater to that.
- Ensure you don’t artificially skew one intent over another by introducing recurring phrases which hold little semantic meaning in the intent – e.g. “please”. Or at least do it consistently.
- Ensure you have a good mix of entity values. But you don’t need to include all possible entity values, and the utterances shouldn’t always reference the same entity value.
- Try to keep a general consistency between intents. Be careful if one intent has 300 utterances and the other has only 10. There are some exceptions (for example, only so many utterances are needed for a HELLO intent.).
- Aim for semantically and syntactically complete sentences. However, it’s OK to have a few smaller more direct phrases.
- Consider training the model with any obvious misspellings, typos, or country-specific variations (e.g. trash vs rubbish, diaper vs nappy).
- More isn’t always better – start with a small manageable set of utterances (maybe 20 or 30) and add more as you find out how well your model is working. Huge initial sets of training utterances can easily mask poor and overlapping utterances.
Those are some general guidelines to consider when generating utterances. In keeping those points in mind, I try to follow a process when I write utterances which helps ensure a reasonable quality of utterances.
Focus On The Unique
I start my utterance writing with the most obvious and typical phrase and create a few versions where I keep the “primary/key word” the same (for example in the OrderPizza intent, “order” might be the thing that makes this intent unique from other intents in the pizza digital assistant) but generally vary the rest of the sentence
e.g.
"I want to order pizza"
"I want to order a pepperoni"
"Can I order a pizza please?"
Vary The Sentence Structure
Having created about 5 or 6 of those utterances I then consider changing the personal pronoun which helps change the structure of the sentence whilst ensuring I don’t overload the utterances with one specific personal pronoun such as “I”. However, I keep the primary keyword the same (in this case “order”).
e.g.
"Can you order me pizza?"
"We’re ordering a pizza for the family"
"Pizza orders for us please"
Vary The Key Term
Then consider different terms for the key word (in this case, the verb “order”). You don’t have to go through a thesaurus for every possible synonym for “order”, I tend to aim for 3 or 4 different terms which would be regarded as typical for the keyword and repeat the rules above.
e.g.
"Can I get a pizza please?"
"I want to get a small veggie pizza with no mushrooms"
"I’d like a pizza delivery"
"Can you deliver me a pizza?"
Note in the above example I’m introducing some natural variance from the initial “I want to order pizza” utterances rather than simply stamping out the same phrases with only the verb changing.
Vary The Secondary Terms
In the pizza ordering example it is the desire to “order” a pizza which makes this intent unique when compared with other intents in the same skill. And whilst the digital assistant is for a pizza shop which only sells pizza, it is also worth introducing alternative terms for what you might call the secondary feature of the intent, i.e. pizza. In doing so you are avoiding giving too high a signal to “pizza” in this intent when it is “order” which is important. And of course you can mix this with some of the variations you’ve already created.
e.g.
"Can you order me a food delivery?"
"I’d like a delivery of some food"
"Can I order one of your stuffed crusts to take away?"
"I want a home delivery of a Mighty Meaty."
The Rule Is To Not Always Follow The Rule
Whilst this might seem like a contradiction, the idea is to follow the rules, but not so absolutely or formulaically that the utterances are predictable and artificial. The rules give you the guidance on how to build good utterances. But by not being too rigid in applying them, you introduce natural variation which ultimately benefits the training model.
Next Steps
Having bootstrapped your NLP model with enough utterances to support some testing with real users, what is the next step in improving the NLP model? Ultimately this is where the real work begins. You should consider doing the following:
- Create a batch test script to give you a baseline measure of how well your NLP is performing.
- Periodically (this could be on a daily basis depending on traffic) analyse the customer conversations for new utterances to boost your NLP model.
- You may consider using the above guidance to clean up the user utterances – for example you might take a user input which is deemed valuable for training the model but remove any extraneous phrases not related to the key user request. From that phrase you could generate a number of variations
- Re-test the NLP using your batch test to measure if classification is improving.
- Add further test phrases to your batch test (ideally phrases you are NOT adding to the model).
- Repeat.
Ultimately there are many factors that contribute to the success of your digital assistant, with a well-functioning NLP model being a primary factor. Using the above techniques should guide you on making better decisions when training your model.
Related Content
TechExchange – First Step in Training Your Bot
Understanding Why Intent Resolutions Are Sometimes Not What You Expect
Best Practices and Tips for Conversational Design and User Experience in Oracle Digital Assistant
TechExchange: Building A Production Quality FAQ In An Hour With Oracle Digital Assistant
