The Python Podcast.__init__

Share on social media:

[00:00:13] Unknown:

Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great.   When you're ready to launch your next app or want to try a project you hear about on the show, you'll need somewhere to deploy it. So take a look at our friends over at Linode.   With the launch of their managed Kubernetes platform, it's easy to get started with the next generation of deployment and scaling powered by the battle tested Linode platform, including simple pricing, node balancers, 40 gigabit networking,   dedicated CPU and GPU instances, and worldwide data centers.  

Go to python podcast.com/linode,   that's l I n o d e, today and get a $100 credit to try out a Kubernetes cluster of your own. And don't forget to thank them for their continued support of this show.   Your host as usual is Tobias Macy. And today, I'm interviewing Nate Jones about his work at Structurally to build conversational AI utilities that augment human sales interactions. So Nate, can you start by introducing yourself? Yeah. Hi, everyone. Nate Jones here, cofounder and head of innovation at Structurally.  

[00:01:13] Unknown:

And do you remember how you first got introduced to Python? Yeah. It was through Structurally. Our team, our back end, our machine learning to what we call our data layer is all built in Python. And my first exposure to it was in college through kind of GIS,   geographic information system means. So   kind of mapping on steroids,   really kind of fell in love with it there, and then we took our engineering team a little bit further than that with machine learning   and enterprise level application, basically, with Structurally.  

[00:01:43] Unknown:

And so can you describe a bit about what it is that you're building at Structurally and some of the story behind how it got started   and some of the ways that you're using conversational AI and just general machine learning in the business? Really got our start out of college, spent our time talking to  

[00:01:59] Unknown:

a lot of salespeople,   basically figuring out what their problem was, and it was   very common that salespeople hate following up and, like, qualifying leads.   They love it when they're at the closing table and can, you know, get really close to the commission,   but they hate the behind the scenes work of chasing leads, and that's really where the money is made.   So we set out to solve that problem with conversational AI.   Over text and email, our product works to basically follow-up with leads from different sources in all sorts of different industries,   have back and forth 2 way conversations  

[00:02:33] Unknown:

that essentially qualify or disqualify them, and then pass them back off to a sales rep to ideally close. Digging more into the conversational AI aspect and some of the ways that you're applying it to these sales conversations, I'm wondering if you can just talk through   what you actually mean by the term conversational AI and how it compares to the sort of wave of chatbots that became popular maybe 5 years ago. Yeah. Yeah. Absolutely. I like that question because  

[00:03:00] Unknown:

we typically have steered away from the word chatbot because it does have some baggage, I would say, from the last few years.   I would say that a chatbot is very simply,   you know, basically an application   that lives in a messaging UI   at a very simple level. So it does very basic things,   pretty straightforward,   and it's usually very evident   100% of the time that it is in fact like a bot. You usually interact with it by, like, clicking buttons   or, like, quick replies.   Super simple,   very rudimentary,   usually based on predefined kind of decision trees.   I would classify   conversational   AI in a different way   in that it's a lot more   humanlike. It's a lot more random. It feels like you're talking to a human. You can say 1 thing, go completely off script   per se,   come back to that, go somewhere else, and the AI can keep up with that while still directing the conversation forward in a business like or, like, the application   like way.  

So conversational AI still usually has a purpose.   It's   usually not there to just hold a conversation for fun. It's usually there for some purpose, in our case, to qualify leads, in other cases, to,   you know, maybe   tease out a purchase of an ecommerce product or something like that. But, ultimately, it feels like you're actually talking to a human.   And I think that that is the really big difference in our product in conversational AI versus kind of the early wave of chatbots.  

[00:04:32] Unknown:

As you mentioned with chatbots, it's generally very evident that you are interacting with this sort of mechanistic   automated system. And so you're not going to try to treat it as you would a regular human.   You're not going to try and have sort of free flowing conversations with it. And I'm wondering   what the useful interaction patterns are for people who are engaging with these conversational AIs where there is more flexibility in the ways that it can respond and the ways that it can understand the cues and context of the conversation? Is it something where   you're still going to be very sort of forthcoming about the fact that it is a mechanistic agent that you're interacting with? Or is it something where you would potentially want to have it masquerade as the eventual person that you're actually to be communicating with in person?  



[00:05:19] Unknown:

Yeah. That's a good question. We kinda split the difference, and we say,   this is an assistant for   whoever we work on behalf of. We're pretty careful and vague on purpose a little bit about the fact that it's an assistant,   but we're not trying to confuse people or, you know, mislead them.   It is still technically an assistant.   And across all of the many millions of conversations that we've had or the AI has had, we've seen that 99.9%   number of our intents that we can look at to say these people are asking questions, trying to figure out if it's a human or realize that it is an actually rope and, you know, a computer.   And so we're really proud of that,   and that wasn't just by accident. Like, we do a lot of things in the conversation to make it feel superhuman like. So when someone says, you know, in real estate, like, where we have a lot of customers or mortgage or leasing or insurance,   a lot of people are saying things like, you know, I'm going through a divorce. I just had a death. I have got a sickness in the family.  

Something like that,   we pick up on all of those,   you know, pretty, like, human empathetic,   you know, points in a conversation and say, like, oh, I'm so sorry to hear that, or congrats on the new job, or thank you for your service if you're a veteran. Things like that that go a long way in the conversations,   as well as little things like   putting purposeful typos in our messages that we actually then correct immediately after with, like, a little asterisk.   So, like, those are the little things that make it feel like a real assistant,   and that's what we set out to do from the beginning.  



[00:06:59] Unknown:

In the process of designing the interaction patterns and the capabilities that you wanted to build into the system, what were some of the lessons that you were able to take from these   previous generation   sort of chatbot oriented workflows   both in positive and negative signals?  

[00:07:16] Unknown:

I would say, in a very simple way, decision trees don't scale   or work in any form.   We've all seen them in typical chatbot builders. If someone says this, you can go down here. If someone says this, you can go down here. But we set out to   have a lot more flexibility   in the way that our conversational engine works,   but while still giving our customers the ability to customize their conversations. Since that's super important, there's a huge new wave today of, like, generative AI   where you can build a chatbot with all sorts of pre trained data or a little bit of your own,   and it just spits out what it thinks is best from the open Internet, basically.  

That doesn't necessarily work when you have business use cases, when   businesses need to say very specific things in very certain situations. So, like, 1 example is   we support use cases for SaaS companies,   and they get customers saying, hey. What's your pricing? Well, every SaaS company has different pricing, so we can't just spit out some random nonsense.   We have to allow them to customize that response.   So we set out with a very robust   script customization   framework   that allows for more what I would consider, like,   random   navigation through a conversation.  

So rather than   you know, if you make it down this part of the tree, you can't switch over to here or you can't go here, you can't go here. That's how conversation   works.   You typically are kinda going all over a tree,   not just down 1 path.   And most typical chatbots say, hey. Once you're down this path, you can either, like, restart or just end. We say, hey. You can actually go over here. You can skip a few questions. You can come back here. We'll keep it all in order,   and we will allow our   users to also   leverage that knowledge to customize  

[00:09:11] Unknown:

conversations as well. To the point of being able to keep track of   what elements of the conversation you've already visited,   what are the other aspects that you want to be able to touch upon, and how do you naturally go from, you know, the 3rd path on branch b back to, you know, the second branch on path a   and then be able to sort of maintain appropriate state information of the conversation? Like, what are some of the complexities or challenges or sort of backing capabilities that you've had to design?  

[00:09:43] Unknown:

That's a very good question because the state management is an overlooked   kinda aspect.   You know, in typical applications, you can kinda manage it   like an app. But in conversational state management, you gotta keep a lot in order.   We have kind of, like, 3 or 4 main concepts that we use in our conversational framework. They're pretty common to most   chatbots or conversational AI. They're slots,   intents,   contexts,   and   we call the other 1 another form of intents, but they're kind of called custom answers or just answers.   So slots are essentially the data that we're keeping track of. We have over   200 or 300. So that could be anything from, like, someone   telling us the location that they're looking for a house in   or the number of employees that work at their company.  

Any piece of information that might be relevant for our AI to extract in a conversation, which it does automatically   and store,   that's a slot. An intent is something that a lead   or a contact will say. So it might say, you know, hey. What's your pricing, or can you tell me about your pricing? That would be the, you know, ask about pricing intent.   Context   is   when we ask a question. So if we say something like,   hey. Can you tell me how many employees work at your company? If the lead just replies randomly with, you know, 500.   Well, what does 500 mean? Out of context, we have no idea. But in context, we know that we just asked the employee question, so employees equals 500,   and that's where we fill the slots.  

So all of that data is used not only in our conversational framework   to customize all of our   conversations, but also in the way that we train our models.   We use context,   slots, and intents all to train our models constantly.   And we have a team of human   AI trainers   who are constantly reviewing our conversations,   millions,   of conversations   and tagging each of our leads' messages   with forms of context, slots, and intents.  

[00:11:54] Unknown:

As you mentioned, you're focused primarily on these business and sales use cases, and so there's a lot of   contextual information about   what direction you might want to go down for a particular conversation. So if somebody is in the sort of exploratory phase of deciding, do they want to purchase a house? What is on the market? You know, they might be asking about the price ranges for certain neighborhoods or certain attributes of a neighborhood, like what the school ratings might be or any particular you know, the distance to the closest grocery store, things like that. And I'm wondering,   as you're designing this conversational framework and deciding points of customization that you wanna expose to your customers,   how much domain expertise do you need to bring into the overall concept of either the specific   industry that the salesperson   is participating in or the overall sort of strategies of sales techniques? And just how much of that is necessary to be able to make that framework   effective and flexible enough for the end user?  



[00:12:56] Unknown:

I would say, to kinda set the stage   here, we think about customization in 3 ways.   1 is the actual script customization,   so the back and forth once the lead actually replies.   2 is, like, the follow-up cadence for unresponsive leads. So we do both of those. That's pretty basic of most CRMs.   And then 3 is   what I mentioned earlier about, like, answers. You can almost think of those as, like, a conversational   FAQs   where there's a huge list of we have about, again, 200 or so of these intents,   anything from, like you were expressing earlier, you know, how's the market, or tell me about your pricing, or   other ones like mention divorce, like I mentioned earlier.  

All of those can be customized too, kind of in a conversational   FAQ way.   So we have   predefined   versions of those follow-up campaigns,   scripts, and answers for each industry.   So real estate, mortgage,   software,   insurance. We have defined   prebuilt   use cases you can kind of think of or playbooks   for all of those different verticals.   And if our customers want to, they can use those out of the box.   If they don't and they wanna, like, add certain things, like   how to answer the pricing question or they wanna tweak how they handle   someone saying that they're already working with a competitor or something like that,   They can do that by essentially copying our duplicates and tweaking.  

So there's not a lot of, you know, domain expertise that we particularly need   outside of setting up those initial kind of default playbooks   across those different kinda customization  

[00:14:41] Unknown:

tools.   Another interesting element of this problem space is, you know, going back to the example of somebody who's trying to real estate market for a particular neighborhood,   there's a lot of   information that needs to be present in the system to be able to reference it effectively so that if somebody says, you know, what is the price range for houses in,   you know, the San Francisco Bay Area, which is, you know, as we all know, astronomical, you have to have an integration with the MLS listings to be able to understand, you know, okay. They're looking in the market district, so that means that these are the houses that are available.   You know, now I need to be able to compute the, you know, mean or median of that range so that I can respond with that answer. Or if they're asking about the school ratings, I now need to say, okay. What are the schools within this geographical boundary?  

You know, now I need to pull that rating information from this other source. And just wondering what your process is for being able to build out those sort of integrations of informational assets that are necessary to make that conversation effective and useful?  

[00:15:45] Unknown:

Yeah. So right now,   we are kind of taking a little bit more passive stance to that is what we found. So we started out,   you know, kind of in some markets integrating really hardcore with those different data sources, like MLS data.   But we found that   upfront answering a lot of our leads' questions with   answers. So, like, if someone says, hey. Looking at 123 Main Street, how many beds does it have? We would actually,   you know, answer that with a with a real answer.   And what we found was people would actually, like, stop talking after that, which was really weird because we gave them what they want. But what we we found after kind of exploring different markets   was   usually   people,   if they hear an answer they don't like, they just give up. So if they were looking for a 4 bedroom, and we said 123 Main Street as a 3, they're just they're gone.  

So we took a little bit more of a passive approach to   answering really specific questions like that.   And instead, we say things like through those default   kind of scripts that we that I mentioned earlier,   Let me have a team member look up the number of beds and get back to you. By the way, can you tell me the how many bathrooms you're looking for?   That keeps it going. That keeps them on the hook, and then we can pass it off to an agent to to give them that answer.   But that said, through that entire script customization kind of framework, like I said,   there's opportunities   to   bring in   custom answers. So, for example, the pricing 1 is a really good example again, where if someone says, hey. Looking at your entry plan, you know, how many leads does that have?  

That can be an answer that we can then plug into our system through script customization   on behalf of individual users   to answer that kind of auto 1 by 1 basis.  

[00:17:40] Unknown:

And so digging more into   the platform level capabilities of what you're building, I'm wondering if you can talk through some of the ways that you have   designed and architected   the sort of machine learning processes,   your training mechanisms, how you're thinking about the conversational design elements, and some of the sort of technical   challenges and engineering challenges that you've had to deal with as you've been building out the company?  

[00:18:05] Unknown:

Yeah. So I kinda mentioned some of, like, how we train the data to start using slots, contexts,   and   intents. So that was, like, step number 1 was identifying a framework   that we can actually use to tag data and train data,   and   that was where we started. We've kind of evolved over time to really treat this like a,   translation problem   where   we get a   piece of text from a lead, whether it's over text message or email or live chat, and we want to translate that raw   English, Spanish, whatever language   into   a   computer   readable,   essentially,   format   of of an intent   and slot value that our product understands. So, for example, if a lead says something like,   I would like to buy a home in San Francisco with 2 beds. That's the English. We're translating that to buying home   beds equals 2.  

And that's the translation problem. So we actually have really, honed in on on treating this like a neural machine   translation problem, and that's kinda how we've always thought about the model itself. And that's the model that we're at least still using today.   There's a lot that goes into that, but it really starts with tagging data   appropriately using intents and slots, like I mentioned.  

[00:19:32] Unknown:

Obviously,   the core model that you're building is this sort of conversational capability, which has elements of natural language processing and natural language understanding. I'm wondering what are some of the other ways that you're applying ML either to   help with the sort of operational and engineering aspects or sort of   additional contextual models that might   work in tandem with the sort of core infrastructure?  

[00:19:57] Unknown:

Yeah. That's   probably my favorite question, Vas, because it's something that's been on our mind a lot late lately.   I kinda mentioned earlier when we were talking about, like, what domain expertise do we need to get into new markets or new industries, or how does that all work? And we always start by   defining,   like, prebuilt scripts   and drips   or follow-up cadences and answers. So there's a lot of content there. If you can imagine   every playbook that a sales and marketing team has, they have to distill it into a script   and think through every aspect of a conversation. So   if we ask this question, a lead says this, then it this hat needs to happen. If they ask pricing, I need to answer it this way. If they tell us their usage or number of employees, I need to answer this way.  

The number of things that our customers can customize   is essentially   infinite. They can string together   tons of different scripts for all different use cases, for all different answers.   And that's great,   but it's also overwhelming.   And   we have taken some of the burden off of our customers   to get started in using the product, prebuilt kinda default playbooks that we offer.   But, obviously, that doesn't fit everyone's perfect   use case or business,   and there needs to be some aspect of customization upfront.   So 1 area that we're really interested in right now is actually using   generative AI   to write our scripts for us, basically.  

And that's not necessarily a problem that, like, we're in love with solving from the ground up.   We built our entire conversational   model   from the ground up for our distinct use case. But there are other   products that are really good at generative AI, like the whole wave of GPT 3,   that we've been exploring for our use case because we have tons of data to say,   you know, in this context, in this qualifier with these slots,   this question should be asked. So we're looking at ways that we can actually   kind of bootstrap our   scripts   using   GPT 3 to generate the the follow-up cadences   and the actual 2 way back and forth  

[00:22:14] Unknown:

so our customers don't have to. And to your point of cadence and your earlier mention of intentionally adding in typos and corrections, I'm wondering,   what are some of the ways that you are using the   sort of analytical   aspects of looking at the conversations that are taking place and   determining sort of the effectiveness,   whether that's the length of the conversation or number of back and forth elements to it and understanding, you know, this is the optimal delay for timing or, you know, do I need to randomize the delay to make it seem more human like? And, you know, how frequently do I want to introduce these typographical errors or other, you know, human seeming foibles?  

[00:22:55] Unknown:

That's a huge area that is pretty untapped for us right now. Like, we're obviously looking at some basic stuff. We track all of that. 1 of my favorite   data visualization   models that we don't do a lot of but have thought about building in a more productized way is actually using,   like, a Sankey   diagram where   every lead starts at, you know, 100% on this side.   And some   funnel into this part, some funnel into that part, and then so on and so forth. So there's actually some great   chatbot analytics tools out there.   Chatbase is 1 that's actually offered by Google that I think got moved into   Dialogflow or something like that recently   that actually offers a Sankey   diagram just like that. And I think that's been the most useful because some people   in conversation   will say   every answer   upfront. They'll say I'm looking for a 2 bed,   2 bath under 500 k. I'm preapproved, not working with an agent, wanna move in the next 5 months. 1 message.  

Others will drag it out over 10 messages.   And it's not that 1 is better or worse. We have to account for all situations in between.   So we don't necessarily like to look at, like, length of conversation, although it is a good leading indicator   to have a successful conversation. Because if someone stayed on the hook that long, it's   pretty obviously pretty obvious that they're interested   and feel like they're talking to a human.   But we're a lot more deliberate in caring about   what is actually said in a conversation.   And we actually can track every   intent that was executed   in slot that was filled   in a conversation   in kind of a time based approach   to say this was said after this. This was said after this. This slot was filled after this. And   that's something that we look at pretty closely,   but haven't been able to build, you know, a huge robust kind   of mind map of our millions of conversations, although we're moving more towards a data warehouse approach to allow us to do to do that because we have   tons of data stored   on a conversational,   very granular level   from these conversations over millions of messages.  



[00:25:10] Unknown:

In terms of the actual   engineering aspects, I'm wondering if you can talk through some of the   model design,   you know, if you're using more sort of traditional statistical approaches, if you're doing deep learning, sort of what your decision process was for identifying   which frameworks were most effective or most useful for you given the objectives that you have or the experience of your team and just some of the   sort of ecosystem   aspects of being able to build out a business like this and relying so heavily on Python?  

[00:25:39] Unknown:

I'll speak briefly to it, but would love to let the actual AI engineer   talk about the math. So 1 thing that we could possibly link to is what our AI engineer wrote a   blog about his approach to our   sequence to sequence models with a dash of reinforcement learning. So it's a medium article. It goes through all of the   math   that goes into our model and then different approaches. But at a high level, we ultimately landed on using PyTorch   in the early days. We've been moving around a little bit   lately as we're developing a new model from kind of scratch today,   where our machine learning engineer is actually a contributor   to Allen Institute of Artificial Intelligence as well. So we use some of their tools,   and I think we're now moving towards more of a GPT 2 model that we've built in house.  

So there's kind of a lot going on. I would say that that article, as well as just checking out some of his other articles would be a great   in-depth look at the math and the stats behind some of these different approaches.  

[00:26:48] Unknown:

In terms of   the sort of complexities of being able to operationalize these models, I know that there are a lot of challenges that come into play when models hit the real world, and you have to deal with things like concept drift and accuracy and performance, you know, where performance might be mapped along multiple different   axes of, you know, time to generate an inference or performance in terms of the accuracy of the inference that's generated. And I'm wondering, what are some of the sort of engineering and operational  

[00:27:17] Unknown:

challenges that you and your team have been dealing with? Yeah. So I would say the model pipeline itself was a challenge early on.   When we started the company,   everything from preprocessing to postprocessing   was happening in a single model, a single   application, basically. And then we said, no. Okay. Let's break each out into different applications, different   endpoints, essentially. And then we actually went back to just having everything predicted through 1 model. We decided that that was best. So I think that was a long process to figure out what worked best there. But, ultimately, 1 of the things that we're most interested in,   again, from, like, an operational   proactive approach is I talked about we have a kind of a human in the loop team that's monitoring conversations when they kinda go south to bring them back,   but also   doing that, their training data,   constantly,   both, like, after conversations and during conversations.  

So we get a really good   live look at how our model is performing in production   down to the second. Right now, it's happening. And we're constantly looking at the f score or the f 1 score   to say, okay.   How different was   what was predicted versus what was approved   or annotated?   And I think that's, you know, boiled down at a really high level even from,   you know, senior leadership team.   We love to look at that number to say, is our model improving   in this way? Because   our human team shouldn't be labeling the same types of messages over and over and over. We should get better at those ones we've seen and rely on the human team to   really focus on the ones that we've never seen before and ultimately drive that down with broader data. So we really like to look at that score kind of in real time and constantly for improvements.  



[00:29:09] Unknown:

In terms of the training data, you mentioned that you want to cut down on the amount of human intervention necessary to do the labeling. I'm wondering if you can just talk through   some of the ways that you're able to source that training data, particularly as you're moving into new markets,   and what level of effort you've put into looking at things like snorkel AI or sort of automated data labeling to be able to   scale the process of being able to actually generate enough training data?  

[00:29:37] Unknown:

So I would say that   we like to own our training data in house because it's so nuanced and so   robust. Any individual   message can have upwards of,   well, technically infinite or,   you know,   close to,   number of combinations of slots and intents and context on them to be labeled.   So we've got we've invested a ton into   training our human   AI trainers to make sure that the lay the data is labeled   very accurately.   But 1 of the really interesting pieces you asked there was, how can we leverage that to get into new,   you know, use cases? That's a really good question because if you think about it, you know, we're restricted to our AI essentially knowing   only   about the slots in context that we've told it to. We have, like, 300, and it's constantly growing. But every time we add 1, essentially, it's new. So we might add a new qualifier that's favorite color.  

Never never used before, but how will we know that a lead saying purple or yellow   is actually their favorite color? Things like that.   1 of the ways that we've actually kind of   overcome overcome that   is   by leveraging   scripting   in a way that uses other data from other industries. So   if you think about it, like, you know,   automotive, for example,   is not that different conversationally from, like, real estate or mortgage. We're asking things like, are you already working with a salesperson,   or what's your time frame? What's your price range? So we have tons of data in those context. That's why context is so important   that the lay the data itself isn't actually all too different. So getting into new markets   can actually   reuse   those contexts or questions   to kinda seed new data in new markets and new scripts. So that's been really helpful. But 1 of the other ways that we're thinking about it is, you know,   obviously, with more things like GPT   3, we're moving into,   you know, few shot or 0 shot learning.  

And that's the whole benefit of it is even if we were to add a brand new context,   our product has never seen before. It should leverage all of the training from the,   you know, massive dataset that it's trained on to have,   you know, a guess at what the accurate prediction is   or training it with a few examples or few shot, it should be able to kinda hone in on what that looks like. So we've thought about different ways again to actually leverage things like GPT 3 to spit out example   data   labeled   in the way that we expected,  

[00:32:22] Unknown:

and then just retrain that back into our model. You mentioned GPT 3 a few times and some of these large general purpose models. I'm wondering what are some of the other areas of NLP and AI and transfer learning advancements that have been most beneficial to the area of   conversational AI and being able to build these types of interactive interfaces.  

[00:32:44] Unknown:

I might send you back to our   AI engineer's most recent article.   So I think that the GPG 3 large general products hit the nail on the head. His most recent article talks about   training the Allen NLP library   on 11, 000, 000, 000   parameter model in a single node. So I think that   that is in part kind of the   breakthrough over time that GPT 3 hasn't been able to learn on.   And just the breadth   of data that we can have and   compute, I guess, really has been   probably our most beneficial   learning because, you know, in the early days, even our little model had very little data, but our model today has, you know, nowhere near 11, 000, 000, 000 parameter model. But it's pretty large.   And, you know, the training time that it takes has gone up to retrain our model consistently,   but it stayed pretty level in the advancements and how we're able to actually   just deal with   the training aspect,   as well.  



[00:33:51] Unknown:

And in terms of the   ongoing research   and industry developments that are happening in the space of AI, I'm wondering what are some of the   aspects that you're keeping a close eye on with the hopes of being able to benefit from and apply to the work that you're doing at Structurally?  

[00:34:08] Unknown:

Yeah. I think, obviously, I've talked about GPT 3. That's the 1 that we love to keep an eye on the most. I don't think that we have any plans to ever move our conversational model to that because, again,   usually, you have to move to a completely generative kind of conversation approach, and then we would have to lose all of our customization   at that point, basically.   So it just doesn't make a lot of sense for us to ever move there yet. Maybe there'll be some changes that support that a little nicer.   But right now,   you know, we wanna be in control of   what   the output of the messages   is.  

We just want to be in control   from an AI machine learning perspective of what's being predicted to allow that to happen. That's where we're really honing in on. But then I would say   GPG 3.   I would also say   better conversational   tools. So there's a really great company called Voxable   that just came out with actually a GPG 3 powered   conversational design assistant.   There's a number of companies out there. Voiceflow is another that's pretty focused on kind of, like,   conversational design as a whole,   because I think that is such   a overlooked but important   aspect of conversational   AI that doesn't get a lot of love   because it's so hard. If you want to build an app   that can sync through every possible turn of a conversation, it's almost   really hard to comprehend.  

And a company like Voxel   really understood that and honed in on that and said, hey. We understand that it's hard to kinda think through example conversations to design your   conversational AI apps.   What if we   use tools like GPT 3 to, like, seed example conversations for you so you can play them out better and tweak them. There's a huge mental burden that kinda comes off of you when you can kinda see examples of that happening. It kinda gets the wheels turning. So I think that's a huge opportunity   kinda in the conversational design space, but powered by AI too.  

[00:36:14] Unknown:

Another interesting element of what you're doing is the fact that you are a   relatively early stage company. You're building your core capability around a machine learning model and an AI process, which I know can be quite time and labor and cost intensive. And I'm wondering what you have seen as some of the   sort of challenges of being able to   fund the   bootstrapping process to be able to get to a point where you're able to actually start the flywheel spinning and maintain momentum   and what you've seen   in your experience within the industry of the sort of availability of capital and willingness of investors to be able to give you sort of the additional runway that's necessary to be able to actually   build out those initial capacities   to prove out the functionality   of the   project that you're trying to build out? So I think  

[00:37:08] Unknown:

the right   investors   know that   AI doesn't just come about. It starts with   very specific data,   and it kinda grows from there.   Our investors   fully understand that.   What we've kinda run into now was, you know, there's always that kinda cold start problem of, well, I don't have much data. I think that you can actually kinda get around that a lot quicker than we did back in the day with things like GPT 3 to really seed your data really quick.   When we started, we just said, well, we kinda have to build humans in the loop to start so we can get every aspect, every message tagged. We're not gonna take any for granted,   and that's what we did. We even seeded some data ourself internally,   you know, pretty rudimentarily,   get it started.  

And then I would say we're kinda at a new point where it's like, we really need to see   our AI   start to perform in these   markets or use cases where we have a ton of data   and kind of, you know,   cut out   all of our human in the loop   except for very small use cases. And our investors and even large partners understand that that's happening.   And so I think that just from a VC perspective,   savvy AI   investors   know kind of those different waypoints as you grow your company,   and they'll kinda   expect you to get there. But I will say that, you know, at least for companies that started when we did, there was probably a little   slower going because none of these models existed   when we started. None of these   concepts really even existed, and they were really developed during this time.  

And I think we have a lot more   democratized   kind of approach to some basic NLP than we did at the beginning   of   when we started the company. So I think that even now today,   investors might see that as   shortening the timeline   to get to market.   But, ultimately, they know that at some point, you're still gonna run into this blocker where it's like,   you know, you still kind of will always need humans in some regard. There's no such thing as fancy self learning AI. It always takes some sort of   labeling to some degree, and they understand that. And they might just expect you to hit that quicker, but then solve it a little bit more aggressively once you get there. And another aspect of building a business around AI is  

[00:39:36] Unknown:

sort of the   computational expense of training and running the models. And I know that particularly in the deep learning space, a lot of people are starting to lean more heavily on GPU   capabilities, but then there have also been   efforts underway to apply things like network pruning to be able to reduce the overall weight of the model and allow it to run at speed on CPUs. And I'm wondering   what you see as the potential benefits of that area of exploration and research as we do start to,   you know, move away from the brute force capacity of GPUs   to be able to cut down on that overall computational cost?  



[00:40:16] Unknown:

We've done a little bit of everything here again too. So we actually started with   a in house   GPU rig that we   trained all of our models on, you know, sandbox and staging and production models. And that worked well   for a time, but then we outgrew it. We moved to cloud GPUs basically at that point, and now we're using the Google Workspace   GPUs.   And even still, like, our relatively large model, but not massive by any means, is taking   usually about 4 to 5 days to retrain.   And that's   cumbersome because, you know, we wanna work in our development sprints to see improvements   weekly in our models.  

But when you're trying to retrain models constantly, it takes 4 to 5 days.   It just kinda adds up. So   we don't have a great answer there yet. We're using   the cloud to try and get some leverage   on our models in different ways there.   But, you know, obviously, there's gonna be a lot of work that will be done there that can benefit us.  

[00:41:23] Unknown:

And as you have been building out the structurally business and working with your customers and working in this space of conversational AI, what are some of the most interesting or innovative or unexpected ways that you've seen either the structurally platform or conversational AI generally applied?  

[00:41:39] Unknown:

My favorite use   case is probably through   email.   So   our product supports multiple channels, text, email, live chat.   And 1 of my favorite use cases is just generally   different,   like, software companies or other companies   reaching out to their   previously   canceled or churned or lost customers   over email and just say, hey. You know, it's been a while since we talked.   We have some new features, products, offerings.   Wanted to see if you'd come back.   And   there's actually a lot that goes on behind the scenes   to make email work   and to actually, like, make it work   in a human like way. So we have to be able to use a custom   address. We don't wanna use, you know, some random   at   structurally dot, you know,   production whatever.  

They want to say, you know, at insurance company.com.   And, like, that whole process in itself is actually relatively tricky,   especially when we have to route their messages, only certain messages from their email to us to get predicted, labeled, and reply on their behalf.   So that has been a really fun and tricky   ongoing engineering problem as a whole that I really like because   when you're having a conversation   over text or over live chat, you're kinda used to it,   like, from a chatbot perspective even though it's still really early. But when you're talking to someone over email, it just feels harder to,   you know, spoof as a human. You don't really ever get, like,   spammy type bots over an email because it's so hard to kinda go through all those different layers, I think. And so I think, you know, we are being very careful in making sure that it comes across human,   but I think we succeeded really early on there to make it feel human. And, you know, there's just a lot that goes into that, which has been really fun and unique to see how people are using it. As you have been building out the business and working in this space, what are some of the most interesting or unexpected or challenging lessons that you've learned in the process?  

I   originally   thought that   building   script customization was impossible.   I'll be the 1st to admit that I told my team that when we started that, you know, hey. We'll never be able to customize these conversations like   competitors could. Like, we had early competitors   that   were human based, so they just kinda spun up like teams from across the world,   kinda gig economy, part timers, just   having conversations manually.   And, obviously, you could build a little UI around that to say,   you know, hey, human behind the curtain, say this in this scenario.   Pretty easy because a human can kinda determine where that might need to be put into a conversation.  

But how would you ever do that for   AI? How would you ever string together all the different scenarios or turns in a conversation?   I didn't think it was possible to start, but our team figured it out   and allowed a really complex   customization layer on top of the AI   to make it happen. So I would say that was my biggest learning, and I'm excited, like I said, to see where conversational design as a whole goes because I think there's a lot more that could be done there. We built a   robust   script customization   UI, but it's also very complex   because of all of the different turns that goes into a conversation.  

And I'm excited to see what companies come out   to   make that simple so that anyone, anywhere can really quickly spin up   very humanlike   complex   conversations   through conversational AI.  

[00:45:29] Unknown:

And for people who are looking to be able to automate more of the interactions   that take place within their business or within their product, what are some of the cases where conversational AI is the wrong choice and they might be better served with a simple, you know, decision tree based chatbot or actually having a, you know, proper human in the loop? Yeah. So 1 area that we've seen  

[00:45:51] Unknown:

not work well for us is   when you have a very referral   heavy business.   So you very rarely would want to ever put   another assistant or anything in the way of   you and a friend, basically. So if you work in a business that's very   kind of sphere of influence heavy or referral based,   I would say that you're much better off keeping that to very simple conversations since you already know them.   There's no easy conceivable way to   ever, you   know, put the prior relationship you have with someone   into some app that a conversation can, like, piggyback off of. Maybe   someday, that could be possible.  

But right now, you know, it doesn't make a lot of sense for any of our customers to ever have an AI or assistant for that matter kinda interject in those referrals or sphere based   relationships.  

[00:46:46] Unknown:

As you continue to work on the product and capabilities of Structurali, what are some of the things you have planned for the near to medium term? Yeah. I think I've talked a lot about  

[00:46:56] Unknown:

making the script customization   really   streamlined through potentially GPT 3 or other generative   AI.   The other thing is   making our follow-up.   So I kinda mentioned the 3 different types of   customization in our product. 1 is the the drip campaigns or the follow-up   builder. Right now, that's a very   basic product, and it's not terribly productized   yet.   We have   yet to see a way to combine   all 3 of those products in a really seamless way. And this is kinda why I was   mentioning, like, I'm really excited about, like, the UI UX aspect of   this a lot more than the machine learning because that's already kind of getting there and ahead of times.  

I think that the, like, behind the scenes stuff is way way further ahead than the the UI side of things, so I'm excited for that to catch up. And   we have really big plans to combine   the drip follow-up and   the script follow-up and the custom answers into kind of 1 seamless builder to say, design a conversation.   And you can say,   if a lead doesn't reply for a day, send this message. But then if they do reply with this, reply in this way. But then if they ghost you for 2 days, follow-up again. But if they say this and they wanna be followed up within 6 months, actually follow-up with this message in 6 months. So it's really about, like,   designing a super   intelligent,   essentially, like, salesperson   who's very experienced   in putting that into a whole product that we're really rallying around.  



[00:48:35] Unknown:

Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And so with that, I'll move us into the picks. This week, I'm going to choose a tool that I've been using for a little while now called Vantage, and it's   a better cost management platform for your AWS   spend than what AWS offers. So it makes it so you can actually somewhat understand where the different costs are coming from. So I've been able to   get a lot more benefit of being able to understand where we can cut out some resources or optimize our spend there. So if you're running in Amazon, definitely worth taking a look at what they're building there. And so with that, I'll pass it to you, Nate. Do you have any picks this week?  



[00:49:14] Unknown:

Yeah. From a tool perspective, 1 of the tools that we've been using is Videoform.   New to us, pretty competitive to other, like,   video recording sales tools.   We've been excited about it because we use it internally, but we're also gonna leverage it in our product   to allow our customers to   send   videos   and images or GIFs of themselves in the conversation   to kind of, again, make our conversations feel even more human like. So   really appreciate what they're doing. Love the approach to kinda cut through the noise in terms of sales messages with  

[00:49:51] Unknown:

personalized video. Alright. Well, thank you very much for taking the time today to join me and share the work that you're doing at structurally and in the conversational AI space. It's definitely a very interesting problem domain. So it's been fun to learn more about some of the ways that you're tackling it and some of the   happenings in the space. So I appreciate all the time and effort you're putting into that, and I hope you enjoy the rest of your day. Yeah. Absolutely. Appreciate being on. Thank you.   Thank you for listening. Don't forget to check out our other show, the Data Engineering Podcast at data engineering podcast.com   for the latest on modern data management.  

And visit the site of pythonpodcast.com   to subscribe to the show, sign up for the mailing list, and read the show notes.   And if you've learned something or tried out a project from the show, then tell us about it. Email host at podcastinit.com   with your story.   To help other people find the show, please leave a review on Itunes and tell your friends and coworkers.