SPR’s Post

Transcript

Ohh, hello and welcome to this episode of SPR's What's on Tech. My name is Matt Mead. I'm the CTO of SPR. And SPR's What's on Tech is where we discuss everything related to modern technology and we talked to experts. And today I've got with me Mr. Steven Devoe. Steven is our Specialty Director of Data. That means that everything AI and Data rolls up to Steven. So Steven, thanks for being here today. Great to see you, man. Thanks for having me. Steven, you and I have been talking a ton lately about future proofing AI solutions, and I thought it'd be kind of fun to continue that conversation here today. Absolutely. Let's face it, Matt, We're in a world where AI is changing all the time. One day, one model might be considered the best for a certain task, and then the next you're going to find that there's a significantly better model for what you're trying to do. You don't want to find yourself in a situation where one day you're using a model and your product has evolved to need something different, and you really can't leverage other model besides the one you're using today. Yeah. Couldn't agree more Steven. And and So what you're talking about, what we're talking about is not hey, I need to upgrade from ChatGPT 3.5 to 4.0 because that's so trivial because the API has not fundamentally changed and it's an easy upgrade. Not that we're just talking about complex rag chat bot implementations here today, but imagine Steven, if you're doing a complex rag based chat bot implementation where you had multiple pipelines coming into a vector database and then you had maybe one or more large language models. In all these technologies needed to be coordinated together. I mean that's the sort of complexity that I think is really the type of thing that we're trying to target here today talking about future proofing AI. In isolation, none of those things are overly complex, but when you combine them all together, they all have dependencies on each other and you may have certain data pipelines that perform in one way. Combine that with a vector base database, plus you know some type of retrieval algorithm. Uh. And then throw an LM on top of that and all those pieces have to fit together at any given time. You may want to change one or more of those things as your product evolves and you want to do that in a way that doesn't slow you down. Well, one of the things that we always talk to our clients about is get the fundamentals right. Data is at the heart of AI. So we need to make sure that our data pipeline is absolutely rock solid. And hey, while we're at it, let's automate it to the extent possible so that the initial training and any sort of retraining. Could be done with the least amount of human intervention and kind of. In the ML OPS world, a lot of that has been proven out. There's a lot of best practices that exist today just given how long ML has been around. In the LM and AI world, there's a lot less. Best practices to be had. So at the core of it, you want to ensure that you're taking scalability, flexibility, security, all those traditional best practices and making sure those are built in from the start, but they just take a little bit more forethought because this is also new, right? And as we're talking about this concept, Steven, I think sort of the elephant in the room is like, hey, wait a second, like this concept of abstraction has been around for decades in software development and systems. Design, you know, the idea where we isolate or encapsulate a portion of the implementation so that the specifics around that implementation don't sort of bleed out into the rest of the software code base. And then that way it's easier to swap out that piece in the future if necessary or if desired. As you mentioned, people have been doing this for decades in software engineering, whether they wanted to isolate a a database or a payment system, whatever it may be. Uh, really we've been talking about flexibility so far in terms of using different models for performance and doing more and more complex tasks. At the same time. You want to think about the lower level tasks and using the most cost effective model and the fastest response times to give you the best ROI on your dollar spent. You also want to think about things like regulation, which is almost certainly imminent in certain legal jurisdictions. You don't want to find yourself in a situation where you're dependent on one specific model type, and all of a sudden you can't use that because that model doesn't comply with the laws and regulations in those areas. The 3rd and final thing, A lot of these models are still under managed quotas where you are limited to the number of requests you can make per minute or total tokens you can use per minute. And that obviously puts a ceiling on how much you can use this and leverage it for your business purposes. So scaling across different models is obviously one great way to mitigate against that risk. That is such a great point, Steven. I'm glad you brought that up. And then again, I'm going to steal a few things. You said you you've mentioned scalability and I'm going to bring in costs. These are two things that are really important and can lead to the need to make fairly significant changes to an AI model and therefore warrants this future proofing around AI. We see clients all the time change the frequency with which they're calling a model. And so that can change just how the system performs #1, but then second. We can have a huge impact on the cost, right, for a system and it can change the cost over time. I think it's no secret that people are concerned about the cost of AI models and how effective they are. So you always want to consider ways to reduce the cost or use the best model for the job. Hey, Stephen, I think this was really informative and I'm so glad you were able to join me here today. Thanks so much. Thanks for having me man.