Leveraging Copilot Studio to chat with Phi 3 and Llama 3.1 in Microsoft Teams

OpenAI’s ChatGPT taught us that a successful AI implementation puts the user at the center. They were the first to turn an instruct model into a chat model and provide it through an intuitive and seamless user experience. The rest is history.

In this article, we’ll discuss how IT departments, and specifically data science teams, can elegantly surface their own proprietary models to their end users, wherever those users are. All of that can be done safely, securely, and in no time.

My colleague Nico recently showed me a simple Microsoft Teams bot where end users can chat with different AI models, regardless of where those are deployed in the backend.

Let’s dive in!

Rémi: Hey Nico, welcome back since we talked how to extract data from PDFs with OCR+GPT. Feels like an eternity. Can you explain what we are solving here?

Nico: Thank you, Rémi. It starts with the realization that many data science teams are under pressure to deliver and iterate quickly on AI for their organizations.

The problem is that the best data scientists may not have the resources or the desire to spend too much time developing custom user experiences for their AI models. They may also not want to be responsible for the full-stack enterprise security of the solution, especially when operating in a fast-paced environment.

Additionally, creating a conversational AI user experience is simple but challenging. It can quickly flip on you. To write a good chat solution, you need not only good AI but also good user experience.

Rémi: That’s where a tool like Copilot Studio can help you, right?

Nico: Yes! With a tool like Copilot Studio, we are solving this problem. We empower data science teams with a no-code chat user experience platform. Push the user experience and security requirements to the tool and focus on the primary work where you can make the biggest impact for the organization, like building, testing, and deploying proprietary AI models at scale.

And with Copilot Studio, you benefit from a decade of Microsoft's experience in conversational AI, with thousands of organizations across the globe already using the tool in production.

Rémi: Do you have an example for us?

Nico: Yes. It starts by spinning up our AI models in the back end.

Deploying Phi 3 and Llama 3.1 in Azure AI Studio

For today's exercise, we're using Azure AI Studio because it's simple. From there, we deploy two leading open-source models: Phi 3 Medium from Microsoft and Llama 3.1 from Meta as Model as a Service.

Again, we are focused on solving the user experience. The architecture remains the same regardless of where we chose to host our proprietary AI.

Leveraging Dataverse to store our conversations

The next thing we need from a backend perspective is two tables to save securely conversations and related messages. For that, we will use Dataverse and a simple model-driven app on top to access conversation histories.

This simple message table will also come in handy if there is a need to summarize the conversation into an email summary, a Word document, generate an image, or even use a PowerPoint presentation template to create a deck from a conversation, just as examples.

Rémi: That’s great – now let’s talk about the front-end part of the architecture, Copilot Studio?

Copilot Studio, the conversational AI User Experience

Nico: In conversational AI, we traditionally start with the first topic that gets triggered when the user opens the bot: it is the ‘Conversation Start’ system topic in Copilot Studio. In this topic, we initialize a few front-end variables (like the model) and create a new conversation in Dataverse with Power Automate. We return the Dataverse record ID of the conversation we just created as a bot front-end variable.

 We are using Power Automate in Copilot Studio as our data orchestrator between the front end and the backend.

Rémi:  That’s right – when you call Power Automate from Copilot Studio, you have access to all the premium connectors because they are included with your license!

Leveraging the fallback topic to chat with the model

Once we have our front-end variables set up, we focus on our ‘Fallback’ system topic. This is the topic that triggers when no other topics match the user's intent. In our case, this is the core of the solution, where we call our AI model in the backend.

This ‘Fallback’ topic keeps getting triggered, allowing us to build the conversation recursively. We save our messages in Dataverse until the user requests a new conversation.

Leveraging ‘custom topics’ to override model responses

To clear the conversation history (like starting a new conversation in ChatGPT), we use a ‘custom topic’ to reset the variables and create a new conversation record in the Dataverse.

This way, all the user has to do is ask the bot for a "new conversation." The concept of using a custom topic to override our model call is a powerful tool from a chat UX perspective.

In fact, we use this same concept to offer the user the ability to switch between Phi 3 and Llama 3 models during a conversation, similar to how you would switch between GPT-4-o mini and GPT-4-o in the ChatGPT app.

Live demo and source code

Rémi: That’s amazing, any chance we can see a live demo and, better, put our hands on this Copilot Studio solution?

Nico: Absolutely.

If you want to see a live demo, join us on September 4th for our Power Platform Master Series webinar. This is the second season the team has been running this series. It will be the first one of the season, and it should be a good one!

The bot solution is relatively simple, with two system topics, three custom topics, three Power Automate flows, and two tables. We are working on putting it on GitHub as open-source solution so everyone can build from there. Stay tuned and join us September 4th!

One last thing: if Copilot Studio is not yet part of your organization, feel free to use the Copilot Studio trial version. That way, you can start experimenting for yourself.

Rémi: Thank you Nico! This was great, à la prochaine!