Leveraging Copilot Studio to chat with Phi 3 and Llama 3.1 in Microsoft Teams
Microsoft Designer prompt: combine the power of the LLM Phi3, the LLM Llama and Microsoft Teams

Leveraging Copilot Studio to chat with Phi 3 and Llama 3.1 in Microsoft Teams

OpenAI’s ChatGPT taught us that a successful AI implementation puts the user at the center. They were the first to turn an instruct model into a chat model and provide it through an intuitive and seamless user experience. The rest is history.

In this article, we’ll discuss how IT departments, and specifically data science teams, can elegantly surface their own proprietary models to their end users, wherever those users are. All of that can be done safely, securely, and in no time.

My colleague Nico recently showed me a simple Microsoft Teams bot where end users can chat with different AI models, regardless of where those are deployed in the backend.

Let’s dive in!

Rémi: Hey Nico, welcome back since we talked how to extract data from PDFs with OCR+GPT. Feels like an eternity. Can you explain what we are solving here?

Nico: Thank you, Rémi. It starts with the realization that many data science teams are under pressure to deliver and iterate quickly on AI for their organizations.

The problem is that the best data scientists may not have the resources or the desire to spend too much time developing custom user experiences for their AI models. They may also not want to be responsible for the full-stack enterprise security of the solution, especially when operating in a fast-paced environment.

Additionally, creating a conversational AI user experience is simple but challenging. It can quickly flip on you. To write a good chat solution, you need not only good AI but also good user experience.

Rémi: That’s where a tool like Copilot Studio can help you, right?

Nico: Yes! With a tool like Copilot Studio, we are solving this problem. We empower data science teams with a no-code chat user experience platform. Push the user experience and security requirements to the tool and focus on the primary work where you can make the biggest impact for the organization, like building, testing, and deploying proprietary AI models at scale.

And with Copilot Studio, you benefit from a decade of Microsoft's experience in conversational AI, with thousands of organizations across the globe already using the tool in production.

Rémi: Do you have an example for us?

Nico: Yes. It starts by spinning up our AI models in the back end.

Deploying Phi 3 and Llama 3.1 in Azure AI Studio

For today's exercise, we're using Azure AI Studio because it's simple. From there, we deploy two leading open-source models: Phi 3 Medium from Microsoft and Llama 3.1 from Meta as Model as a Service.

Again, we are focused on solving the user experience. The architecture remains the same regardless of where we chose to host our proprietary AI.


Llama 3.1 and Phi 3 Medium deployed as a Model as a Service in Azure AI Studio

Leveraging Dataverse to store our conversations

The next thing we need from a backend perspective is two tables to save securely conversations and related messages. For that, we will use Dataverse and a simple model-driven app on top to access conversation histories.

This simple message table will also come in handy if there is a need to summarize the conversation into an email summary, a Word document, generate an image, or even use a PowerPoint presentation template to create a deck from a conversation, just as examples.


Example of a MS Teams conversation with Phi 3 saved in Dataverse

Rémi: That’s great – now let’s talk about the front-end part of the architecture, Copilot Studio?

Copilot Studio, the conversational AI User Experience

Nico: In conversational AI, we traditionally start with the first topic that gets triggered when the user opens the bot: it is the ‘Conversation Start’ system topic in Copilot Studio. In this topic, we initialize a few front-end variables (like the model) and create a new conversation in Dataverse with Power Automate. We return the Dataverse record ID of the conversation we just created as a bot front-end variable.


Creating and resetting the conversation in Copilot Studio

 We are using Power Automate in Copilot Studio as our data orchestrator between the front end and the backend.

Rémi:  That’s right – when you call Power Automate from Copilot Studio, you have access to all the premium connectors because they are included with your license!


Creating a new conversation record in Power Automate called from Copilot Studio

Leveraging the fallback topic to chat with the model

Once we have our front-end variables set up, we focus on our ‘Fallback’ system topic. This is the topic that triggers when no other topics match the user's intent. In our case, this is the core of the solution, where we call our AI model in the backend.

This ‘Fallback’ topic keeps getting triggered, allowing us to build the conversation recursively. We save our messages in Dataverse until the user requests a new conversation.


Calling Power Automate from Copilot Studio Fallback topic to chat with our AI model


Calling our model chat API from Power Automate HTTP premium connector (incl. in Copilot Studio)

Leveraging ‘custom topics’ to override model responses

To clear the conversation history (like starting a new conversation in ChatGPT), we use a ‘custom topic’ to reset the variables and create a new conversation record in the Dataverse.

This way, all the user has to do is ask the bot for a "new conversation." The concept of using a custom topic to override our model call is a powerful tool from a chat UX perspective.

In fact, we use this same concept to offer the user the ability to switch between Phi 3 and Llama 3 models during a conversation, similar to how you would switch between GPT-4-o mini and GPT-4-o in the ChatGPT app.


Custom Copilot Studio topic to allow end user to switch from Phi 3 to Llama 3


 Switching AI model from Phi 3 to Llama 3 in MS Teams

Live demo and source code

Rémi: That’s amazing, any chance we can see a live demo and, better, put our hands on this Copilot Studio solution?

Nico: Absolutely.

If you want to see a live demo, join us on September 4th for our Power Platform Master Series webinar. This is the second season the team has been running this series. It will be the first one of the season, and it should be a good one!

The bot solution is relatively simple, with two system topics, three custom topics, three Power Automate flows, and two tables. We are working on putting it on GitHub as open-source solution so everyone can build from there. Stay tuned and join us September 4th!

One last thing: if Copilot Studio is not yet part of your organization, feel free to use the Copilot Studio trial version. That way, you can start experimenting for yourself.

Rémi: Thank you Nico! This was great, à la prochaine!

Leo Wang

AI and Automation, Business Intelligence, Enterprise Mobility and always in Web3.

2mo

This is fantastic Remi!

Rémi Dyon Very insightful. Thank you for sharing

Jamie Hu

Data and AI Specialist at Microsoft | ✨AI Advisor | Speaker | Disruption Digest Podcast Host | ☁️Digital Transformation with Azure

2mo

Fascinating topic, Rémi Dyon - and I love the Q & A style with Nico Sprotti!

To view or add a comment, sign in

Insights from the community

Others also viewed

Explore topics