This year, as my firm has delved deeper into commercially implementing large language models (LLMs), we've encountered a crucial decision point that many in the industry are also facing: Should you rely on an LLM hosted by a vendor like OpenAI, or is it better to configure and fine-tune your own LLM within your organization? Our experts have strong opinions on this matter. The former approach—using a vendor-hosted, non-configurable LLM—is, in many ways, akin to alchemy. It might yield some interesting results, but it’s unlikely to provide the precision and reliability required for most serious business applications. On the other hand, configuring your own LLM holds significant potential, but it demands considerable expertise to tune and retrain the model with proprietary data.
When deciding which path to take, the first step is to select the right LLM for your specific needs. This selection process should not be taken lightly. With over 4,000 LLMs currently available from platforms like Hugging Face, each with its unique attributes, the task can seem daunting. However, the key to success lies in proper model profiling.
Profiling an LLM is all about understanding its strengths and weaknesses in relation to your intended use case. Not all LLMs are created equal—some are better suited for tasks involving language generation, while others excel in reasoning or data extraction. By thoroughly profiling an LLM before committing to it, you can significantly reduce the effort and cost associated with fine-tuning. For instance, selecting a model that already aligns closely with your domain requirements can save countless hours and resources in the long run.
The cost of fine-tuning LLMs has skyrocketed with each new version. GPT-2 cost around $2M to retrain, which ballooned to $10M with GPT-3, $40-50M for GPT-3.5, and approximately $100M for GPT-4. Shockingly, GPT-4o is now in the $10B range! Critically, by carefully selecting and profiling the right LLM, these costs can be mitigated—you don’t always need the biggest model; far from it.
Moreover, the exponential growth in the number of available LLMs means that the market is becoming increasingly crowded. This proliferation further underscores the importance of having a well-honed profiling strategy. Selecting a team with demonstrable experience is critical. Our team has profiled hundreds, if not thousands, of LLMs, building a library of experience that enables us to shave weeks or months off project timelines.
Looking ahead, the future is bright for organizations that take the time to properly implement these tools. This technology, like all technology, will likely run in our environments for many decades into the future, lighting up limitless hitherto unseen possibilities.
However, without a well-designed strategy, LLMs have the potential to become expensive albatrosses and, worse, could cause irreparable harm.
#LLM #AI #BANKING #TRUSTWORTHYAI
