Vast.ai

Medusa is slightly different than other types of speculative decoding in that it adds a piece of the original model to do the speculation. TGI is the first major serving framework for large language models that enables Medusa-style speculative decoding.

As the year winds down, rumors are intensifying around NVIDIA's highly anticipated GeForce RTX 5090 GPU. Industry insiders are divided on the release date, with some sources suggesting a launch just in time for Christmas, while other reports point to a formal announcement at CES 2025 in the new year.

In the complex landscape of data center operations, understanding and adhering to various compliance standards is crucial.

Medusa is a method of specular decoding. Speculative decoding speeds up inference of large language models by having a smaller model multiple tokens and lets the larger model just verify. If the Verification for the large model is cheaper than generating the tokens themselves. If the smaller model is accurate enough, then the cost to generate tokens goes down overall. Medusa is slightly different than other types of speculative decoding in that it adds a piece of the original model to do the speculation.

This guide will show you how to set up SGLang to serve a language model on Vast.

The L40S was developed to meet the surging demand for GPUs that can handle the intense computational requirements of machine learning training and inference. How does it stack against the L40 -- and which one do you need?

SGLang provides an OpenAI-compatible server, allowing you to easily integrate it into chatbots and other applications. As companies develop their AI products, they often face challenges like rate limits and high costs when using these models. With SGLang on Vast, you can run your own models in the form factor you need, at a much more affordable price point. As inference demand grows with agents and complex workflows, SGLang on Vast excels in performance and affordability where it matters most.

When deciding between the A100 and H100, consider your specific workload requirements. If you need top-tier double-precision performance and superior memory bandwidth, or you're dealing with next-gen HPC at datacenter scale and trillion-parameter AI, the H100 is the clear winner. For a more versatile and cost-effective solution that still delivers powerful AI performance, the A100 is a solid choice.

vLLM is now more flexible than ever as it also supports embedding models. This brings vLLM's dynamic batching and Paged Attention to embedding models for much faster throughput, all from the docker image that developers are used to. This guide will show you how to setup vLLM to serve embedding models on Vast.

#TGI is an open source framework for #LLM inference. It specifically focuses on throughput for serving, automatic batching, and ease of use with the Huggingface Ecosystem. TGI provides an OpenAI compatible server, which means that you can integrate it into chatbots, and other applications. Here's our guide for running it on Vast AI: