Calculatrum’s Post

DigitalOcean Droplets are Linux-based virtual machines (VMs) that run on top of virtualised hardware.

DigitalOcean Droplets are Linux-based virtual machines (VMs) that run on top of virtualised hardware.

The wait is over! There is now some Microsoft guidance on using Karpenter with AKS: https://lnkd.in/egb3P5iJ

#tbt #tbthursday If NVMe is the answer, what are the questions? If NVMe is the answer, then what are the various questions that should be asked? Some common questions that NVMe is the answer to include what is the difference between NVM and NVMe? Is NVMe only for servers, does NVMe require fabrics and what benefit is NVMe beyond more IOPs. Lets take a look at some of these common NVMe conversations and other questions. Main Features and Benefits of NVMe Some of the main feature and benefits of NVMe among others include: Lower latency due to improve drivers and increased queues (and queue sizes) Lower CPU used to handle larger number of I/Os (more CPU available for useful work) Higher I/O activity rates (IOPS) to boost productivity unlock value of fast flash and NVM Bandwidth improvements leveraging various fast PCIe interface and available lanes Dual-pathing of devices like what is available with dual-path SAS devices Unlock the value of more cores per processor socket and software threads (productivity) Various packaging options, deployment scenarios and configuration options Appears as a standard storage device on most operating systems Plug-play with in-box drivers on many popular operating systems and hypervisors Continue reading about NVMe, common questions and answers here https://lnkd.in/dUDwnfx #nvme #ssd #flash #pmem #tier0 #storage #pcie #sas #das #cloud #packaging #server #io #networking #compute #gpu #cpu #benchmark #performance #pace #sds #s2d #datainfrastructure #edge #ai #ml #dl #tradecraft #management #dataprotection #mvp #mvpbuzz

Liquid cooling is on the minds of every infrastructure team that has to contend with the massive TDPs of GPU clusters. Supermicro has several liquid-cooled server options, they had an amazing stack on display at Computex. But it's not just the hardware, management and visibility into the loop is critical too. Supercloud Composer is Supermicro's way of aggregating all of the critical data of the entire liquid-cooling estate. Supermicro NVIDIA StorageReview.com Jordan Ranous

Yandex introduces YaFSDP, a method for faster and more efficient LLM training This enhanced version of FSDP significantly improves LLM training efficiency by optimizing memory management, reducing unnecessary computations, and streamlining communication and synchronization. Here’s an overview of YaFSDP based on this Medium article. How it works: - Layer sharding: YaFSDP shards entire layers for efficient communication and reduced redundancy, minimizing memory usage across GPUs. - Buffer pre-allocation: YaFSDP pre-allocates buffers for all necessary data, eliminating inefficiencies. This method uses two buffers for intermediate weights and gradients, alternating between odd and even layers. Using CUDA streams, YaFSDP effectively manages concurrent computations and communications. Furthermore, the method ensures that data transfers occur only when necessary and minimizes redundant operations. To optimize memory consumption, YaFSDP employs sharding and efficient buffer use while reducing the number of stored activations. Comparatively, YaFSDP has demonstrated a speedup of up to 26% over the standard FSDP method and can facilitate up to 20% savings in GPU resources. In a pre-training scenario involving a model with 70 billion parameters, using YaFSDP can save the resources of approximately 150 GPUs monthly. For those interested in implementing this method, Yandex has made it open-source and available on GitHub: https://lnkd.in/dTQnU6-w

WOW !!!! Microsoft just open-sourced one of the most significant papers of 2024. 'bitnet.cpp'. 1-bit LLMs !!!! Basically, it means you can run a 100B param model on your local device (single CPU) highly quantized with BitNet b1.58. These LLM weights are represented by just 1 bit. In-short, instead of the usual 32 or 16 bits for storing weight parameters, 1-bit LLMs use just a single bit (0 or 1) per weight which massively cuts down on your memory needs. Just for an eg. a 7b model would normally require ~ 24-26 GB. with 1-bit, its < 1 GB (0.8 to be more precise) !!!! Go Play 😊

How can we start processes faster? With virtual machines running for only a fraction of a millisecond the overhead of spawning a new VMM becomes relevant. Furthermore we often have to start thousands of them to get reliable performance numbers. This seems to be a good area to improve our tooling. 1. My usual approach to start many proceses relies on xargs. Quite simple and pretty much available anywhere as it is mandated by Posix. However, with over 200ms it is one of the slowest approaches to start a thousand processes. Even a simple loop in the shell is faster. 2. A search for a better tool led to xjobs. Less features and the use of the vfork() system-call reduces the runtime by more the 3.3x. Still, only one-third of the CPUs are utilized. 3. Using a single thread to create processes turned out to be the bottleneck here. Running xjobs on each core gives us another 1.9x speedup. We are down to 32.1ms. 4. The echo from coreutils is neither the smallest nor the fastest payload to use. Replacing it with our own minimal re-implementation called echo.pico drops it to 10ms - a 3.2x speedup. 5. For the last step we put everything into vfork.pico - a non-std Rust tool. Since we neither have to read from stdin nor duplicate file descriptors we can be faster than xjobs. With this we get the final 6.1ms or another 64% gain. Overall this improved our tooling by 33x.  Now back to optimizing virtual machines. #lessismore

DYK - By using NVIDIA Triton Inference Server, NIO successfully streamlined its image preprocessing and postprocessing pipeline to enhance efficiency and reduce network transmission. Learn more > https://nvda.ws/3JAbt5K

DYK - By using NVIDIA Triton Inference Server, NIO successfully streamlined its image preprocessing and postprocessing pipeline to enhance efficiency and reduce network transmission. Learn more > https://nvda.ws/3JAbt5K