Doug Green’s Post

Explore the capabilities of #NVIDIAMetropolis microservices for developing #visionAI applications with customizable, cloud-native APIs. Learn how to leverage APIs to streamline software development, including video streaming, AI-based insights, and analytics, and integrate them into client applications. https://nvda.ws/48KfoYl

There's often confusion about NVIDIA Triton's capabilities with real-time streaming. The answer is, it fully supports them. Just published a new guide on Inferless blog which explains how by integrating SSE with NVIDIA Triton Inference Service you can build real-time AI streaming apps! 🚀 Full guide, demo, and code below. Guide: https://lnkd.in/d5XvdDZk Demo: https://lnkd.in/dD5NAbsC Code:https://lnkd.in/dAEZg63g

Efficiency matters in video engineering, especially when it comes to transcoding. NETINT's Quadra, an ASIC-based transcoder, is engineered to address this need. Our tests show that Quadra not only maintains quality but also improves throughput, offering cost savings in large-scale video encoding, particularly for user-generated content. Whether you work for a content platform, social media site, or streaming service, Quadra can enhance your transcoding capabilities. Read the full article: Transcoding UGC: ASICs Are the Lowest Cost Option ▸ https://lttr.ai/AKmp8

We just released our latest case study on the synergy between CABR, our Content Adaptive Bit-Rate optimized video encoding technology, and ML workflows. In this case study we got to play with the Face Fusion package (link in first comment), and were looking specifically at super efficient AV1 encoding - all the quality in so much smaller files :) https://lnkd.in/dj9rTU4N

An exciting new paper for video to video translation for streaming applications. It runs at 20 fps (512x512 res) on a single A100 gpu. Existing methods generally focus on processing video in short batches of few seconds, causing delays, or generate frames independently which lead to flickering. The paper introduces a feature bank to store features from past frames, ensuring smooth and continuous translation. The bank features are used in two ways to extract information from past frames - by extending attention to attend over the bank features, - by explicitly fusing bank features into current features. The paper also introduces a feature merging approach (inspired from methods on token merging). This allows updating the feature bank with new features while maintaining its size. The paper achieves near real-time throughput (20fps) while maintaining competitive performance with existing works. Curious to learn more? Check out the full paper: https://lnkd.in/gBxgArP7 and this post from the authors: https://lnkd.in/gEq6jayy

How does the Apple Vision Pro and "Spatial Video" change the rules of video production? Garrick Chow, Luna Checchini, and I are investigating this on our new YouTube channel - "Spatial Video Insights". I think the Apple Vision platform can change the way we work with computers. But, fully immersive 3D video production is where I'm most excited. We'll be shooting spatial video, running tests, and learning the best practices for creating this new form of video content. And, of course, we're sharing everything we learn. https://lnkd.in/dbFebhsX

How can spatial computing enable new ways to explore 3D content creation? As part of Episode 2 of SVG NEXT Conversations, leading XR expert and Apple's former senior manager of prototype development Avi Bar-Zeev discusses where XR distribution platforms are headed and how Apple Vision Pro enables the production of unique 3D experiences like the 'Alicia Keys: Rehearsal Room'.

A CODEC solution with GPU(1) #neuralnetwork #AI #AIcodec #codec #Olympic Watching TV(Olympic games) I can see all the noises on the pixels. And I do know that even TV streaming is about the codec algorithm. Now that we, the programmers, have to mind the GPU programming more and more I came up with the idea how we could possibly enhance the TV streaming codec performance by utilizing GPU. -For those who are not quite aware of the definition of codec I will tell you what codec means: It is compressing and decompressing video data so that we don't need to transfer all the video data with the giant pixel size.- Firstly we do need to understand the typical camera moving: zoom-in/out, rotating/transforming etc.. Do the typical codec algorithms really care such? I can not be sure... There are many codec models and they work on the gap between the previous and the current pixel or the nearby pixels (fourier). If one pixel's value can be transferred using the matrix and differentiation methods we can save much more process. Let's start with a simple example where an object image floating on a background image (moving in a certain pattern). We separate those two different images, accumulate 3 or 5 frames and calculate the differentiation and make the matrix result of it. Now all the codec data we would make is: the video duration, one object image and its size, the background color and the equation of the moving pattern! After that we can step to the next difficulty so that we can reach the real video's level. Thank you for reading my article and I will come with the next solution.

🔊 Exciting news in the world of speech recognition! 🎙️ Our latest blog post introduces the Stateful FastConformer with Cache-based Inference for Streaming Automatic Speech Recognition. This paper proposes a highly efficient and accurate streaming speech recognition model based on the FastConformer architecture. The model is designed to eliminate the accuracy gap often seen between training and inference time for streaming models. It's compatible with various decoder configurations and has shown improved accuracy and reduced latency when compared to conventional models. Read the full paper here: https://bit.ly/3NNK4zL [cs.CL]

NVIDIA's DeepStream is a chained beast of Streaming Video Analytics with Artificial Intelligence. Powers: 1. great plugins with available sources that could infer your AI model in any format at optimal GPU utilization, eat every video, and spit out magical insights in the storage of your choice 2. many sample applications that demonstrate usage of those plugins 3. every version makes steady progress towards ease of use and modification (e.g. ServiceMaker) Chains: 1. difficult to single out some specific function from samples as usually it relies on global context or a lot of utility functions 2. the tools build on top of that (e.g. PipeTuner will also be tightly coupled with how sample apps are organized) Join me on a side quest to free some power-ups of DeepStream. In the first episode, we demonstrate how to extract tracking metadata in KITTI format, convert it to MOT format, and apply tracking evaluation to compute key multi-object tracking (MOT) metrics with TrackEval (one of the standard utilities to compute metrics for many popular MOT benchmarks). https://lnkd.in/erXw2MBn Do you have some proposals for which power-ups could be freed for more modular reuse? Drop a comment or a dm.