Marktechpost Media Inc.

FREE AI WEBINAR: 'The Best Platform for Serving Fine-Tuned Models: Predibase Inference Engine' (October 29 from 10 am - 11 am PT) Register here: https://lnkd.in/gwEKavCM What is covered during this live webinar: ✅ Turbo LoRA and FP8 for 4x throughput: Learn how Turbo LoRA and FP8 significantly increase fine-tuned model throughput. ✅ Observability Tools: Leveraging logs, graphs, and event tracking for real-time monitoring and system health insights. ✅ Autoscaling & Cold Starts: How our autoscaling feature minimizes cold starts and ensures optimal burst capacity to handle spikes in traffic. ✅ Multi-Region High Availability: Ensuring consistent service with multi-region load balancing, automatic failover, and the ability to seamlessly move jobs between clusters. ✅ VPC Deployment: The benefits of deploying within your own private cloud, with complete control over data and infrastructure. Predibase #ai

Agent-as-a-Judge: An Advanced AI Framework for Scalable and Accurate Evaluation of AI Systems Through Continuous Feedback and Human-level Judgments Meta AI and King Abdullah University of Science and Technology (KAUST) researchers introduced a novel evaluation framework called Agent-as-a-Judge. This innovative approach uses agentic systems to evaluate other agentic systems, providing detailed feedback throughout the task-solving process. The researchers developed a new benchmark called DevAI, which includes 55 realistic AI development tasks, such as code generation and software engineering. DevAI features 365 hierarchical user requirements and 125 preferences, offering a comprehensive testbed for evaluating agentic systems in dynamic tasks. The introduction of Agent-as-a-Judge enables continuous feedback, helping to optimize the decision-making process and significantly reducing the reliance on human judgment. The Agent-as-a-Judge framework assesses agentic systems at each task stage rather than just evaluating the outcome. This approach is an extension of LLM-as-a-Judge but is tailored to the unique characteristics of agentic systems, allowing them to judge their performance while solving complex problems. The research team tested the framework on three leading open-source agentic systems: MetaGPT, GPT-Pilot, and OpenHands. These systems were benchmarked against the 55 tasks in DevAI. MetaGPT was the most cost-effective, with an average cost of $1.19 per task, while OpenHands was the most expensive at $6.38. Regarding development time, OpenHands was the fastest, completing tasks in an average of 362.41 seconds, whereas GPT-Pilot took the longest at 1622.38 seconds.... Read the full article: https://lnkd.in/g_cRrfsW Paper: https://lnkd.in/gnPtcEiJ Dataset: https://lnkd.in/eQ82a_DQ Listen to the podcast as well on 'Agent-as-a-Judge': https://lnkd.in/gASqGEDi AI at Meta KAUST (King Abdullah University of Science and Technology) Mingchen Zhuge Changsheng Zhao Dylan Ashley Wenyi Wang Dmitrii Khizbullin Zechun Liu Ernie C.

Salesforce AI Introduces ReGenesis: A Novel AI Approach to Improving Large Language Model Reasoning Capabilities Large language models (LLMs) have revolutionized how machines process and generate human language, but their ability to reason effectively across diverse tasks remains a significant challenge. Researchers in AI are working to enable these models to perform not just language understanding but also complex reasoning tasks like problem-solving in mathematics, logic, and general knowledge. The focus is creating systems that can perform reasoning-based tasks autonomously and accurately across various domains. Read the full article here: https://lnkd.in/eQmh2VhH Paper: https://lnkd.in/eZBMped6

Katanemo Open Sources Arch-Function: A Set of Large Language Models (LLMs) Promising Ultra-Fast Speeds at Function-Calling Tasks for Agentic Workflows Katanemo has open-sourced Arch-Function, making scalable agentic AI accessible to developers, data scientists, and enterprises. By open-sourcing this tool, Katanemo enables the global AI community to contribute and adopt its capabilities. Arch-Function empowers industries like finance and healthcare to build intelligent agents that automate complex workflows, transforming operations into streamlined processes. The Katanemo Arch-Function collection of LLMs is specifically designed for function-calling tasks. These models understand complex function signatures, identify required parameters, and produce accurate function calls from natural language prompts. Achieving performance comparable to GPT-4, Arch-Function sets a new benchmark for automated API interactions. Built around a 3-billion parameter model and hosted on Hugging Face, it supports flexible APIs, ensuring seamless integration into enterprise software. Arch-Function is optimized for speed and precision, completing tasks in minutes that previously took hours while effectively adapting to dynamic requirements... Read the full article here: https://lnkd.in/ggWPqWzt Model Card on Hugging Face: https://lnkd.in/g4ytXkxN Katanemo Michael van Dijken

Google AI Researchers Propose ‘MODEL SWARMS’: A Collaborative Search Algorithm to Flexibly Adapt Diverse LLM Experts to Wide-Ranging Purposes There is a need for flexible and efficient adaptation of large language models (LLMs) to various tasks. Existing approaches, such as mixture-of-experts (MoE) and model arithmetic, struggle with requiring substantial tuning data, inflexible model composition, or strong assumptions about how models should be used. These limitations call for a methodology that can adapt LLMs efficiently without extensive tuning or restrictive assumptions, especially in low-data settings. Read the full article: https://lnkd.in/eeedWAe2 Paper: https://lnkd.in/eWqXggWQ

AutoDAN-Turbo: A Black-Box Jailbreak Method for LLMs with a Lifelong Agent Large language models (LLMs) have gained widespread adoption due to their advanced text understanding and generation capabilities. However, ensuring their responsible behavior through safety alignment has become a critical challenge. Jailbreak attacks have emerged as a significant threat, using carefully crafted prompts to bypass safety measures and elicit harmful, discriminatory, violent, or sensitive content from aligned LLMs. Read the full article: https://lnkd.in/et4h96mW Paper: https://lnkd.in/exapYqRp

Neural Magic Unveils Machete: A New Mixed-Input GEMM Kernel for NVIDIA Hopper GPUs Neural Magic introduces Machete: a new mixed-input GEMM kernel for NVIDIA Hopper GPUs, representing a major advancement in high-performance LLM inference. Machete utilizes w4a16 mixed-input quantization to drastically reduce memory usage while ensuring consistent computational performance. This innovative approach allows Machete to reduce memory requirements by roughly 4x in memory-bound environments. When compared to FP16 precision, Machete matches compute-bound performance while greatly improving efficiency for memory-constrained deployments. As LLMs continue to expand in scope, addressing memory bottlenecks with practical solutions like Machete becomes essential for enabling smoother, faster, and more efficient model inference. One of Machete’s key innovations lies in its technical implementation. Built on CUTLASS 3.5.1, Machete leverages the wgmma tensor core instructions to overcome compute-bound limitations, resulting in faster model inference. It also incorporates weight pre-shuffling, which allows for faster-shared memory loads, effectively mitigating bottlenecks that typically arise in large-scale LLMs. This weight pre-shuffling mechanism optimizes shared memory by allowing 128-bit loads, increasing throughput and reducing latency. In addition, Machete has improved upconversion routines that facilitate efficient conversion of 4-bit elements to 16-bit, maximizing tensor core utilization. Together, these innovations make Machete an effective solution for improving LLM performance without the overhead typically associated with increased precision or additional computational costs... Read full article here: https://lnkd.in/gZcQ4Xch Neural Magic

Google AI Research Introduces Process Advantage Verifiers: A Novel Machine Learning Approach to Improving LLM Reasoning Capabilities Researchers from Google Research, Google DeepMind, and Carnegie Mellon University have introduced an innovative approach to overcome these limitations by developing Process Advantage Verifiers (PAVs). These verifiers offer step-level rewards that measure the progress of the reasoning process instead of just assessing the outcome. PAVs are designed to evaluate each step in the reasoning trace based on how much it improves the likelihood of producing a correct solution. This method contrasts with traditional PRMs that focus on immediate correctness, allowing the model to learn from steps that may not directly lead to the correct answer but increase the chances of success in the later stages of reasoning. The key innovation in PAVs is using a “prover policy,” distinct from the base policy that the LLM is following. The prover policy evaluates progress by measuring the difference in the probability of success before and after a reasoning step. This enables the LLM to explore a wider range of potential solutions, even when early steps do not immediately lead to a correct solution. The research team implemented this by training PAVs to predict “process advantages” for each reasoning step under a prover policy. These advantages are similar to reinforcement learning concepts, where a step is evaluated based on the future expected reward, helping the model to navigate complex problem spaces more efficiently.... Read the full article here: https://lnkd.in/gXUjsFyR Paper: https://lnkd.in/g5dXwfYS Google DeepMind Google

MEGA-Bench: A Comprehensive AI Benchmark that Scales Multimodal Evaluation to Over 500 Real-World Tasks at a Manageable Inference Cost A major challenge in the evaluation of vision-language models (VLMs) lies in understanding their diverse capabilities across a wide range of real-world tasks. Existing benchmarks often fall short, focusing on narrow sets of tasks or limited output formats, resulting in inadequate evaluation of the models’ full potential. The problem becomes more pronounced when evaluating newer multimodal foundation models that need comprehensive testing across numerous application domains. These models require a benchmarking suite capable of evaluating their abilities in various input and output scenarios while minimizing inference costs. Read the full article here: https://lnkd.in/eTN8-N5P Paper: https://lnkd.in/eg-uU49P

Simular Research Introduces Agent S: An Open-Source AI Framework Designed to Interact Autonomously with Computers through a Graphical User Interface Simular Research introduces Agent S, an open agentic framework designed to use computers like a human, specifically through autonomous interaction with GUIs. This framework aims to transform human-computer interaction by enabling AI agents to use the mouse and keyboard as humans would to complete complex tasks. Unlike conventional methods that require specialized scripts or APIs, Agent S focuses on interaction with the GUI itself, providing flexibility across different systems and applications. The core novelty of Agent S lies in its use of experience-augmented hierarchical planning, allowing it to learn from both internal memory and online external knowledge to decompose large tasks into subtasks. An advanced Agent-Computer Interface (ACI) facilitates efficient interactions by using multimodal inputs. The structure of Agent S is composed of several interconnected modules working in unison. At the heart of Agent S is the Manager module, which combines information from online searches and past task experiences to devise comprehensive plans for completing a given task. This hierarchical planning strategy allows the breakdown of a large, complex task into smaller, manageable subtasks. To execute these plans, the Worker module uses episodic memory to retrieve relevant experiences for each subtask. A self-evaluator component is also employed, summarizing successful task completions into narrative and episodic memories, allowing Agent S to continuously learn and adapt. The integration of an advanced ACI further facilitates interactions by providing the agent with a dual-input mechanism: visual information for understanding context and an accessibility tree for grounding its actions to specific GUI elements.... Read full article here: https://lnkd.in/ga5QT94r Paper: https://lnkd.in/gEXSqF8y GitHub: https://lnkd.in/dpsCmgWR Simular Jiachen Yang Ang Li #ai