Theory Ventures’ Post

The internet today is made for human consumption. 🤖 As soon as you have an AI agent utilize a website, you realize how poorly designed the internet is for automation. I know because I tried. A lot. 🕸 In the future, a majority of online transactions will happen through agents and the internet needs to adapt accordingly. 🖥 Every website will have both a human-centric view (i.e. what exists today) and an agent-centric view. The agent-centric view of each website will consist of 3 things: 1. An embedding space (information layer) 2. An API (action layer) 3. An agent Business owners asking "Why do I need an embedding space?" will be similar to business owners asking "Why do I need a website?" back in the 90's. 📜 Read more in my latest blog!

WebVoyager: Transforming Web Agents with LMM Power #AI #artificialintelligence #Collaborativeresearch #evaluationprotocol #GPT4Vsmultimodalcomprehension #Introducing #LargeMultimodalModel #Limitations #llm #machinelearning #Software #webagent #WebVoyager

Thank you Nick Pogrebnyakov and Digital Nova Scotia for putting on yesterday's 'Adding AI to Your Company’s Products' presentation. It was my first time at The PIER at the Seaport, great spot! Nick, your presentation gave some great pointers on how to get started with AI in our products, from the business side to the tech stuff. Can't wait to look at those resources you mentioned and find some ways to add AI to my web solutions! #AI #Web #Dev

Project: A comfortable group chat interface Time to build desktop and mobile html/css: ~7 minutes Time saved: 3 - 5 hours Tool used: ai.Dev-FE on NoApp.ai

Here's a article where you'll gain insights on integrating and utilizing Google AI’s Gemini image input-based model within your React application.  https://lnkd.in/gJ3MXhEK

🤩 I just published the second part of my blog "Google AI Gemini Image Input API in Web using React 🤖 (Part# 2)". This time, we dive deeper into the practical implementation of Gemini within a React application with image-based models. Learn how to integrate and utilize Google AI’s Gemini image input-based model within your React application. Check out the article to gain insights on extracting information from Google AI by submitting a specific image through this integration. Read the blog here: https://lnkd.in/dF-QZXtj #GoogleAI #React #Gemini #WebDevelopment

Anthropic's Claude 3.5 Sonnet is taking a big lead in AI Usability with yesterday's announcement of automatic test case generation via the Anthropic Console. The built-in prompt generator now easily tests, evaluates, and compares results. Using the prompt generator, you explain the task you want to achieve, and Claude creates high-quality prompts for you. This decreases the need for users to become expert prompt engineers. In my opinion, prompt engineering is analogous to HTML/CSS development during the internet revolution. No one manually develops in HTML/CSS anymore, automated solutions such as Squarespace do this for you. Eventually, I believe we will not need to be expert prompt engineers either. The Anthropic Console allows you to define variables for your prompts by providing an example input such as an email or SMS template. The Console can automatically generate test cases of your test prompts based on the input you defined. The test cases are automatically executed, and a human can evaluate the accuracy of the results. Tweak the prompt and re-execute the tests to see the impact of your change quickly. We are rapidly moving toward a future in which humans describe the task they want to achieve and provide sample inputs, and AI writes the code, generates test cases, and executes them for humans to evaluate. #artificialintelligence

Real-time Object Detection in the Browser with BlazeFace! BlazeFace is a lightweight and advanced model from Google AI, designed to deliver high performance in web and mobile applications. 🔍 Learn more and explore the possibilities here: https://lnkd.in/dpYSmimH #AI #MachineLearning #ComputerVision #BlazeFace #WebDevelopment #RealTime #Innovation

🚀 Personal assistants of the future will be able to operate computers just like humans — by controlling user interfaces. To help make this vision a reality, we are excited to introduce AndroidWorld, a new benchmark for building and evaluating computer control agents. 🌍📱 Why AndroidWorld? Last year, we released AitW (https://lnkd.in/eQJpWzDa), a large-scale dataset for Android control collected from human experts during the pre-LLM era. However, while creating this, we discovered that to effectively evaluate automation agents, we need interactive, real-world environments. AndroidWorld is an open environment that offers: 📝 116 diverse tasks across 20 real-world apps 🎲 Dynamic task instantiation for millions of unique variations 🏆 Durable reward signals for reliable evaluation Key features of AndroidWorld: 🌐 Access to millions of Android apps and websites 💾 Lightweight footprint (2 GB memory, 8 GB disk) 🔧 Extensible design for easy addition of new tasks and benchmarks 🖥️ Integration with MiniWoB++ web-based tasks To evaluate AndroidWorld, we built M3A, a new agent optimized for Android. M3A can operate dozens of apps, showcasing the potential of multimodal models like Gemini and GPT-4 to create agents out of the box. Our best-performing agent achieves a 30.6% success rate zero-shot, compared to 80% for humans, demonstrating significant room for improvement. 🎥 Every time a task is created in AndroidWorld, it is dynamically instantiated, controlled by a random seed, similar to MiniWoB++. We found that agents are highly sensitive to these random seeds, impacting their performance. 🔗 Learn more about AndroidWorld and M3A: 📄 Paper: https://lnkd.in/eUUDwD83 💻 GitHub Repo: https://lnkd.in/e2_VWNXg 🌐 Project page: https://lnkd.in/eF7K8Snj P.S: We also tested a Web Agent on AndroidWorld to explore cross-domain automation. While web agents can solve Android tasks, they significantly lag behind Android-optimized agents like M3A. This highlights the importance of developing domain-specific agents for optimal performance. cc: Divya Tyam, Sarah Clinckemaillie, YIFAN CHANG, Marybeth Fair, Oriana Riva, Robert Berry, William Bishop, Alice Li, Folawiyo Campbell-Ajala, Wei Li, #AI #MachineLearning #AndroidWorld #Automation #Research #ArtificialIntelligence #Benchmarking #AIResearch #ComputerControlAgents

🚀 Embracing the Future of Web Browsing with AI! 🌐 I’m excited to share my experience with WebVoyager, an advanced vision-enabled web-browsing agent that's poised to revolutionize web automation and data scraping. Developed by He et al., this tool seamlessly navigates web pages by controlling both the mouse and keyboard, automating complex tasks with ease. Key Features: Visual Annotations: Leverages Set-of-Marks-like image annotations to interact with web elements efficiently. Full Browser Control: Automates web interactions by controlling the mouse and keyboard. ReAct Loop Architecture: Uses a reasoning and action loop to determine the next steps based on annotated screenshots. How It Works: Annotated Screenshots: Analyzes browser screenshots with annotations to decide the next action. Comprehensive Toolset: Equipped with tools for clicking, typing, scrolling, waiting, navigating back, and searching via Google. Multi-Modal Model: Utilizes GPT-4V to interpret and decide on actions, ensuring accurate web interactions. Why It Matters: WebVoyager represents a significant leap in automating intricate web tasks, bridging the gap between human intuition and machine precision. This tool can revolutionize web scraping, automated research, and much more, making it a valuable asset in the AI and web browsing landscape. I believe WebVoyager is a glimpse into the future of web automation, offering unprecedented efficiency and intelligence in handling web-based tasks. Exciting times ahead! 🔗 Explore WebVoyager on GitHub https://lnkd.in/dxVF9DVH #AI #Automation #WebScraping #TechInnovation #MachineLearning #WebAutomation #ArtificialIntelligence #FutureOfTech #AIResearch #WebTech #LangChain #WebVoyager #Innovation #TechTrends #AIAdvancements Feel free to connect if you’re interested in discussing this groundbreaking technology further! 🌟