I'm thrilled to share that my team and I have had our paper accepted at the ACM, Association for Computing Machinery #IVA2024 conference! Our technical paper, "Estuary: A Framework For Building Multimodal Low-Latency Real-Time Socially Interactive Agents," introduces #Estuary, our new open-source framework designed to empower conversational agents with bleeding-edge AI technologies. In simple terms, imagine real-life Pokémon or human characters you can interact with using a high-tech AR headset like the #AppleVisionPro, connected to Estuary. These characters can understand and navigate your environment in real time! Key Features of Estuary: • 𝐇𝐚𝐫𝐝𝐰𝐚𝐫𝐞 𝐀𝐠𝐧𝐨𝐬𝐭𝐢𝐜: Runs on your own computer and is networked over a local area network to empower a multitude of client devices. • 𝐈𝐧𝐭𝐞𝐫𝐨𝐩𝐞𝐫𝐚𝐛𝐥𝐞 𝐅𝐫𝐚𝐦𝐞𝐰𝐨𝐫𝐤: Integrates AI technologies like Speech-to-Text, LLMs, Text-to-Speech, and more to come. • 𝐏𝐫𝐢𝐯𝐚𝐜𝐲 & 𝐒𝐩𝐞𝐞𝐝: Can operate entirely off-cloud for better privacy and faster response times. • 𝐂𝐮𝐬𝐭𝐨𝐦𝐢𝐳𝐚𝐛𝐥𝐞: Choose your combination of powerful AI models without client hardware limitations, dealer's choice! • 𝐒𝐮𝐩𝐩𝐨𝐫𝐭𝐬 Unity: Compatible with devices supporting the #SocketIO protocol. • 𝐅𝐮𝐭𝐮𝐫𝐞 𝐑𝐞𝐚𝐝𝐲: Will soon be capable of streaming video data for advanced multimodal tasks! We hope Estuary becomes a powerful tool for researchers and indie developers alike. Stay tuned for the public release! A huge thanks to my co-authors and advisors: Basem Rizk, Miru Jun, Andres Artze, Caitlín Sullivan, Sharon Mozgai, and Scott Fisher. And, of course, gratitude to all the developers, creatives, and scientists whose open-source contributions made this possible! #AI #TechInnovation #XR #AR #MR #OpenSource #ICT #Research #Polyspatial
Desmond MJ Kim
3mo
That is crazy!!!!!!!! Awesome news Spencer
THIS IS SO COOL! 😁