Waabi’s Post

Excited to share our latest blog post discussing "Toward Robust Multimodal Learning using Multimodal Foundational Models." In this post, we delve into the challenges of incomplete multimodal data in real-world scenarios and present TRML, a framework designed to address scenarios involving modality absence. TRML employs generated virtual modalities to replace missing modalities, aligning semantic spaces for robust multimodal learning. Our approach demonstrates superiority on three benchmark datasets. Read the full post at https://bit.ly/3vQ7q1p.

📢 Exciting News! Our paper, titled ''IMPRINT: Interactional Dynamics-aware Motion Prediction in Teams using Multimodal Context,'' has been accepted to the Transactions on Human-Robot Interaction (THRI) Special Issue on Artificial Intelligence for Human-Robot Interaction. 🤖 This is joint work with Md Mofijul Islam (joint first-author) and my advisor Prof. Tariq Iqbal. In IMPRINT, we introduce a novel multi-agent motion prediction framework that models the interactional dynamics of multiple agents and additional context from multiple data modalities to accurately predict human motion. We further provide an interpretation of how our proposed framework incorporates the multimodal context information from all the modalities during multi-agent motion prediction. If you're keen on diving into the details, you can access the full manuscript here: https://lnkd.in/gwM-_8aT #MotionGeneration #GenerativeAI #HumanRobotInteraction #MultiAgent #MultiModal

I’m happy to share that our paper, “Getting More for Less: Using Weak Labels and AV-Mixup for Robust Audio-Visual Speaker Verification” was accepted to INTERSPEECH 2024! In our paper we show how Deep Metric Learning (DML) can be enhanced through multitask learning techniques to improve the quality of audio-visual speaker representations without increasing model complexity during inference. We also show how noisy data can be leveraged to improve generalization by introducing the GE2E-MM loss as well as AV-Mixup, a multimodal data augmentation technique. Our system boasts state of the art performance on official test splits in VoxCeleb1! It was a great teamwork by myself and Homa Fashandi, PhD, P.Eng. and I’m very excited to be able to present our work in Kos, Greece this September! #selfsupervisedlearning #deepmetriclearning #multimodal #multitask #deeplearning #computervision #speakerverification #dataefficiency

A very talented group of researchers and some very cool work! Briefly: Different neural networks, when initialized with different parameters and trained on different data, develop latent representations that are not standardized. It would be better if they were, so different model components can be stitched together and already perform well on various applications without additional training (zero-shot stitching). This work presents a new and efficient way to standardize latent spaces.

Great news to share: We have had three papers accepted in diverse, well known and impactful conferences in the topics of vision and language and  EEG based brain-computer interfaces !  Let us congratulate all our team members and collaborators for their great effort and creativity. Maria Parelli, Dimitrios Mallis, Markos Diomataris for the paper “Interpretable Visual Question Answering Via Reasoning Supervision” in Int’l Conference Image Processing (ICIP) 2023  Alexandros Benetatos   and  Markos Diomataris  for the paper “Generating Salient Scene Graphs With Weak Language Supervision” in the European Signal Proc. Conf. (EUSIPCO) 2023 ! Christina Sartzetaki , Panagiotis Antoniadis, Nick Antonopoulos, + Agamemnon Krasoulis, Serafeim Perdikis, for the paper entitled  “Beyond Within-Subject Performance: A Multi-Dataset Study of Fine-Tuning in the EEG Domain” in the SMC 2023 ! More to be shared soon! #deeplabai #conferences #paper #SMC2023 #EUSIPCO2023 #ICIP2023 

I am happy to share that my paper has been accepted for publication in the IEEE Open Journal of Control Systems! In the domain of aerial systems, there is an extensive body of literature on path planning and flight control. However, generating fast trajectories for moving aerial vehicles remains a significant challenge, especially when assessing the performance of navigation algorithms. In our paper, we introduce a groundbreaking approach to tackle this issue. Our novel method leverages high-level reinforcement learning to develop fast trajectory generators, offering a new perspective on how to achieve rapid and efficient path planning. Dr. techn. Babak Salamat states, "We believe our results can pave the way for a novel methodology in reinforcement learning, enabling the development of fast trajectory generators." Furthermore, our research rigorously demonstrates the superiority of our approach over classical reinforcement learning policies, particularly policy iteration algorithms. This advancement highlights the potential for significant improvements in the efficiency and effectiveness of trajectory generation in aerial systems. Title: Global Multi-Phase Path Planning through High-Level Reinforcement Learning Abstract: In this paper, we introduce the \textit{Global Multi-Phase Path Planning} ({\ttfamily$GMP^{3}$}) algorithm in planner problems, which computes fast and feasible trajectories in environments with obstacles, considering physical and kinematic constraints. Our approach utilizes a Markov Decision Process (MDP) framework and high-level reinforcement learning techniques to ensure trajectory smoothness, continuity, and compliance with constraints. Through extensive simulations, we demonstrate the algorithm's effectiveness and efficiency across various scenarios. We highlight existing path planning challenges, particularly in integrating dynamic adaptability and computational efficiency. The results validate our method's convergence guarantees using Lyapunov's stability theorem and underscore its computational advantages. Link: https://lnkd.in/eg6-HFGT #AerialRobotics #PathPlanning #ReinforcementLearning #IEEE #ControlTheory #TrajectoryGeneration

Happy to share our new paper, accepted at [ICML] Int'l Conference on Machine Learning, in which we tackle ill-posed learning tasks, uncertainty prediction and conditional density estimation. In this work we address these challenges by exploring the Winner-Takes-All (WTA) training scheme. This approach can be used to train a multi-head network to produce diverse, plausible outputs for each input. One major open problem is evaluating these multiple predictions probabilistically. To address this, we introduce a novel density estimator based on WTA learners. Our estimator shows promising properties, both theoretically in terms of quantization and probabilistic convergence, and experimentally through extensive evaluation on both synthetic and audio data. Check out our paper at https://lnkd.in/e4QesXsq A joint work with David Perera, Cédric Rommel, Mathieu Fontaine, Slim Essid, Gaël Richard and Patrick Pérez Multi-head #neuralnetworks #machinelearning #Uncertainty #DensityEstimation

Ok. I have taken a week to decompress, get a lot of much needed rest, and am ready to go! I have been diving into the SLAM pipeline from the basic mathematics and algorithms used, as well as the various approaches being used today and future directions being discussed. Such a very interesting topic autonomous navigation is! I like to view it not only on its own mathematical, probabilistic merits, but from a comparison to how such is accomplished in animals and insects. For example, using semantics and NNs for visual odometry. What is gained vs its computational costs (hence power, real estate, and component cost). Another angle I'm interested in from my readings of various papers on the subject is how can these theoretical systems be used in industry. Theoretical clean solutions using a single sensor vs the reality of covering all cases in actual application e.g., multi-sensor fusion from a hierarchical, loosely coupled fusion to a tightly coupled solution. Now to decide what ZenoRobotics will offer in the near future in way of products, educational services, blogs, media coverage of related events and conferences, etc.

Introducing Ego-Exo4D — a foundational dataset for research and benchmark suite focused on skilled human activities to support research on video learning and multimodal perception. More details ➡️ https://bit.ly/3RsWkIb The centerpiece of Ego-Exo4D is the largest ever public dataset of its kind comprising both first-person “egocentric” videos and accompanying third-person “exocentric” views of the same task. The dataset is not only multiview, but also multimodal. With data captured using Project Aria, it includes time-aligned seven-channel audio, IMU, eye gaze and much more. Releasing this work together with a consortium led by FAIR and including Project Aria + 14 university partners, we hope it will help to catalyze new work in the research community in video learning.

I am extremely fascinated by this project as it fundamentally transforms the methods of data collection and accelerates the learning pace of AI much more! In my view, the Ego-Exo4D is a pivotal step towards democratizing access to high-level training. By offering advanced, immersive learning experiences, it opens doors for those who might not have the means to access traditional education and training. This innovation holds the promise of substantially narrowing the skill gap, bridging the divide between those who can afford conventional educational pathways and those who cannot. It could be the next big step towards bridging educational equity and inclusivity! What's your opinion?