Retrocausal’s Post

Transcript

Our latest peer reviewed paper will be presented at the European Conference on Computer Vision, ECCV 2024 in Milan, Italy. It describes our novel self supervised approach to effectively aligning video sequences in time using 2D skeleton heat maps. This work has been performed by the research team at Retro Causal Inc. Video sequence alignment involves the temporal matching of frames across two videos, ensuring accurate synchronization. By aligning frames across videos, we can enhance the understanding, analysis, and interpretation of video data. Sequence alignment finds application in various areas, such as identifying abnormal activities or events in a video, classifying actions or activities performed in a video, and dividing a video into meaningful segments based on the underlying temporal structure, enabling better understanding and analysis of video data. Sequence alignment encounters several challenges, such as the high cost of manual annotation and supervised scenarios, ambiguity and fine grained action classification, and the inability to capture temporal context within a video. Overcoming these challenges is crucial for effective alignment techniques. To address these challenges, we propose a novel self supervised method based on 2D skeleton heat maps. Our method effectively and robustly align sequences in time while preserving privacy information which is important for human activity understanding applications. In the initial step of our approach, we extract the heat map representation of a sequence and generate an augmented version to enhance its discriminative capabilities. Heat maps provide a compact representation of the two D skeleton information, enabling efficient processing and alignment. Both sequences are then passed through the attention blocks. The attention mechanism allows our model to focus on relevant parts of the sequence and capture the essential temporal and spatial features. By employing a spatial transformer, we extract the spatial characteristics embedded in the sequences and a temporal transformer is used to capture time based relationships within the sequences. This enables our model to understand the relationships and dependencies between different parts of the sequences, enabling a more comprehensive understanding of their dynamics. The cross attention module matches the frames between the sequences based on their spatial and temporal information, ensuring accurate and coherent alignment across the videos. To incorporate positional information, we add positional encoding before each transformer. Positional encoding provides the model with information about the order and position of sequences. Specifically, the spatial position encoding helps the model understand the relative positions of joints and limbs, and the temporal position in coding enables the network to explicitly consider the temporal ordering. Their representation you obtained from the alignment process is further enhanced using a projection head which projects the learned features into a higher dimensional space represented as Z. This step improves the discriminative power of the model, making it more effective at aligning sequences. The contrastive regression loss is computed as the final step in our approach. This loss function plays a crucial role in training the model to align corresponding frames accurately, while also promoting the learning of discriminative representations. By encouraging the model to align frames with similar spatio temporal context and simultaneously pushing apart frames with different spatiotemporal context, this loss function facilitates precise temporal. Alignment. The 2D heat lamp augmentations are used to generate correspondence labels for self supervised training of our proposed network. This eliminates the need for expensive manual annotation and allows the model to learn alignment directly from the unlabeled data. Our method exhibits significant improvements across all metrics and data sets as compared to previous methods. Experimental results show that our approach achieves state-of-the-art performance in various video understanding tasks. Furthermore, our 2D skeleton based approach exhibits higher robustness against noisy and missing inputs than the 3D skeleton based approach of Casa. Finally, representing skeletons as heat maps makes it convenient to fuse with other grid like modalities such as RGB videos, optical flows and depth maps. Here we extend our model to take both 2D skeleton heat maps and RGB videos as inputs and perform late fusion. Our RGB plus pose model achieves the best performance across all metrics and datasets, advancing the state-of-the-art. Here's an example showcasing the accurate alignment of three videos despite vastly varying viewpoints from the reference video. This example highlights the robustness of our approach to aligning videos with diverse content and perspectives. Thank you for watching. Please read our paper on archive.