CMU MLSP

Congratulations Dr. Ankit Shah!

🌟New image tokenizer for MLLM: ImageFolder 🚀 Imagefolder 🚀 generates images in just 10 steps ⚡⚡ with much shorter token length💡, fewer costs💡, and better quality💡 compared to previous tokenizers. To summary: Imagefolder🚀 achieves significant improvement compared to SOTA VAR, 3.30 v.s. 2.60 (ours) FID 📈 for ~300M generator, 680 v.s. 265 (ours) 📈 Token length, 11.3 v.s. 58.0 (ours) 📈 Linear probing accuracy, 4 : 1 (ours) ⚡Training and inference cost, 🔗 Project: https://lnkd.in/dkwciY8D 📑 Paper: https://lnkd.in/dM698ZDk 💻 Code: https://lnkd.in/dWrsbNFP 👀 Our results demonstrate significant improvements, and we will be sharing further findings and releasing checkpoints soon. #LLM #Autoregressive #Generation #Tokenizer #Research

🚀 Excited to share my very first Medium article! 🎉 I’ve just published an article titled "Understanding Encoder-Only Architectures in Transformers: Key Pre-Training Tasks Explained". In this piece, I dive into the intricacies of encoder-only transformer architectures, breaking down the key pre-training tasks that power many of today’s advanced models. If you’re curious about the technical underpinnings of transformers or looking to deepen your understanding of how these models work, this article is for you! This piece stems from my research at the CMU MLSP lab. Check it out and let me know what you think. Feedback and discussions are always welcome! 😊 #AI #MachineLearning #NLP #Transformers #ArtificialIntelligence #Medium

In this paper, we are investigating the hypothesis that the spoken utterance carries more information than the written transcript of what was spoken, and that this difference can result in different take-aways, which would manifest as differences in the summaries derived from the two. Although all current evaluation methodologies are text-based, and are naturally biased towards text- (i.e. transcript)-based summaries, we nonetheless find that the summaries derived directly from the spoken utterance do indeed provide different, and in some instances superior interpretations than those derived from the text transcription.

Excited to share our paper accepted at #ACL2024 Main conference titled "Speech vs. Transcript: Does It Matter for Human Annotators in Speech Summarization?" with Suwon Shon Mark Lindsey Hira Dhamyal Rita Singh, and Bhiksha Raj. In this paper, we take the first steps towards understanding how humans may annotate for speech summarization by isolating different contributing factors. Humans can summarize speech when it is presented by (a) listening to the speech directly and then constructing the summary based on this, or (b) reading a textual transcript of the content spoken in the recording, and using that to construct the summary. It is unclear whether summaries that result from these two different approaches are different or similar. In this paper, we try to examine differences in resulting human summaries arising from: (a) different source modalities ( speech or transcript) (b) errors in the source modality (manual transcript or automatic transcript with errors) , and (c) expertise of human annotators. We collect data from multiple human annotators and devise a range of metrics to holistically assess differences between summaries across multiple dimensions. We find interesting differences between speech-based and transcript-based summaries from our data collection and analysis. Transcript-based annotation is valuable if transcription errors are minimal and longer and more informative summaries are desired; while speech-based annotation is desired for higher information selectivity, factual consistency, and resilience to transcription errors. Please read our paper and attend our virtual poster session for further discussions.

The beginning of my new journey 🔥 - starting PhD program at Carnegie Mellon University School of Computer Science

Congratulations to our esteemed Prof. Bhiksha Raj for his remarkable achievement! #isca #ai #achievement #speech

I’ll be attending CVPR 2024 in Seattle, USA! I’m looking forward to networking with peers, collaborators, and leaders in computer vision to share ideas and explore potential collaboration! CVPR 2024参加します！よろしくお願いします🙇

Delighted to share that I have just successfully defended my thesis at CMU 🎓! And heartful gratitude to my advisors and committee members Bhiksha Raj, Shinji Watanabe, Rita Singh, Anurag Kumar, and everyone who supported me and loves me during this journey. I will join Google as a Research Scientist next week in New York City, working on speech-related research ♊️. Looking forward to the new chapter ahead!

I have presented OpenFusion on May 15 at Semantic Scene Understanding 2 Session at CC502 at ICRA 2024, Yokohama 😁 Preprint: https://lnkd.in/e979TtKr Code: https://lnkd.in/e_CBZbRt #ICRA2024