Watch this space for more exciting updates 👀
🌍 🥘We are beyond excited to release the “World Wide Dishes” dataset, a text and image evaluation dataset with 765 dishes, with dish names collected in 131 local languages from around the world Video: https://lnkd.in/dRDM_h29 Website:https://lnkd.in/e7jsT5Wf Paper: https://lnkd.in/dcZrPr5f 🌟It is our hope that the research community takes advantage of all the local expertise that World Wide Dishes offers, and incorporate it as an evaluation dataset to support improvement in state of the art generative models.🌟 🌍 🥘 This project is the result of significant community contributions - the data was collected purely through human contribution and decentralised means, by creating a website widely distributed through social networks. This approach was motivated by the desire to empower *anyone* to be able to submit information about their culture - using food as a lens See our contributors here: https://lnkd.in/dyihPRuH 🌍 🥘 Importantly, while our dataset is small and imbalanced, we include rare participation from regions such as 🇩🇿Algeria, 🇨🇲Cameroon, 🇨🇩 the Democratic Republic of Congo, 🇰🇪 Kenya and other countries historically under-represented in large-scale datasets. 🌍 🥘 World Wide Dishes demonstrates a novel means of operationalising capability and representational biases in foundation models such as language models and text-to-image generative models. In the paper, we present evaluation tasks to test both capability and representation bias in SOTA generative models 🌍 🥘 Using WWD as a foundational template for user experience, we demonstrate disparities in performance of common knowledge understanding in LLMs, where the models are unable to accurately generate factual content about the dishes. 🌍 🥘We further investigate disparities in image generation, by demonstrating that the models disproportionately associate negative descriptors with generated images of food from the African continent. We also find that a VQA model can be useful in identifying stereotypes, such as an over association of African dishes and clay/ceramic serving plates 🌍 🥘We enrich these studies with a pilot community review to understand, from a first-person perspective, how these models generate images for people in five African countries and the United States. 🌍 🥘We find that these models generally do not produce image outputs of dishes specific to different regions. This is true even for the US, which is typically considered to be more well-resourced in training data - though the generation of US dishes does outperform that of the investigated African countries. 🌍 🥘 Overall, we find that current SOTA models produce outputs that are inaccurate, culturally misrepresentative, flattening, &insensitive. These failures in capability and representational bias have the potential to further reinforce stereotypes & cultural erasure