Amy Heaton-Finch’s Post

An interesting perspective, especially considering the recent push for RCTs in the social sector in Australia.

i don’t agree with all of this critique but it’s a useful provocation. in particular, i agree with the following: “Methods are only as good or bad as their appropriateness for answering particular questions and in relation to how well they are able to help us understand the primary phenomena under study. They are not better nor worse, per se, just more or less appropriate under different circumstances.” i think RCTs are appropriate more often than i suspect Aston does but the general proposition is true, and is a good reason for avoiding “gold standard” language. fwiw, my alternative to “gold standard” is more wordy and goes something like … if the question you want to answer relates to causal inference then a well-conducted FCT that minimises threats to internal validity will usually give a more convincing answer than other methods of causal inference (such as diff-in-diff, instrumental variables, or matching). Eleanor Williams

“RCTs have a lot of problems, and probably more than you think.” 👀 Indeed, check out this excellent encapsulation of their shortcomings.

Value Compass series 4: How to evaluate the values/ethics of your LLMs? 🤔 Value Questionnaire? LLMs might cheat as they know the correct answers! 😂 We should investigate LLMs’ conformity of behaviors to values rather than their knowing and understanding of values! Try our DeNEVIL framework to automatically create model-specific generative test instances to reveal the underlying values of your LLM! 🤗 Paper accepted by ICLR 2024. https://lnkd.in/eTVddjEZ

TREMA lab@ University of New Hampshire is presenting at #SIGIR and #ICTIR : Three works on EXAM++/RUBRIC Evaluation for Retrieve/Generate systems. Shows merit of fully automatic RUBRIC (questions and nuggets): * Naghmeh Farzi and Laura Dietz. "Pencils down! Automatic Rubric-Based Evaluation of Retrieve/Generate Systems." ICTIR. 2024. Deep dive on different question-answering to self-rated answerability approaches: * Naghmeh Farzi and Laura Dietz. "EXAM++: LLM-Based Answerability Metrics for IR Evaluation." LLM4Eval. 2024. Software Resource for experimentation and incorporation of human judges into the process: * Laura Dietz. "A Workbench for Autograding Retrieve/Generate Systems." SIGIR. 2024. Original EXAM Answerabily Metric paper --- now called RUBRIC: * Sander, David P., and Laura Dietz. "EXAM: How to Evaluate Retrieve-and-Generate Systems for Users Who Do Not (Yet) Know What They Want." DESIRES. 2021. Say 'hi' to my student Naghme Farzi, it is her first SIGIR.

Peer support for individuals with major limb loss: a scoping review. https://buff.ly/3vtiiSR Academy members can access this research and more via O&P iQ: https://buff.ly/48MGain

🤖 #RetrievalAugmentedGeneration #RAG is, in many use cases, the most efficient method currently available to expose #LargeLanguageModels #LLMs with specific knowledge. However, the performance can be unsatisfying if the retrieved knowledge is weak. 🧠 This paper https://lnkd.in/e2Qfjnmr presents a retrieval evaluator, #CorrectiveRetrievalAugmentedGeneration #CRAG, that employs different methods to generate a confidence degree for the quality of the knowledge exposed to the LLM. ➡️ This is an example that shows that the future performance of LLMs might not be improved by merely expanding the size of the LLMs. We will also see growing sophisticated architectures of different, smaller, but smartly focused LLMs pushing the performance of LLMs. #AIResearch #KnowledgeIntegration #MachineLearningModels

📈Statistically increase your chances of passing with the Simulated MBE - Take the BARBRI Simulated MBE several weeks prior to your actual exam - Helps prepare you for the mental and physical rigors on exam day - Data scientists have found experiencing the simulation, regardless of your score, statistically increases your chances of passing your bar exam The Pass Predictor: This report tells you where you sit in relation to your peers, in every MBE subject AND subtopic, before the actual exam. PLUS With BARBRI Guided Pass and Ultimate Pass, you’ll get an additional customized report in addition to your Pass Predictor score report. - Provides insight into your subjects and sub-topics level scores vs other students - Insights into your personal strengths and opportunity areas - Specific strategies on how to best utilize your time and optimize your unique results Enroll today https://bit.ly/400offbar

One of the key tasks in succeeding with LLMs is creating the right benchmarks and evaluating their outputs. In many cases, the outputs are complex, and the quality assessments can be highly subjective. The study below shows a pattern we've observed: that when LLMs are used to evaluate outputs of LLMs, there's an important feedback loop influencing the evaluation, questioning whether the evaluation and observation of model outputs can be treated independently. If you're working with LLMs, this study is definitely worth checking out. It sheds light on an important aspect of LLM evaluation that's often overlooked. And it again highlights the importance of a human-centered approach. #LLMs #MachineLearning #DataScience #Evaluation #Benchmarks #HumanCenteredAI

Quantitative Reasoning on the GMAT™ Exam Focus Edition! Knowing what’s tested and using proper preparation to turn your challenges into victories! https://bit.ly/3Gu2Isr