Navigating the AI Frontier: Lessons Learned in Creating Coach for Career Development

I’m taking a moment here on this very special day to share some reflections on the remarkable experiences we’ve had over the past 18 months. Today we officially launched Coach, the result of a tremendous amount of work from a tremendous number of brilliant people. (acknowledgements at bottom!) Launching a world-class AI platform that provides personalized, accessible career guidance at scale as a nonprofit is no small feat, and along the way we’ve discovered some things to be shockingly difficult and others surprisingly easy. This post isn't really about Coach itself (although it's an amazing platform you should check out) instead I'll be focusing on what it's been like to build Coach.

AI Vertigo and the thrilling surrealism of building with AI

The experience of working with LLMs is completely different from other forms of engineering. The technology is so broadly applicable, and so easy to use, that it's hard to find problems that we can't support in some way with LLMs. It becomes quickly dizzying as we abstract out to ever-more-meta altitudes. Need a chatbot? Use an LLM. Chatbot needs an agent for complex tasks? LLMs managing LLMs. An agent to coordinate multiple agents? LLMs managing the LLMs that manage the LLMs. Prompt engineering too slow? LLMs can help. A system to triage chats and ensure quality? LLM it. And how do we verify the system's reliability? LLMs can help. It's "turtles all the way down" where every level of the system is another potential place where AI can help. That's part of what makes it so much damn fun to work with, and it's also why it's so important to stay grounded in the human experience by keeping humans in the loop at every level at all times.

Some of the more concerning revelations

We discovered early in 2023 that despite a huge amount of tooling being built to help companies put LLMs into production, there was virtually zero tooling being built to enable safety and quality monitoring. Without rigorous AI-specific monitoring, you can build an AI chatbot but you'll never know if it's doing a good job or be able to take action when it doesn't meet your standards. With input from AI experts, we spent months building our own tooling, integrating cutting-edge platforms, and testing. Now we've got the systems and practices in place that let me sleep well at night, knowing that we're making not just a great Coach, but a safe and reliable Coach as well. We were willing to spend a tremendous amount of time and treasure to build it, but others will struggle to follow. Using AI responsibly is something you do in practice, not in theory, and better AI safety and quality monitoring tooling in the industry is key to normalizing it. Fortunately, there's been progress in the market, and we're hopeful for the quality tooling ecosystem. Bet here.

Whiplash, but the good kind

The whiplash of finding ourselves suddenly central to a brand new world has been exhilarating. Early on, it became clear how much people wanted Coach. The demand for personalized, accessible career guidance was immense, and we realized how uniquely valuable we could be in this evolving labor market. Our technical strength, our history of large-scale career advising serving millions of learners, and our extensive network and partnerships positioned us perfectly to create Coach in a uniquely collaborative manner. But we knew we couldn't do it alone. Thankfully, a coalition of over 20 organizations formed to collaborate on navigating the complex landscape of AI ethics and career development best practices -- the AI Career Coaching Coalition. Given the success of this coalition model of AI development, we're making further investments in collaborative design: we've joined the EDSAFE AI Alliance for knowledge-sharing, we're convening a committee of career counselors and coaches to advise us, and we continue to run pilots with multiple partners. This work is not only critical -- it grows even more critical as the labor market shifts around us every day.

What's next

Today we're announcing the official launch of Coach, not only to individual learners (who can sign up for free), but also to educational and workforce development institutions. Coach is "feature complete", meaning that it has every feature for learners, job seekers, educators, counselors, and advisors that we believe are necessary for an educational or workforce development institution to get started. To be clear: there's still much work to be done taking Coach to the next level, equipping it with better and better data, and tuning Coach's behavior to be even more engaging and effective. But this launch makes it official: Coach is ready.

Here's what you can do to get involved:

1. Sign up yourself or someone you know at www.aicareercoach.org
2. Re-share this post on LinkedIn, and directly with innovative educational and workforce development programs
3. Upvote our launch announcement on product hunt
4. Tell a journalist, podcaster, or influential person you know about Coach
5. Email us to get involved as a volunteer or advisor

Acknowledgements (because it takes a village!)

Over 200,000 people have been involved in making CareerVillage what it is, and we appear to be on the same path with Coach. Over the past 18 months an incredible array of institutions and individuals have been involved with building or reviewing the Coach system. Our deepest gratitude to Opportunity@Work, Year Up, Generation , Per Scholas , The College Board , Bill & Melinda Gates Foundation, GitLab Foundation , Schultz Family Foundation , Google , Salesforce , ServiceNow , Atlassian , The Frederick A. DeLuca Foundation , Cisco , Twilio , Hewlett Packard Enterprise , THE SOLID FOUNDATION , PwC , AT&T , Verizon , MENTOR , National Association of Workforce Boards , yoonji KIM , Rebecca Gitomer, CPA , Natalie Dunn , Alejandrina Correa Reymond , Laney Emerson , Ebony Staten , Eric Fershtman , Eric Reyes , Abby Lupi , Alexandra Carpenter , Angelie Miranda , Maeve Cannon , Marisa Fitz , Donna Curry , Olutosin Sonuyi , Thelma Boamah , Matías Caporale , Kimberly Rojas Ramírez , Luis Eduardo Mosquera Zapata , Gabriela Golmar , Joshua Adegoke , Gurpreet Lally , Sharyn Grose , Sandro Olivieri , Sarah Shaw, Ricardo Vivas, Byron Auguste , Claire Hartman , Matt Zieger , Aras Jizan , Matt McCann , Scott Gullick , Julia Freeland Fisher , Michael Horn , Jeff Livingston , Kyle Thornton , Shannon Farley , Kevin Barenblat , Fast Forward , Christina Morefield , Elizabeth Dettke , Nick Fellers , Heidi Kraemer , Justin Pollack , Jessica Pliska , Chris Koehler , Frank Chu , Michael Park , Fred Tan , Michelle Holmes , Grace C. , Erin Baudo Felter , Erin Dieterich , Joel Straker , Brigitte Hoyer Gosselink , Gerald Chertavian , McKinsey & Company , Saurabh Sanghvi , Akshaya Kumar Sharma , Kathleen Weiss , Yoko Kono , Allison Danielsen , Clare Bertrand, M.S.Ed , Jobs for the Future (JFF) , American Student Assistance , Lee Waldvogel , Evelyn Van Til , Adomat, Rachel , Emily Anthony , Clarence Wardell III, PhD , Giovana Espejo , Chauncy Lennon , Marie Groark , Jessica García-Kohl , Rebecca Wang , Lyndsay Harris Kyei , Alan Norris , Sarah Koenig , Michelle Chin, CPA , Cariad Shepherd Louis , Karl Rectanus , Jaime-Alexis Fowler , Design Run, Daylight, Sophilabs, Deloitte, GlideFast, Athina AI, Big Brothers Big Sisters Bay Area, University of Florida School of Nursing, AVID, Empower Work, Hats & Ladders, Mission Bit, Mouse, Portal Schools, UpChieve, ASU, Big Brothers Big Sisters Eastern Massachusestts, Braven, CAPS Network, EDSafe AI Alliance, MassHire Berkshire County Workforce Board, MassHire Franklin County Workforce Board, MENTOR California, San Francisco Unified School District, Southwest Technical Education District of Yuma, and the countless other partners, volunteers, and supporters of our work who have helped us along the way! It truly takes a village to do this work.

Please join me in congratulating each and every one of them!