The Data is In. Here’s How Generative AI is Reshaping Software Engineering.
By Kyle Lacy
Generative AI is the latest shockwave to transform software engineering. Our industry has spent a year and a half experimenting with generative AI coding tools, building new AI features, and preparing for a future that will undoubtedly be disrupted by AI’s advances. But despite all this experimentation, engineers haven’t yet been able to fully measure and understand the ongoing impact of AI.
As part of Jellyfish’s 2024 State of Engineering Management Report, we surveyed more than 600 software engineers and engineering leaders about how they’re implementing AI and what effect it's having on their operations. The answers finally give us the opportunity to assess how much generative AI has changed software engineering — and make some educated guesses around what’s coming next.
Even before the arrival of GenAI coding tools, software engineering was already experiencing seismic shifts. After decades of taking orders from business and product executives, engineering leaders and their teams are now strategic partners who help to guide strategy, grow revenue, and ensure the organization works as efficiently as possible. GenAI is cementing that transformation.
How software engineers are using generative AI
Beyond general sentiment, we wanted to understand specifically how generative AI was impacting the work of engineering teams. Unsurprisingly, a majority of engineers are enthusiastic early adopters of new technologies like GenAI. More than six in 10 respondents (61%) said their team has embraced AI to augment their engineering practices.
So engineering organizations are deploying tools like GitHub Copilot and OpenAI Codex. But what are they actually using it for? Nine in 10 (94%) AI users said that the integration of AI positively influenced their team’s productivity, 81% said AI increases the quality of code, and 81% said AI helps align engineering and business strategy.
Organizations using GenAI are enthusiastic about the benefits: 89% believe it improves engineering and product operations, 87% said it improves the speed and predictability of software delivery, and 82% said it benefits the health and happiness of engineering teams. These responses show a comprehensive halo effect around generative AI tools: it’s not just about writing code more quickly; there are tangible knock-on effects for individual engineers and their organizations as a whole.
Of course, not everyone has embraced AI. Of the non-AI users, 48% said their team has not adopted the tech due to concerns about security, 34% said the reason was a lack of expertise, and 24% said budget constraints had prevented them from using AI. Respondents were also concerned about AI creating too much room for error and ethical implications. Nearly two in five executives who had not embraced AI said they believe AI is a gimmick, but just 3% of engineers and managers shared that opinion.
Why engineering organizations must be able to measure AI efforts
Engineers are finding success using GenAI, and they’re optimistic about the technology’s potential. But sentiment alone isn’t enough. The key to driving ROI from generative AI initiatives is measurement.
Engineers, their managers, and executives agree that metrics are important for measuring quality and alignment with organizational goals. The right metrics ensure that engineers are given impactful tasks and see the value in their work. Moving forward, companies must be able to objectively measure not only their team’s adoption of AI but — more importantly — the technology’s impact on building better products and shipping them faster. Jellyfish’s new GitHub Copilot Dashboard allows engineering teams to start tracking these metrics in real-time, from the percentage of engineers using AI tools to the percentage of AI suggestions being accepted. A majority (58%) of respondents to this year’s SEMR said that they expect generative AI to help alleviate burnout by automating repetitive tasks. It’s now time for engineering teams to start tracking the results of their AI experiments and making adjustments accordingly.
For our complete insights on generative AI adoption and impact, read the full 2024 State of Engineering Management Report. The survey offers insights now just around AI, but also around the role of engineering teams in their organization, the always-changing hiring market for engineering talent, and alignment challenges between company executives and individual engineers.