The emerging usability of ChatGPT in software development

Interesting tool or indispensable resource?

According to industry analysts, there are many organizations experimenting with the ability of ChatGPT and similar AI chatbots to assume a valuable role within software development projects. One of these roles, they said, revolves around using ChatGPT to introduce developers to new and unfamiliar programming languages.

"ChatGPT is good at providing clear, natural-language suggestions or for getting started with and using a new programming language," said Diego Lo Giudice, principal analyst with Forrester Research. "But [developers] also use it to generate code from English prompts to document existing code, to get a clear explanation of a program or to translate from a programming language to another -- asking about APIs, libraries, parameters to use in coding and more."

Lucas Dunker, a Northeastern University student and software developer working as a Harvard Medical School software engineering co-op, uses ChatGPT in his programming workflow, though he admits this is mainly restricted to simpler tasks that don't involve much complicated work.

"We use it just for general creative advice, maybe for naming a project or naming variables," Dunker said. "We don't really use it for a lot of our actual code because a lot of our codebases have already been built. They're usually a little too complex for ChatGPT to add to or do much to."

Emerging possibilities but healthy skepticism

By giving it access to programming language resources, Lo Giudice estimated that ChatGPT can currently work with over 50 programming languages. And the more data or code you train generative AI with, the better it works.

"All of the most popular languages are good for ChatGPT: COBOL, Java, JavaScript, TypeScript, Rust, C, C++, Python, R and SQL," Lo Giudice said. "But it will also generate on the more obscure programming languages. You will just have to pay more attention to the output."

Some developers have found success combining code produced by ChatGPT with the native code of their own applications. Since ChatGPT doesn't reside within an organization's local IDE, however, developers can only cut and paste the ChatGPT-generated code into an existing codebase. This might seem inconvenient for developers that don't normally like to leave their IDE when they are coding but is a functionality that tools like GitHub Copilot should help developers attain, Lo Giudice said. However, he warned that a lack of control over which libraries ChatGPT chooses to link to could get developers in trouble if they don't pay attention.

"Developers would have to pay attention to the library ChatGPT [pulls in]. … They might be old or [incompatible with] the rest of the code," Lo Giudice said. "Since it is currently trained on data up to 2021, it won't know about the latest, very new libraries."

For some, the hope is that, once developers acquire some experience working with ChatGPT-generated code, issues like this become less of a concern. However, Dunker is certainly skeptical when it comes to trusting AI to make rational decisions regarding what developers should do with their code.

"If you have a preexisting application and you ask ChatGPT, 'Is there anything that I can do to improve on this?' it may give you some things, but they may not always be accurate. So, building on preexisting applications is fairly difficult. If you're starting a new project, it might be very helpful to start with tech UPC [Unified Parallel C] code and say, 'Here's a boilerplate, and I'd like to improve on it.' But to do it the other way around, at least anecdotally, has been very difficult."

The observed limitations of ChatGPT

Perhaps the biggest challenge for developers working with ChatGPT is the need for strong discipline when it comes to clarifying when segments of code came specifically from ChatGPT. That includes ensuring those specific segments of code are thoroughly reviewed and tested, Lo Giudice said.

Developers should also beware of assuming all code provided by ChatGPT is accurate, Dunker cautioned. In fact, he personally advocated limiting ChatGPT to the realm of template code, boilerplate logic or basic algorithms.

"ChatGPT is very good at generating code that looks accurate, but that does not always mean that it is accurate," Dunker explained, adding that even code that looks syntactically correct may have been written incorrectly from a functional standpoint.

TuringBots may foretell ChatGPT's future

To explain how ChatGPT and similar generative AI will affect developers in years to come, Lo Giudice references Forrester's guidance on TuringBots:

A TuringBot is an AI-powered software that augments developers' and application development teams' automation and semi-autonomous capabilities from design to delivery of code and applications.

"We defined this new term in 2020 in honor of Alan Turing," Lo Giudice said. "ChatGPT is a TuringBot that does design, code and unit testing, as well as generating functional tests. Other TuringBots, such as GitHub Copilot or Tabnine, only do coding and unit testing. All of these TuringBots are very powerful, being based on generative AI."

According to Lo Giudice, most TuringBots only focus on one part of the development lifecycle, such as builds, tests or deployments. Longer term, however, he believes TuringBots will expand those capabilities to cover multiple aspects of the process, if not the entire lifecycle.

"As more time goes by, they will pick more steps of the SDLC, either because TuringBots will collaborate or because -- like ChatGPT seems to do -- being based on an LLM [large language model] can do more than one SDLC step -- not all and not all well yet."

Of course, organizations need to establish new practices on how to prompt technologies such as ChatGPT to get the best out of them, Lo Giudice said. Development teams will work with AI as companions, manage them and scale their use.

"Short term, the use will be ad hoc and experimental," Lo Giudice said. "It will likely not be on scale in organizations until they develop proper governance to diminish risk."