Will GPT-3 take over software developer jobs?

GPT-3(Generative Pre-trained Transformer 3) is a mega machine learning model, created by OpenAI, can write it’s own op-eds, poems, articles, and even working code, so we are going to discuss whether job of a software developer is safe from GPT-3 or not?

What is GPT-3 ?

GPT-3 is a neural-network-powered language model. A language model is a model that predicts the likelihood of a sentence existing in the world. For example, a language model can label the sentence: “I take my dog for a walk” as more probable to exist (i.e. on the internet) than the sentence: “I take my banana for a walk.” This is true for sentences as well as phrases and, more generally, any sequence of characters.

Like most language models, GPT-3 is elegantly trained on an unlabeled text dataset (in this case, the training data includes among others Common Crawl and Wikipedia). Words or phrases are randomly removed from the text, and the model must learn to fill them in using only the surrounding words as context. It’s a simple training task that results in a powerful and generalizable model.

The GPT-3 model architecture itself is a transformer-based neural network. This architecture became popular around 2–3 years ago, and is the basis for the popular NLP model BERT and GPT-3’s predecessor, GPT-2. From an architecture perspective, GPT-3 is not actually very novel!

Why is it so special ?

GPT-3 is really big. With 175 billion parameters, it’s the largest language model ever created (an order of magnitude larger than its nearest competitor!), and was trained on the largest dataset of any language model. This, it appears, is the main reason GPT-3 is so impressively “smart” and human-sounding.

But here’s the really magical part. As a result of its humongous size, GPT-3 can do what no other model can do (well): perform specific tasks without any special tuning. You can ask GPT-3 to be a translator, a programmer, a poet, or a famous author, and it can do it with its user (you) providing fewer than 10 training examples. Damn.

This is what makes GPT-3 so exciting to machine learning practitioners. Other language models (like BERT) require an elaborate fine-tuning step where you gather thousands of examples of (say) French-English sentence pairs to teach it how to do translation. To adapt BERT to a specific task (like translation, summarization, spam detection, etc.), you have to go out and find a large training dataset (on the order of thousands or tens of thousands of examples), which can be cumbersome or sometimes impossible, depending on the task. With GPT-3, you don’t need to do that fine-tuning step. This is the heart of it. This is what gets people excited about GPT-3: custom language tasks without training data.

Now, coming back to the main question

Will GPT-3 replace software developers?

The answer is no it will not, because it is essentially focused on machine learning. It doesn’t have creative thought and problem solving is restricted to “what is known” and what can be projected from data. That may sound like AI but it isn’t because GPT-3 doesn’t understand what it is doing - there is no conscious process involved - which severely limits creativity.

Software engineering covers an entire process, from gleaning customer requirements, constructing interfaces, building algorithms and integrating a (software) solution with hardware and IT infrastructure (as required). In a nutshell, it’s messy! and GPT-3 cannot handle that.

But coding is a different matter entirely. Machines like GPT-3 will become successful in repetitive or derivative software - i.e. that which is not pioneering or ground breaking. You want a web page and server managing database and security? Sure, GPT-3 should be able to write it for you. You want a program to predict the stock-market based on trends and historical data? No problem! So yes, changes are coming - some big ones, but that is the nature of life itself.