Artificial Intelligence 2.0: The Language Wars are coming

There have been major technological breakthroughs recently, such as CRISPR-cas9 gene-editing, AlphaFold2 in protein-folding, and the spread of blockchains and cryptocurrency. But it’s a fair bet that natural language processing via AI and Deep Learning will have the most impact on the most people. For, advances like GPT neural networks mean that machines can process language in ways practically indistinguishable from the way humans do.

These language systems give us the first glimpse of Artificial General Intelligence, something that broadly comprehends the real world, as opposed to domain-specific systems (e.g. AlphaGo) that are competent only in narrow domains. This is, literally, AI 2.0.

Language can be dangerous. We have heard of Deep Fakes, fake videos that look like the real thing, but fake text is equally, if not even more, frightening. Some of the scenarios Rajiv Malhotra outlines in Artificial Intelligence and the Future of Power show how the destructive capacity of AI is being harnessed by certain nations and companies, and how their impact is already overwhelming. India’s very sovereignty may be in jeopardy.

The non-profit research entity OpenAI of San Francisco started developing a series of GPTs (Generative Pre-trained Transformers) a few years ago. These are Neural Networks that don’t need human training but use Deep Learning techniques to crunch gigantic amounts of content to discover their own rules of, in this case, language. 

Neural networks imitate human brains, which have billions of neurons. Each generation of GPTs has increased the number of nodes in the model: GPT-3, released in June 2020, uses 175 billion parameters, about 100 times more than predecessor GPT-2.

What this means in practice is that GPT-3 has built its own statistical model of language; that is, it knows to a fair degree of accuracy what comparable statements in real content look like. It can reproduce the kinds of content humans produce, including their nuances, styles, flights of fancy, quirks, etc. So GPT-3 can mimic real human content. For example, given enough samples of Vikram Seth’s writing, it can produce output that will be indistinguishable from Vikram Seth’s.

The cost of training and operating such systems is astronomical. They are trained on diverse content from books, web pages, journals, academic papers, chat logs, github repositories, legal and medical texts, patent documents, computer science and physics troves, and so on. For example, an openly available large dataset, called Pile™, is 1.25 Terabytes of text from 22 diverse sources, so that it reflects the eclectic nature of language tasks a model will encounter, including producing computer code.

There are, however, at least five huge problems with something like GPT-3.

The first is that even though GPT-3 was explicitly meant to be open-source, by some strange twist of fate, OpenAI sold exclusive rights to Microsoft for billions of dollars. That means that, suddenly, it is not open source in the normal sense of the term. Only Microsoft has access to source code. Fortunately, an effort named GPT-neo from Eleuther.AI, is attempting to create a truly open-source GPT alternative. 

The second problem is that this system only understands English right now. Although it is possible that it can be trained to understand, say, Sanskrit, it is not clear that Microsoft would have the desire to release the source code for that to happen. In other words, even though it would really help Indians if GPT-3 could help analyze the large corpus of Sanskrit, such an effort is unlikely unless Microsoft can be persuaded to permit such use of the product.

The third problem is how we can be fooled by our own expectations. We tend to forget that GPT-3 has no real understanding of the semantic meaning of any text, and is a mere statistical prediction engine of what goes with what (albeit a sophisticated one). It is only correlation, but it appears to be reasoning. Subhash Kak has suggested they cannot, and will not, be conscious, in Why a computer will never be truly conscious; we must not assume they somehow are.

This issue includes, paradoxically, the question of inbuilt bias. The selection of datasets has frequently led to concerns that biases are internalized by the AI. There may be subtle biases because only certain perspectives are included, or because the texts may have assumptions that are misogynistic or racist, for example. Diversity in data sources can mitigate this, although GPT-3 too has raised questions of ethics and what kind of oversight is needed.

The fourth problem is that of deep fakes. Video is now beginning to appear that is uncannily close to reality, but in fact is manufactured. Similarly it is possible for GPT-3 to synthesize text that is essentially indistinguishable from the real thing. A text prepared by the program after being exposed to a little of your writing will be very much like what you would write. How can you prove you didn’t actually write it (it could be inflammatory, seditious, etc.)?

The fifth problem is that of creativity on an abstract level, and of real jobs in concrete terms. If GPT-3 can reproduce Kazuo Ishiguro’s style with accuracy, do we need a Kazuo Ishiguro in the first place? Isn’t it enough to have the simulacrum? What, then, happens to true creativity, the soaring flights of the human intellect? Are we obsolete, then?

Consequently, there is the question of jobs. Automation and AI/ML are already making some jobs obsolete; GPT-3 could make lawyers jobless because it can write legalese from a simple, layman explanation of an issue (and the reverse). It can already develop real computer code from a brief description of the expected output (say user interfaces). Does this mean that paralegals and software developers will find their jobs disappearing?

Undoubtedly, given the ingenuity of the researchers in the field, we will see ways in which these problems can be overcome; and that will mean that finally we have a path to that long-elusive dream: generalized Artificial Intelligence.