Quotes from Stephen Wolfram

The most important precedents deal with the whole idea of symbolic programming - the notion of setting up symbolic expressions that can represent anything one wants, and then having functions that operate on both their structure and content.

There are a few very small incompatible changes - I really doubt most people will ever run into them.

Well, the first thing to say is that we've worked hard to maintain compatibility, so that any program written with an earlier version of Mathematica can run without change in 3.0, and any notebook can be converted.

So the thing I realized rather gradually - I must say starting about 20 years ago now that we know about computers and things - there's a possibility of a more general basis for rules to describe nature.

So the thing I realized rather gradually - I must say starting about 20 years ago now that we know about computers and things - there's a possibility of a more general basis for rules to describe nature.

What will limit us is not the possible evolution of technology, but the evolution of human purposes.

It's always seemed like a big mystery how nature, seemingly so effortlessly, manages to produce so much that seems to us so complex. Well, I think we found its secret. It's just sampling what's out there in the computational universe.

Von Neumann was in many ways a traditional mathematician, who (like Turing) believed he needed to turn to partial differential equations in describing natural systems.

I'm committed to seeing this project done. To see if within this decade we can finally hold in our hands the rule for our universe, and know where our universe lies in the space of all possible universes.

Whatever input it's given, the neural net is generating an answer. And, it turns out, to do it in a way that's reasonably consistent with what humans might do. As I've said above, that's not a fact we can "derive from first principles". It's just something that's empirically been found to be true,

But it's notable that the first few layers of a neural net like the one we're showing here seem to pick out aspects of images (like edges of objects) that seem to be similar to ones we know are picked out by the first level of visual processing in brains.

When we make a neural net to distinguish cats from dogs we don't effectively have to write a program that (say) explicitly finds whiskers; instead we just show lots of examples of what's a cat and what's a dog, and then have the network "machine learn" from these how to distinguish them. And the point is that the trained network "generalizes" from the particular examples it's shown.

So how does neural net training actually work? Essentially what we're always trying to do is to find weights that make the neural net successfully reproduce the examples we've given. And then we're relying on the neural net to "interpolate" (or "generalize") "between" these examples in a "reasonable" way.

Particularly over the past decade, there've been many advances in the art of training neural nets. And, yes, it is basically an art.

Here's what happens if one repeatedly "applies the model"—at each step adding the word that has the top probability (specified in this code as the "decision" from the model): What happens if one goes on longer? In this ("zero temperature") case what comes out soon gets rather confused and repetitive:

Neural nets—perhaps a bit like brains—are set up to have an essentially fixed network of neurons, with what's modified being the strength ("weight") of connections between them.

Not surprisingly, this is nonsense.

Or you could do what is the essence of theoretical science: make a model that gives some kind of procedure for computing the answer rather than just measuring and remembering each case.

There's nothing particularly "theoretically derived" about this neural net; it's just something that—back in 1998—was constructed as a piece of engineering, and found to work. (Of course, that's not much different from how we might describe our brains as having been produced through the process of biological evolution.)

We somehow want all the 1's to "be attracted to one place", and all the 2's to "be attracted to another place". Or, put a different way, if an image is somehow "closer to being a 1" than to being a 2, we want it to end up in the "1 place" and vice versa.

And—as we'll discuss later—these weights are normally determined by "training" the neural net using machine learning from examples of the outputs we want.)

Ultimately, every neural net just corresponds to some overall mathematical function—though it may be messy to write out. For the example above, it would be: The neural net of ChatGPT also just corresponds to a mathematical function like this—but effectively with billions of terms.

This short book is an attempt to explain from first principles how and why ChatGPT works. In some ways it's a story about technology. But it's also a story about science.

The first thing to explain is that what ChatGPT is always fundamentally trying to do is to produce a "reasonable continuation" of whatever text it's got so far, where by "reasonable" we mean "what one might expect someone to write after seeing what people have written on billions of webpages, etc." So