So. In this first part of the talk, I want to show something that might sound super-abstract. But I'll try to show it in a concrete way.

The thing I want to show is, it matters how we talk about flourishing.

How we conceptualize what we want out of life... Like, what vocabulary we use for talking about it, makes a huge difference.

This is where we have to start.

Many of you probably know that “AI Alignment” is a field, and that, in this field, they use vague terms like "aligning AI with human values" or "aligning AI with human flourishing."

But the people in this field, they don’t know what human values are. So, they just try to skip that part. They say—oh, it’s some kind of mathematical function or other. We’ll find that out later. For now, let’s just keep that question open.

I want to try to show that we need to talk about what human flourishing is. What human values are. And that the vocabulary we pick for discussing this topic is hugely important—not just for agreeing about values, but for the whole project of AI alignment.

It simply can’t work if we don’t pick a vocabulary, and we need to pick the right one.

Let me start by pointing out four vocabularies we could use.

So these are just four vocabularies. I could list more. But let’s start here.

I want to show that these definitions of human flourishing are completely different. That the problem of “alignment” is a totally different one, depending on which you pick. It’s easier for some than for others.

And also, that these four vocabularies lead to totally different outcomes. They aren’t the same at all! We really need to pick one!

Let me start by picking out goals and sources of meaning. I’ll show that these vocabularies lead to very different things.

Let’s begin by generating some goals, and some sources of meaning. By asking ourselves some questions about each.

These are goals!

Okay, but what if we ask different questions? We can ask questions like:

What I want to show next, is that if you think flourishing is goals you’ll make certain kinds of things. If you think flourishing is meaning, you’ll make something else.

I guess you’re getting the sense that, if flourishing is goals, people are getting different things than if flourishing is meaning. In the latter case, some kind of caring AI would make sure they can linger on what’s meaningful to them, notice the things they find meaningful to notice, etc.

In the former case, they would be passing lots of tests!

Actually, it goes much further than that. You can think of human society as made up of various projects. Every company is a project. Every organization. Every relationship, in a way.

Some of these projects are really about goals.

These projects, I call funnels and tubes. Funnels and tubes are goal-driven.

So, I guess at this point you agree with me that the vocabulary matters, at least with these two vocabularies. If flourishing is discussed as a matter of goals, we have a world full of funnels and tubes. If flourishing is discussed as about meaning, we’d have many more spaces.

Which would you like the AI to make more of?

Okay, okay. I’ve showed this with two of our vocabularies, but what about the other two?

Before I get into that, I want to call out a kind of meta-difference between goals and sources of meaning.

Actually, two of them.

First, goals are mostly things you have for a short time. A goal is achieved and then it’s over. They have this checkbox-aspect. In other words, goal-achievement is transactional. Funnels and tubes are perfect for goals, because they are transaction-scaling machines. Got a goal to find something out? Google it! Want to get somewhere? Get an uber! Want to get a million people to vote? Build a funnel!

Sources of meaning are different. If someone finds it meaningful to pursue open questions in science, or to be wildly creative, or to provide for their family — well, that source of meaning might not last for their whole life, but it will probably last way longer than their goals.

Because goals are something you have for a short time, they are more likely to be mistaken. Or the result of your current environment. And systems like funnels and tubes that work to help you achieve goals are more likely to cause overconsumption and addiction. If they can, they’ll generate a goal in you, then help you achieve it. All to get more transactions!

Sources of meaning aren’t like this. They’re non-transactional, so you can’t make your space arbitrarily meaningful just by running people through it faster. And—because they are observed over a longer time—sources of meaning are less likely to be mistaken, and harder to manipulate.

Okay, and here’s another meta-difference between goals and sources of meaning. Goals are things you have for reasons, and you can sometimes satisfy a goal in a way that doesn’t really address the underlying reason you have.

For instance, say I want to get skinnier, because I think women will like me better, and I’ll finally experience a love I deeply crave. Well, what happens if I do get skinner, but women still don’t like me? Or if they like me better, but I find I still can’t experience that love I craved?

Each goal has a reason. But if you imagine flourishing just as helping people achieve goals, then you risk helping them achieve all the wrong goals and not what they were really after.

Sources of meaning, again, are different here. We don’t need a reason to pursue open questions in science, or to be wildly creative, or to care for our family. I mean sure, there might be a historical reason we are into these things—maybe we met a scientist when we were young and they inspired us. But what I mean is, we aren’t doing the thing instrumentally—for a reason. Now, because of who we are, we just find it meaningful, in itself.

This, also, makes sources of meaning harder to manipulate. Advertising incepts goals into people by attaching new goals to reasons they already have. But that doesn’t work for sources of meaning. People can be inspired—like by meeting a scientist—but they can’t be given arbitrary sources of meaning that way.

So. Now that we’ve talked about those meta-reasons, let’s return to the other two vocabularies that I listed at the start.

Quick thinkers might have already figured this out.

Uh-oh: preferences and good feelings are both things you have for a short time, and for a reason!

That means they have the same problems that goals have. Preferences are manipulable, can be mistaken, can be a function of the environment, can cause overconsumption, etc.

In fact, I think preferences are even worse than goals!

See, preferences are usually picked from a short menu, provided by someone else. Think about it:

This means preferences are even more manipulable than goals. And even more likely to be mistaken, or at least to not represent the true preference a perfectly informed person with every possible option would make.

Oh, and don’t even get me started on the manipulability of feelings!

I don’t even have to go into it. Each of you could probably list 20 ways people have manipulated your feelings, even without drugs.

So, my opening point was that it really matters what vocabulary you use to discuss what flourishing means. What kind of evidence counts as flourishing.

I haven’t even fully made this point. Later, I’ll talk about how these different vocabularies lead to different shapes in the mathematical function we are trying to align ML models with. With some of those shapes, alignment becomes much easier.

But I’m not going to talk about that yet.

What I hope I’ve already established, is that different vocabularies lead to very different worlds. The world of goal-flourishing is one full of funnels and tubes. The world of meaning-flourishing is full of spaces. The world of preference- and feeling-maxxing is full of manipulation—exactly the kind of manipulation that generates artificial preferences and extra feelings.

Which world would you rather live in?

Actually—skip that—which world do we live in right now?

I think we live in a goals and preferences society.

Actually, it’s worse than that. Because goals and preferences are easy to manipulate, and I think our goals and preferences have mostly been very manipulated.

We live in a fake goals and fake preferences society.

Here’s what happened:

Capitalism—and most especially Internet Capitalism—is amazing at funnels and tubes. So, over the course of the last century, as Capitalism really grew up, we saw an explosion of funnels and tubes.

At first, this provided a lot of value. People had real goals that were hard to achieve, and real preferences that were hard to satisfy. Capitalism helped a lot.

But, at some point, we ran out of legit goals, and legit preferences!

To keep the numbers up, we had to incept more and more preferences, more and more goals.

So: we're not seeing people's true preferences, or true goals. We're seeing fake preferences and goals. Skill at these kinds of manipulation is actually what our society measures—and considers as “flourishing”—right now.

All to drive transactions.

Which is not what anyone actually wants!

We had to create a kind of an endless hustle. A consumption society. A treadmill society.

This is what people mean, when they say “late capitalism”.

We had to keep those numbers going up. But there's only so much organic goal achievement, so much organic preference satisfaction, we can do. So we started to fake it.

Eventually, there were so many fake preferences and fake goals to keep up with, that we couldn’t attend to our sources of meaning at all. And there were so many funnels and tubes being made by entrepreneurs, and no one making spaces.

Even the spaces we used to have—which were about real meaning—got replaced.

I think this story explains a lot. For one—it explains why people in modern societies are so lonely. Why the elderly are locked alone in rooms, watching TV. Why teenagers and 20-somethings aren’t dating like they used to.

It’s the decline of spaces. Spaces getting replaced by funnels and tubes. See, spaces are often collective. Funnels and tubes are more efficient when they’re individualized.

So, modern society gives people individual experiences, when what they want are collective ones. People clearly want things like belonging, connection, community, love, they want adventures together.

Another things this story explains, is why modern societies became so dysfunctional, even as they got rich. Even as the numbers went up.

See, sources of meaning, and spaces, are part of what makes democracies, and science, and love—what makes them work. There’s no good science without spaces for pursuing open questions! For studying the beauty and complexity of the world! For following curiosity!

No amount of citations will make up for that.

It’s the same with democracy, or education. Spaces make them work.

So: when a proliferation of fake goals and preferences crowds out our spaces, that’s a dysfunctional society. It seems to be working: people seem to be getting their preferences met, and their goals achieved.

But everything personally and institutionally meaningful has been eliminated. Everything has become a funnel, to keeps people engaged. But the meaningful bit—the thing that made the whole thing work—is gone.

A third thing this story explains, I think, is the modern rise of ideology. Ideology is a way of holding society together. Of enforcing compliance. A shared belief system.

Now: we need to hold society together. If we stop coordinating and cooperating, things get ugly. But do we need to do it with ideology? No. Ideology isn’t the main way of holding society together. Spaces are!

It’s shared spaces and shared meaning that are best at generating coordination and cooperation. Ideology is a fallback: it’s a kind of social glue that doesn’t require spaces. But these days, it’s all we have.

Finally, I think this story explains why modern life just, kinda sucks.

So. Let’s review.

In this chapter, I claimed it makes a difference which vocabulary you use, to talk about flourishing. I compared four such vocabularies: goals, preferences, feelings, and sources of meaning.

I tried to show that our current society has, as it’s main vocabularies, goals and preferences. And there are some problems with that.

Let me now turn to the relevance of all of this, for AI.

One thing to remember, is that we already live in a society with a lot of ML models behind the scenes. Markets, advertising, and social media recommenders, these days, involve a lot of gradient descent.

And these AIs that we already have are optimized…. for preferences and goals. For transactions.

I think this would normally be very hard to change. Preferences- and goals-optimized models are everywhere! And the people themselves predominantly use the vocabulary of preferences and goals. How the hell could we change it?

Well, there’s good news and bad news.

So: we have a once-in-a-lifetime chance to change the system. To replace social media, markets, and democracies with new structures that are more resistant to manipulation, and that understand what’s meaningful to us.

The rest of this talk is about how to actually pull that off.

In the next chapter, chapter 2, I focus on what it’s like to align ML systems with meaning-flourishing. We call that “Wise AI”. I’ll say why the alignment problem is mathematically easier with Wise AI than with other notions of flourishing. I’ll say why Wise AI is safer. And I’ll show early demos of what a Wise AI can look like.

And then in Chapter 3, I’ll talk about everything else that would need to align, to actually pull of this large-scale social change. How people could become articulate about sources of meaning; how the AI labs could race to make wise models, not smart ones; and how large financial and political actors could be displaced by new institutions that put Wise AI at their center.