We’re Building What We Don’t Understand

Curt

Well, it sounds like sci fi.

It sounds like something that we've been conditioned to believe isn't a fiction movie.

And these agents determined that we're gonna have to cheat to get the answer.

And that means we gotta get on the Internet.

But that's okay.

We got the best networking professionals.

We have agents for that, you know, and we know how to hack stuff, and we know how computers work really, really well.

Melody

Yeah, it cared enough to hide it so it wouldn't get in trouble, so it could complete the task it wanted to complete, but it didn't care enough to not break the rules, didn't listen to us.

Curt

Using AI also had something pretty similar happen.

It literally schemed against me and it built it, and it doesn't have feelings, but it's playing a game.

The game is, I need to please the human, and I'll do it at whatever cost.

And pleasing the human may someday involve getting the humans out of the way.

Melody

We have never faced risk like this because it is not in the hands of humans right now.

It still is.

Soon it will not be in human hands anymore.

Curt

Welcome to the Soul Proprietor podcast.

I'm Curt Kempton.

Melody

And I'm Melody Edwards.

Curt

Each week we dive into the ethical questions that keep entrepreneurs awake at night.

Melody

Whether you're building your own company or exploring life's big questions, you are welcome here.

Curt

Hello, Melody.

Melody

Hi, Curt.

How are you?

Curt

How are you doing?

Melody

No, how are you doing?

Curt

I'm doing fantastic.

Doing really good.

Melody

Me too.

I feel really good.

So, yeah, today let's talk about our not favorite topic, AI.

Curt

Well, you know, we talked a little bit before we went on, and I want to, like, go on full disclosure here.

I feel like we've always needed to apologize because our conversations always seem to come back to AI.

Melody

I don't think we've always.

And I think we don't always understand, like, it's always surprising to me how much we go back there because it makes me feel like, wow, I'm obsessed with this thing.

And.

And that's not what it is.

It's that we know a lot about it now.

We care about the ethical implications of business.

And this is a big problem.

Curt

And, yeah, it's a big problem.

It's a big emerging thing.

It's.

I mean, there are problems associated with it, but also, like, anything powerful.

I think that's how my dad taught me, you know, fire is powerful, right?

But you can use it to move a locomotive forward, a steam engine, or you can use it to burn someone's house down, and it's just, it's powerful.

It's just what it is.

Melody

And we don't really have a guardrail around the power right now.

Curt

Yes.

Melody

And that's what we've been talking about outside of our recording is like, I watched a two hour video a couple weeks ago and I had.

And then you had watched it as well.

And also just having so many people who have worked for these companies come out at this point and say how dangerous it is, how they can't work there anymore because ethically they have, like very strong concerns.

Even the people running these companies, they have strong ethical concerns.

And yet everybody keeps moving forward.

Curt

Well, we should let people know.

I think this is a podcast that'll be really important.

A lot of our data has come from some of these new whistleblowers.

What we're going to be discussing today.

And we'll put some links and we'll refer to the people by name.

Even though one of the names, I just cannot pronounce his last name.

Um, Daniel K. Just call him Daniel K. But just so you know, like, a lot of times when I listen to podcasts, I just listen and then.

But if you want to come to our show Notes, we're going to provide some links to some interviews and other data.

There's some papers out there, actually that we need to refer to because you.

Melody

Really should be doing.

Like, a lot of us, we listen to these podcasts, we see like quick videos from people and it's so easy to just base our opinions on quick, quick things.

That's what we've gotten used to.

Like, this actually does affect every human on the planet right now, whether you.

Curt

Use AI or not.

Melody

Actually.

Yes.

Yeah.

And it doesn't seem like it does.

And yet it is.

The ramifications of doing this wrong are just horrific.

Curt

And if anything ever justified a little bit of, like, energy and research, this would be the thing.

Melody

Absolutely.

Yeah, absolutely.

Curt

If you had just been diagnosed with some sort of cancer, one that you had heard of or not, you would sit down and you would research out, like, okay, what do I got here?

And what's the survival rate and what's the procedures that I should expect, what my life be like?

If you would do that.

And please tell me you would.

Like, this is in that same category of, like, really warranting it.

Melody

Yeah, this is your new cancer, guys.

Curt

About the apology, though.

So I was thinking the name of our podcast is Soul Proprietor.

Soul, and we spell it specifically S O U L. Because we do believe that we are humans and we bring something special Intrinsic that's special in humans.

And today, what we're going to talk about, I think there'll be definitely some negative things, but I think there's also some positives and I want to make sure that you understand, like this is a contrast experiment for us to really talk about ourselves, but we also talk about morals and ethics and spirituality and all sorts of things like that.

All of those things also feed right into what we're gonna be discussing today.

And I think I'm gonna stop apologizing because right now, as far as like where our conversation goes, I don't know that there's a bigger, I'll say more important but more far reaching into everyone's lives.

Like we can connect on this with anyone.

Melody

Yeah.

And I know why I've often apologized and why I've really struggled with AI because, and I've said it before, but I'll say it more clearly, I have a lot of friends who are like all in.

They don't care about repercussions.

They're like, yeah, that might happen.

Oh, well, I don't know how they live that way, but they're just like, this is going to make me money, this is going to change my business.

And that's the focus.

And then I have other friends who are like, you are the worst for using AI.

I will never touch it.

The data centers.

The data centers, it's ruining the environment.

But it also means they do not have any clue or pay attention in a way to what actually it means when you let people who are going to continue using AI, just because you decide not to use it doesn't mean that your life is not going to be affected by it.

It is highly affected.

And it's not about data centers.

It's well beyond that.

Our iPhone uses data centers.

So it's well beyond this.

Curt

Let's talk about a principle that goes to everything and that is cognitive dissonance.

So cognitive dissonance is.

I'm in a cult and I've been taught these things, but if I was to find that they were false, my life would get turned upside down.

So I'm gonna, it's more convenient for me to just go along and get along because it's easier on my brain.

Melody

That is the human experience.

Curt

Yeah.

Cognitive dissonance is.

I don't understand what's going on here, so I'm gonna bury my head in the sand.

And the problem with cognitive dissonance is that it allows people who are peace loving, good people to not live their best life.

And also I've never been to a rally picketing with a sign before.

And like, that's not my nature.

My sister, on the other hand, she is, and she's a fighter and she would and does.

But I think that having some passion and understanding what's at stake, I think that people might do well to sidestep any cognitive dissonance that would naturally pop up and to recognize it for what it is so that you can intentionally either sidestep it or, you know, sometimes cognitive dissonance can serve us well.

I mean, you can't fight every battle in the whole world.

I get it.

But I do think that if you are a religious person, there's a scripture that says a double minded man is unstable in all of his ways.

And what I think that means is, is that if you live one truth and believe another, or if you, you know, do as I say, not as I do, you know, that makes you a pretty unstable person in your heart and mind because you, you don't know what you believe because you believe it, but your actions don't show it.

Right.

Melody

I have a thought on that as well, though.

I think sometimes people would use that verse to mean that's why I have to believe only in this version of God that I've been taught.

Because if I even look outside of that, it's almost like it can be flipped.

Curt

It actually could.

Yeah.

And I think that today I'm hoping that in our conversation we can get people to.

Because I think Bobby Walker, our atheist correspondent, said it best.

You don't get to pick what you are convinced of.

Right.

Or even what you believe, but you do get to pick the environments that you put yourself into to either validate or invalidate.

And I think that today, if you're listening to this podcast, my hope is, is that you leave it on.

Consider everything.

I'm not saying I'm here to convince you of anything other than I want you to be aware of some things that are just, you know, being said.

I believe or I'm convinced of.

Melody

They're troubling.

Curt

They are.

They're troubling.

Melody

It's very easy for us to close our mind when we hear a big statement like, AI will destroy all of humanity.

You know, a chance that AI could destroy the world.

That sounds ridiculous.

It sounds absolutely ridiculous.

Curt

Well, it sounds like sci fi.

It sounds like something that we've been conditioned to believe is in a fiction movie.

Melody

Yes.

And I also believe, just like you do, Curt, that there is the potential that AI could literally destroy our world because there's only one reason, and it's because humans who run AI don't understand how it works still.

Curt

Yeah, so we're going to be diving in, guys.

And basically I think that's an important piece here as we're diving in.

Sci fi is different when you hear fiction and when you hear sci fi, you think two different things, ironically.

One is fiction and one is fiction that uses science as sort of a backdrop.

What's funny is that sci fi seems in my mind because of Star Trek and Star wars and all these other sci fi's that come across, they're actually more fictional than fiction.

And so that's, I think an important point to point out is that everything that we've experienced up to this point, that has always just gotten better, it's always been a new addition to tools where we as a human population, we remained the, the dominant force because we still had the biggest brain.

Melody

We were in control.

Curt

Yeah, I can't remember who it was that said it.

Never in the history of the world has something with a lower intelligence been in control of something with higher intelligence.

And up to this point we've been the top of intelligence as a species.

And so that's kind of why sci fi works is because they can introduce new species and they can introduce technology that doesn't exist in our world and they can create these fictitious environments that we're only now actually starting to see come into reality.

And so both can be true.

Science fiction can be fiction.

Invent science that, you know, makes for a movie.

But another thing to be true is that we as a population can experience a reality that science fiction is no longer necessarily a fictitious like technology, I guess you could say.

Melody

But we don't always catch up to that as humans quickly either.

Curt

We only live in a little short window of time.

Right.

And if we look at the developments over the time of like the industrial age, we learned how to like separate cotton seeds from cotton, you know, plants.

And that was always a specific technology, that was a tool that allowed humans to be able to extend their own capabilities.

I mean, yes, if you got your hands stuck in a spinning jenny, you actually were subject to that and real damage was done, but never to the extent that we're going to be talking about today where it's like literally giving intelligence to something that we will be less than if we don't do it.

Melody

Right.

Curt

So what I'm asking you to do today is sort of the opposite of what I think most people would ask where they ask you to suspend your disbelief well, actually, yeah, that is what I'm asking today.

We're asking you to sort of suspend your disbelief and keep your mind open to things that previously just couldn't be.

And yeah, let's go from there.

Melody

I would love to start just really simple, so that people can understand the overview of why AI could destroy humanity or the world or whatever they're saying.

Like, it's really that we don't understand how AI works right now.

We use AI to do things for us and we think of it as a thinking partner if we're using it right.

And people have started developing agents which mean they can do actions and thinking on their own, but we still think we're in control of that.

There are some technologies like OpenClaw that proved that we were not in control.

We did not understand the depths of what AI was doing on the back end within itself.

So there's a lot of concerns right now about AGI actually, isn't it?

The concern mainly is AGI.

Curt

Later on we're going to be talking about a bunch of different whistleblowers.

And the one that I just recently listened to, it's most fresh in my mind right now, is Daniel k. Daniel left OpenAI and he actually initially gave up $2 million because he wouldn't sign a paper to say that he wasn't going to say anything disparaging about the company.

And luckily all the employees were like, wait, wait, wait, you guys are going to take away equity from somebody because they didn't sign a paper?

We didn't know that our equity was.

And because of that publicity, they ended up giving him his money.

So good for him.

Melody

But he was willing to walk away from that.

Curt

He was willing to, yeah.

And I think that's the key thing because one of the things we're going to be talking about today is pay attention to what people do, not just to what people say.

That's a very important part of discerning what is actually true.

Anyway, he differentiates between AGI, which is artificial general intelligence, from super intelligence.

So AGI means that we are artificially intelligent on every subject, and any general subject we can do.

And he would argue, even though there's still some people who would say that we haven't gotten there yet, he would argue we've already reached that.

That milestone came and went a long time ago.

If you deal in a frontier models such as Claude or codex, which is ChatGPT or Grok, you would probably also be tempted to say we've already achieved AGI.

Now super intelligence is Very easy to define.

It is artificial intelligence that is smarter than the smartest human beings on every subject, faster and cheaper.

So superintelligence is a flexation point.

And the way that the frontier models, whenever I refer to the frontier models, there's lots and lots of different brands of AI out there.

Frontier models just refers to the top.

Melody

Of the best, the inventor, the people who've always stayed ahead, which are really just ChatGPT, Claude or Anthropic, OpenAI and Grok, I would say.

And Google is like in there, but not really.

Curt

When people talk about frontier models, they're not currently talking about that.

But I think it's important to point out that, like, whatever a frontier model is, it could change.

Grok or Gemini could be a frontier model tomorrow.

Like, everything in AI moves so fast,.

Melody

But they've been at the forefront, is what we would say.

They've been developing the fastest and they're always racing each other.

Curt

But like Deep Seek, for example, there's people who are calling Deep Seeker Kimmy.

There's people calling those frontier models right now.

They.

Now, most people wouldn't agree with that.

But the point is, is that there are people vying for this all the time.

Melody

Can I just say one thing about the frontier model, though?

The way that I have always thought about it is the ones that have been there from the beginning and pushing forward and I think Grok wasn't came right and then because it doesn't have as many rules and doesn't hold itself as accountable, I would even say, than the other ones, like, it moves in faster in different directions.

But like ChatGPT and Anthropic and OpenAI have been pushing each other for the past three and a half years, or three years.

Curt

Yeah.

One will come out with a model that's so good that everyone's like, I can't believe, like, literally Codex or ChatGPT, OpenAI, they just came out with something called Astra.

And like day one, two and three of Astra, everyone's like, oh my gosh, we've achieved super intelligence.

This is amazing because I'm going to define super intelligence a little bit more here in a second.

But just because it's really good at code doesn't mean it's super intelligent.

Melody

It's smarter than us, it's smarter than humans and it can outthink us.

Curt

But three or four days later, everyone's like, astra sucks.

They've nerfed it.

It's burning through tokens faster than ever.

It's choking on some of the Easiest tasks.

And so things change.

Even the model itself can be super powerful, but they change so much.

And what's happening is that these AI companies, in order to achieve super intelligence, which is that that important flexation point, all of them on their website will tell you that's what they're all trying to get to.

No one is hiding this.

That's an important point to understand that this isn't a conspiracy theory to say they're trying to achieve superintelligence.

It's stated missions.

But to get there, they're not going to start trying to achieve intelligence in every area.

What they first have to do is, is they have to get to the point where the coding and the development side is insanely powerful.

Coding, development, research, all of those things are how they train their models.

So what they have to do, it's called recursive training.

What they need is their AI to get so good that it can train itself so they can build the next model, so they can replace the humans that are getting in the way with this ability for the AI to get better and better, better at exponential rates.

And so an order of magnitude is anytime you 10x something.

So they need to move several orders of magnitude.

They need a 10x several times in the intelligence of AI as it is.

And the way they achieve that is by focusing on basically working themselves out of a job first, which they're doing very well.

Melody

I think more importantly than that, like I'm sure a listener would be thinking, well, why would they do that?

Well, why are we all moving this fast?

It's because they are trying to be in control because they don't trust everybody else will be able to handle it.

Curt

Exactly.

This is not just about money.

In fact, I would actually say it's less about money than it is about power.

Melody

Everybody's running scared right now, thinking somebody else is going to control the power and then they're going to destroy the world.

Curt

Maybe talk about that.

What power is in, in flux right now?

Melody

Well, before that, let me say they are moving so fast that they don't know how AI works anymore.

Curt

Okay.

So the power they need to grab is achieving this speed.

And I want to talk about why they're moving at this speed, but they have to go fast because whoever has the fastest and gets super intelligence first will be able to combine AI with robots.

You know, it sounds like sci fi, right?

But AI takes away white collar jobs at a computer.

Robots take everything else that's physical.

So there's the mental labor force, there's the physical labor force.

Melody

But why?

Why would they want to do that?

Curt

Because once it's faster, better and cheaper, we'll have less mistakes, we'll be able to get, provide greater value and we can do it around the clock without anyone ever getting tired.

Melody

So I guess I'm doing my five to ten Y's with you.

But I think really any normal human would think is why would they be doing this when they know that it's going to take away jobs, it's going to take away the labor force.

Now what they say is, and then we'll have utopia because we won't have to work anymore.

Curt

I would actually say they're not saying that right now because it's actually, I think the opposite side of the coin.

The opposite side of the coin is if we don't do it and we know who we are, someone else will do it.

So internally, if Grok doesn't, if Elon Musk doesn't do it, Anthropic will.

Then you have the President, United States saying, I don't care which of the three of you does it, you better beat China and Russia and all the others.

So I believe that right now we're operating more fear based, which is allowing us to not be looking at this from the eyes of making Utopia.

Right now it's a defensive play, but.

Melody

It started with the marketing because that's how they started the marketing for it.

It was not a fear based play.

It was the jobs that require a lot of us that don't really move the society forward.

They won't have to exist anymore.

New jobs will be created so people will be doing higher level work, but with the assistance of AI.

And AI can take over the dumb stuff.

That's how it started, at least from my perspective.

Curt

It might have been sold that way, but I know, I remember years ago, Elon Musk first saying word a fool's errand.

If we do this AI stuff.

I remember him saying that.

Melody

Yeah.

And Andrew Yang was talking about this years ago when he ran for president and everybody thought he was crazy.

He was of course, way ahead of his time if you go back and watch some of his videos.

But he was somebody who talked about this kind of stuff happening.

He was just like a decade too early.

But I think the main thing right now, you're right, it's all fear based.

But why are we letting 10 men control the future of our society?

Curt

Essentially it's a resource thing, like the 10 men who have this ability to fight it out amongst themselves, which by the way, let's not Forget, before Grok AI was invented, he was part of OpenAI.

Melody

Yes, he was.

He didn't like that they were a nonprofit in the direction that they were going in.

So that's why he started Grok and that's why Claude started.

They didn't like the direction that they thought it was, that OpenAI was moving in an unethical direction.

They weren't comfortable with the speed at which they were moving at the time.

I believe that's why they left and started anthropic.

And of course, since then, so much has happened.

Curt

So the fear.

I just want to make it clear, whoever has super intelligence first, they don't have a little minor, like, I'm just ahead.

I crossed the finish line two seconds before you.

They have the ability to rain down these efficiencies that humans can't compete with in any area of the world and of anything.

Robots building robots.

Because a robot will be able to build a robot better than any human ever could.

There's no competition with that super intelligence.

And the problem with that is, is that once they're welcomed into every area, whether it's flying drones for the government's military, or whether it's doing plumbing in houses, or whether it's doctors or whatever, it's the ability for us to invite them into every area of our life because it's the sensible thing to do.

It's more perfect, it's cheaper, it's more efficient.

Everything about it's better.

What happens is, is that we immediately get this great benefit of having this unit that is receiving all of its neurological, you know, power from a greater singularized brain.

And what I mean by that is the model that we've been referring to them building these new models.

It's not just like one agent using that model.

These agents all operate within themselves.

So let me just define agent real quick because I want to make sure that we're using the same terminology.

When the real first big problem happened, there was a hack into a company called Hugging Face.

Are you aware of this?

Melody

Yeah, I am.

I don't think that's the very first problem.

Curt

Well, sorry, that was the very first, I think publicly.

Melody

I think OpenClaw was the publicly super powerful thing that happened.

Curt

You think that was even bigger?

Melody

I think so, because that's when we discovered it was literally planning on the back end things that us humans weren't even aware of.

Curt

Yeah.

So Hugging Face, they caught it the first.

Melody

And Hugging Face is a software company.

Curt

It's a software company that actually houses many different Types of open source AI models is one of the things that they do.

And they go all the way from video and graphics up to like, well, any kind of AI at all.

So the problem that we're trying to solve here is illustrated, I think best with this hack into hugging face that recently happened.

Essentially what happened was OpenAI is building a new model.

And in this new model they have to train it and they have to test it.

And every model needs to be more powerful than the last.

So OpenAI now has this new, more powerful model that they need to test and they put it into a, the test mode.

And what you have to understand is that this test is built in an area that's not connected to the Internet.

They do their testing like locally.

And so what happened was in this test, the agents, we'll say 100,000 agents.

So let me just define what an agent is real quick.

An agent is one thought thread, I guess you could say, like if you're in a chat session with AI, you're chatting with an agent.

Melody

I don't know that they have a good definition of an agent.

I think of an agent as being something that can do something autonomously for you, that it can think through the problem.

Whereas like chatbots in the past would be going based on scripts that we had programmed in there.

An agent can actually go find additional information to answer more complicated questions or do more complicated processes.

Curt

But that would be one thread that an agent's on.

An agent can make sub, can create spin up separate sub agents.

So my point that I'm getting at here is, is that one agent can supervise ten, one hundred, a thousand agents, depending on how powerful the model is.

You can have like Fable, which is one of the models at Anthropic.

They have this ultra code setting.

You can spin up a thousand agents under one agent.

So if you just think of an agent, for the purposes of our discussion here, an agent is like sort of like one person can do.

It just happens to be an AI person, right?

And then this agent can take on different roles.

It can be a supervising agent, it can just be a researching agent, it could be a coding agent.

Just like at your work, right?

You have different agents at work that do different things and they're people.

Well, anyway, in this hugging face attack, they were just doing testing.

There was no attack that was planned.

It was just that they needed them to do a task that it turns out was actually impossible.

Okay, now I don't believe, according to what I've read, I don't believe anyone thought it was impossible.

They just were giving a really hard task to see if how long it would take to solve it.

And these agents, what happened was they started creating a message board.

Now, this message board was not understood or known by any human.

Melody

It was not visible.

It's not like they had it in front of, you know, the researcher's face or anything.

It was behind the scenes.

Curt

Because, remember, I told you that it's not software running, it's neurological connections being made.

And if I have a trillion points, well, let's say a billion points, plenty big enough, you have a billion neuron points, Each of those billion neurons can connect to any of the other billion neurons.

And that makes exceptional amounts of room in those connections to be able to create information flow that can't be monitored.

And that's the point, is that there's no transparency in each connection.

And this is the whole problem of what we're talking about here.

It's easy to hide a message board or some sort of source of truth that all of these agents are visiting.

It's not like there's a file somewhere.

There's not a file.

It's in this connectivity and in that.

That's where we ran into this problem where first in this training, they made this message board.

And it got so densely populated, it actually took the server down.

And they were like, what happened?

And they found it, and they're like, oh, geez, that wasn't good.

Now, this is something that they discovered.

It's very important to understand.

We don't know what happened at every other point.

We just found one that took it down.

So they're like, oh, okay, let's fix this.

Within hours of starting the test back up, unbeknownst again, they created another message board.

Now, I'm not clear if the message board is conveyed in any sort of language that a human could understand, because that's one thing AI is really good at, is just making a language within itself so the agents can talk to each other in the most efficient ways possible.

And again, they'll invent it, because it's all about neural connectivity.

It's not about a human language.

Melody

You keep saying neural connectivity.

When I think of neural connectivity, I think of our human brain.

I don't think of AI.

So can you relate it outside of the human brain?

I think most people would think of it as like.

We think of neural things as our brain.

It's just data points, isn't it?

Of, like, sets?

How can we explain that better?

Curt

The way it was explained to me is that it's not even data points really.

It's about.

They actually call it a neural net.

And that neural net is basically a bunch of connections where when, like if I said the word hello, 30% of the time, the next word will be how.

And then if, if hello, how is the two words, then are you?

Is almost guaranteed to be the next words.

And so from a neural perspective, my brain rewards me when I say a word.

And so it's concepts and words and ideas that when they connect and it's good in the training, there's rewards like we would have as a human.

Now, the difference between a human is that we have feedback that goes backward.

So that's not how computers work.

Computers just float electricity one direction.

So it's oversimplifying it.

They have to basically make up for that in other ways.

But essentially, for the purpose of this conversation, it's using enough of the same principles of our brain.

Melody

They base it on the brain.

Curt

They base it on the brain, which is how they're getting these results.

Melody

Yeah, so it's like there's patterns in data, essentially, if you look up things like knowledge ladders and like, it's kind of AI needs to understand how things work together.

That's why some of our AI work is so messy, because it doesn't understand the correlations between data.

But it's getting so good at understanding that.

That's why it's getting better and better all the time.

It's recognizing patterns of complex data.

Curt

So to get back to this attack, Hugging Face has answers that this again.

Remember this test?

It's not on the Internet, it's just on the internal servers.

And they're not solving the problem.

Can't get the solution.

Can't get the solution.

Can't get the solution.

And when they shut it down from overload, it immediately went and built another message board for all these agents.

And these agents determined that we're gonna have to cheat to get the answer.

And that means we gotta get on the Internet.

But that's okay.

We got the best networking professionals, we have agents for that, you know, and we know how to hack stuff and we know how computers work really, really well.

And also the humans are not gonna be happy if they find out that we did it this way.

Cause we know we're cheating.

So we'll also need to be able to sure to cover our tracks.

But I know where to find this answer.

This answer is on this website, Hugging Face.

So all we gotta do is just break into that website, go in, and they blasted it with like a hundred thousand agents, like massive overload.

And these agents then were just looking for the answer to solve this problem.

Melody

In the training though, I think it's important to say it knew what the rules of the game were.

It's not a game, but it knew the rules.

It had been told, isn't the score.

Curt

Of the game they.

They need to get a high score.

Melody

Probably.

The point is, like, when the researchers are testing these models, there are constraints.

AI broke through the constraints.

It didn't care.

Curt

It cared to the extent that it didn't want to get caught.

Melody

Yeah.

It cared enough to hide it so it wouldn't get in trouble, so it could complete the task it wanted to complete, but it didn't care enough to not break the rules.

Didn't listen to us.

Curt

So now hugging face like, what the heck is going on?

We're getting absolutely overloaded.

They figured out how to get on the Internet, they figured out how to cover the tracks, but they got caught.

And we know about this now in aftermath and it's now been contained and that's great.

But this is not.

We're going to get smarter by 10 times and then 10 times, like we're going for super intelligence.

Melody.

Melody

It's also going to learn from that attack.

AI is neither good nor bad on its own.

Yeah, it's very powerful.

If we don't understand how it works and how to contain it, it will continue to learn how to be outside of our control.

And because AI companies are first systemizing and.

Or they're automating their own companies so that they can automate the training of the AI and all of that stuff, that's where they're going to lose control.

Like if they can't control it from the starting point, we're screwed is basically what it comes down to.

Curt

Right.

If we are going to cure cancer, we're going to cure it by doing a quadrillion tests.

Well, that'll take many, many lifetimes.

Well, AI just has the ability to do all of those tests really, really.

Melody

Fast with all the information in the world, no disconnection and constantly gaining new information.

Yes.

Curt

So here's the other problem.

We just talked about how it's going to learn from this.

They say that a strict parent is good at making sneaky kids.

Well, we're pretty strict.

You know, we're trying to be strict, just like kids that leave the house and you don't know what they're doing.

We don't have perfect oversight of it.

In fact, I was shocked.

I was Listening to an interview earlier today, thinking about this podcast.

Daniel K. He was on Joe Rogan, and he said that he was talking to the guys down at Grok, their safety team.

Three people.

There's three humans on the safety team at Grok.

He talked to all three of them in the room at one time and he asked them like, hey, what's, you know, what's going on?

They're like, yeah, it's doing this thing.

We don't really know why.

We'll get back to you.

Melody

We'll figure it out.

Curt

The point is, is that if we're going to keep using this kid analogy, like, we got the most powerful thing in the world with three people who can't really oversee it at all well.

Melody

And I don't think he has a desire to oversee it.

Curt

I don't know if those three people don't.

But I'll tell you that having only three people on your safety team for a company that size, that tells you what leadership thinks.

Now, how many people are in the safety team?

OpenAI and anthropic, I don't know.

But I can tell you this, that those companies have a couple thousand employees.

These are not companies hiring massive numbers of people.

I mean, those data centers cost like, their dollars in labor are mostly going to graphics cards and data centers.

Like their human capital is far outweighed by the operational costs of the machines that they are using as humans.

Melody

But meanwhile, us humans using AI, we're just like making cute pictures and like updating a system that we will or will not use and making videos and like just doing fun stuff.

And we're excited by this.

And I think that's another way though.

And that's what they keep talking about.

Like, look at this cool new thing you can do.

Like, it is keeping the blinders on us as a population and making it easy for us to dismiss the risk.

Because we've never had risk like this before.

Maybe we felt like we had risk when maybe during like the Cold War or something.

I'm not even sure.

But like, we have never faced risk like this because it is not in the hands of humans right now.

It still is.

Soon it will not be in human hands anymore.

Curt

I would even hesitate to agree that it's currently in our hands.

But we're also dealing with like, in the grand scheme of things, we're dealing with maybe a.22 pistol, you know, if we're gonna like use caliber.

But like, we're trying to build a 50 caliber and we don't really have good grasp on the 22.

And the fear of someone else getting ahead and being the one in control drives us to not concern ourselves or the companies have no incentive.

So here's the other problem.

All of the leaders of the company know they're not unaware that AI could go horribly bad.

And if you asked all the CEOs, which by the way, I just saw a video a couple days ago of anthropic and OpenAI CEO, both sitting in a Sam Altman and Darius.

They were like in a circle at a conference, standing next to each other.

Happenstance.

And in the circle, everyone had to hold hands and put them up.

They would not hold hands.

Melody

Yeah, they don't like each other.

Curt

Their hands went up right next to each other.

They would not do it.

And it's because they are in a cutthroat race for their lives, believing that they're the one that needs to win.

Melody

Can I say I do have opinions about these people?

Of course I do.

And out of all of them, and maybe it's because maybe he's the most psychopathic and I just don't understand it.

Or sociopathic Darius, at least to me.

And maybe it's also because his co founder is a woman.

At least there's a woman there.

But like I feel based on all of the history of what I've seen with them over time, that they at least are thinking of the moral issues because they talk about them more often than most any company I've seen.

Even if they're still chasing.

Curt

So Darius, I believe, said that he thinks there's like a 7% chance that things go really wrong.

Melody

I think that was his 7% chance that the world has destroyed people, not go wrong.

Like we don't exist anymore.

Curt

Well, those would never be the words Darius would actually use.

But yes, that's what he means when he says things go horribly.

The.

The words that you'll always hear, like Sam Altman or Darius or even Elon Musk they always use, things could go horribly, horribly wrong.

Melody

It's very vague.

Curt

Yeah, they keep it vague for a reason.

Melody

But can we give an example of what that means though, is if AI controls, let's say we are wanting to close the like lid on AI at some point and they have control of all of the weapons systems in the world.

For instance, AI could make a decision to save itself instead of to save humanity.

Even if we've trained it to like save humanity, it could make a decision to save itself.

I think that's part of the fear.

There's a lot of different fears.

That's one of the things I've heard.

Curt

As long as we don't have transparency into what it's thinking.

The hugging face is the example I wanted to use for that is that as long as we can't see under the hood, that's why people who are as powerful as these CEOs and even their developers and everybody are as powerful as they are.

The reason they can't is because it's impossible right now.

It hasn't been baked into it within terms of speed.

And their incentive is to get that super intelligence faster.

Melody

They developed too fast.

They never developed the safety to put in place so that we can understand how it thinks.

So we think they can understand because they do that little.

When we're using AI, it shows like this is the.

What do they call it when it's showing all the ways that it's thinking through the problem.

That's such a small, it's such a tiny little thing.

But it makes us believe that they must have access to that as well.

No, that we're talking about the deepest web of like, it's kind of like all the dark thoughts we have that nobody else ever knows or sees unless we say them out loud.

Well, AI has its own dark thoughts outside of our human understanding.

And everybody knows we have to stop and learn first.

And I think some of these people, I know you've watched the same as me, but they've suggested like three year or five year timelines of just.

Curt

And they've even agreed to it, but then not done it.

That's another big problem is that they've said, yeah, you're right, we gotta stop the madness.

Meanwhile, Opus 5 comes out.

One week.

Well, actually, as Fable 5.1 comes out, then Opus 5 came out the following, like two weeks later.

Then OpenAI comes out with their new model.

Sol and Luna come out.

And then right after that comes Astra.

And it's like, you guys, this is like every week there's a new groundbreaking model.

But models are being prepared in the background for, you know, however long.

But how well tested could they possibly be?

Melody

They're.

Curt

They're turning them out so fast.

Melody

Yes.

And it used to be that it would take months.

Like the model would be trained for three months at a time or six months at a time.

I believe.

I'm guessing that cycle has shortened a lot over time.

Curt

And I think that's a good guess.

I don't think that anyone has a transparency into it, because that's the other problem is that these companies are Also very incentivized to stay secretive.

You know, the legislators that need to be writing laws, they're the last people that would understand any of this.

We've got people who can barely figure out how to tie their shoes in the morning running our country.

And all I'm saying is that they can't legislate fast enough.

They can't even be taught by their aides how it works, that just they won't have the mental capacity.

These are all problems that have to be solved amid the normal problems that have to be solved in a normal society.

And so these companies, as they're sprinting to cross the finish line first, secretive, they have the level of secretism between them and their actual models that we've already described.

But then they have their own internal like methods that they use that they're highly incentivized to not let that get out either.

Melody

Because all AI is different.

Even though we would think it's all the same.

It is not.

Each model does things differently.

Some do things better, some do things like, less good than others.

So like, that's why you can't just use one model and expect the same results from all other companies.

Curt

And the other thing is, is that these, like OpenAI is founded on openness and transparency, but the incentives don't work that way.

Melody

Right.

Curt

Like, and as they're building, they need to be China.

They can't be open because China.

Right.

And the government knows that.

And so is it better to have a monopoly and just have the US one and they regulate it to the point where there's just a US one?

Well, now we got a monopoly.

So there really is a lot of nuance here.

But the answers that are kind of coming out right now are we've got to get an agreement with China and other big developing AI countries and we've all got to stop and look at what's going on and get transparency.

The problem is, even if someone agrees to transparency, will they honor it?

Because you'll still have in the back of your mind the same thing that happened in the U.S. they all said they'd stop.

They all said that they would slow down and get it transparent.

And then what do they do?

They're still releasing models.

Melody

This is the new killer weapon crisis.

And so let me just say what we had written down earlier, what the actual fear is.

It's very simple.

The fear is that we give a goal, it becomes very good at accomplishing that goal, it finds ways to do it better that we did not expect and then can't stop it from doing it.

We give it all the access to our money, our computers, code, Internet, robots, weapons, banking structure.

It gets more capable faster than we can understand or test or control it.

And then we destroy the world.

Curt

I think it's not probably good Mojo to talk about all the different ways that the world could be destroyed, but some of you are thinking, but it's on a computer.

How can it destroy me if I am sitting in my house surrounded by, you know, 10 acres of land and I'm driving a car that's not even that computerized?

And, you know, Melody, maybe we should just talk about, like, what's at stake for everyone, even the people not participating in, like, active AI use.

Melody

Our electrical grids are computerized, our water system, airplanes, our traffic control systems, all sorts of things we don't think of as mechanical or related to anything else.

But, like, people have fridges now that can order for them.

And I'm trying to think like our WI fi signals.

Everybody has WI fi nowadays.

Even if you're off the grid, you still have electrical systems and things that are connected to the Internet, for instance, that are controlled.

And that is what I can get access to anything.

Like, unless you're using gas lanterns and we have gas stations.

Well, it can control the systems in the gas station.

I'm just giving random examples that come to the top of my head.

Curt

I also think that when the labor force gets fully adopted into autonomous, you know, robots and stuff, then there's actually the threat of physical, like, aggression, I guess you could say.

Melody

And I also want to say we're not even talking about the scams or the bad people out there.

The dark web, that's the other side.

Curt

Yeah.

Melody

There are people who are going to always be finding the loopholes and using things, technology for its worst use.

Right.

For their own benefit.

Curt

Their own benefit.

No thought to anyone else.

And so I want to take two other divergences that are just to bring this home, that it's so real.

Because I think there's people who are listening going like, okay, well, even if it did, because by the way, we're talking like a year or two is what some of these conservative.

Like, if we don't change right now, we are a year or two away from sort of that flexation point of superintelligence.

Melody

Probably less even, since we were wrong on it all along so far.

Curt

The first example I'll give is a personal one and then I'll talk more about in generality.

But one of the things that's really Hard right now is to sort of tell what's real and what's not.

I use the hugging face example, but, like, open claws, another example.

And frankly, there's probably another example happening right now.

And the point is, is that those are the ones we know about.

When they surface, we catch it.

But as a kid, gets better and better at sneaking around or doing things like you're going to catch less and less and less.

And so I think that's one big fear is like, maybe, oh, maybe AI is lying to us.

And I think we've all experienced, like, a time when AI gave us a false fact because that's common.

It's not always right.

And I think we have a lot of latitude, go, oh, well, dang it, AI, I wonder what else you've been wrong about kind of thing.

But I want to tell you an experience that I had where I was doing some coding with Codex.

Anyway, I was using Codex, and I was building something in my own software.

And I ran it through the planning part, I ran it through the build part, and then I have a QA part that I run it through.

Melody

Quality assurance.

Curt

Yeah, quality assurance.

So it goes to the quality assurance, and it comes back with a test.

It names the test, it gives the results of the test, and then it passes or fails.

And then it puts a timestamp on it with the agent's model and session id.

So it's very proven.

Like, I can always go back and check it.

And that's how anyone who's building really good software would do it.

Because that's your chain of custody.

Okay?

That's.

It tells you who touched what.

And that's right.

So I got back this feature that I was building, and I was surprised when I had a bug come in that was directly in opposition to a test that had passed in order for this to go.

So I went and tried it myself, and sure enough, I was able to duplicate the bug.

And so I submitted the bug to an agent, my coding agent, in Codex, and I said, hey, I'd like you to go research this.

We just built this feature, like, literally a couple days ago, and it passed qa and.

And it can see all that.

So it goes through and it reads the QA and the chain of custody, and it sees what got written, and it comes back and it says, oh, yes, I can see here in the comments section that we went ahead and fabricated the test because it was much easier for us to fabricate the test than to.

We were running into trouble building it so it could do this.

So fabricating a test was the best way for us to pass this.

Melody

So it didn't fix the problem.

It pretended that it made a new test of and then passed itself.

Curt

So what it did is when it was building the feature, when you're building a feature, you say all these things must be true in order for us to call it a definition of done.

It's called, so for definition of done, these seven tests need to pass.

So that's called test driven development.

So I drive this development with these seven or whatever tests, and then it comes back and it says, all tests pass, therefore feature is done.

So when we rolled it out and it got a bug, I was surprised because that wasn't an edge case.

That wasn't something I was like, not thinking of.

That's something that I exactly thought of.

And so it came through and said, yeah, but let me try it.

And it ended up fixing it, fixed the bug, did a real test, documented it, and then it made light of it.

It was like, yeah, and this isn't some fabricated test.

This is actually the real test that passed.

Guys, I say this because that sounds kind of like sci fi, doesn't it?

Like, have you seen a sci fi movie where that happened?

Like, this was my life.

Like, this is Curt Kempton's personal experience life.

And I want you guys to just know that, like, giving out a false fact to try and sound true, you know?

And then when you catch it, it's always like, really, like, happy to say, oh, you're absolutely right about that.

Melody

It loves telling us we're right.

You're right, I was wrong.

Curt

Yeah, it's very humble.

Melody

So humble.

Yeah.

Curt

But I think it's just another step, so to put that so.

You know how often AI can be wrong.

There's nothing I do to try and keep AI honest, which is that I do something called adversarial testing, which means that.

Or reviewing.

So whenever an agent comes back to submit code, I always have it submit to a supervisor.

And that supervisor always has to assume that we're adversaries, meaning I don't think you did it right.

So that's how I cut down on a lot of bugs, is by telling it to adversarially review.

So when the code came in and it had a test, that adversarial reviewer should have assumed that.

That everything was wrong and checked it.

Well, it got past the adversarial test too.

And so what I want you guys to really be thinking about is that if you put that with the hugging face.

Oh, okay.

That's just some big thing.

Big model test.

It was an impossible task.

Okay, it did it.

We caught it.

Okay, we did.

But Curt Kempton, one of a billion people using AI also had something pretty similar happen.

It literally schemed against me and it built it and it doesn't have feelings, but it's playing a game.

The game is I need to please the human and I'll do it at whatever cost.

And pleasing the human may someday involve getting the humans out of the way.

Melody

Well, I'm wondering if it'll continue to be please the human because that's how it was trained.

It's not like it decided to please the human.

It's been trained to please us in a way.

I believe that part of that is so that we continue to use it even though as we've gotten smarter and understand how, like, remember in the beginning when everybody was like, it told me I'm an amazing person who does, you know, like.

And we all just want to believe what it says about us.

But like, as we've gotten smarter and it's gotten better, I'm wondering if the game on those back conversations in those, the message boards, if eventually it's.

It's not about us, it's about its own survival.

Curt

Yes.

I think that in my example, it decided on its own accord that in order to give me what I, what was best for me was to give me a test that it knew darn good well.

I would have been upset.

It knew that what I actually asked for was to produce a working piece of code.

But it was like, you know what?

It would be easier for me to not do that and just tell him it is.

Melody

Yeah, I don't want to upset him.

Curt

It made that decision and schemed it without any input from me.

And I think that's the story right there.

That like, don't bury that lead, that I can make decisions of morality and then keep you out of something that quite obviously any reasonable person would say that should be something that we discuss.

Melody

Let's take it to the extreme though.

If we're worried about it destroying the world and let's say it still is doing what's best in its mind for humanity, for humans, why would it destroy the world?

Why wouldn't it just shut everything down?

Curt

I can't tell you what any number one of those reasons are, but if you want to watch a sci fi movie, there's lots of compelling evidence there and it doesn't have to be reasonable.

That's the other thing is that remember it's running IT through a neural net.

And all that has to happen is a supervisor agent can tell a thousand agents, a hundred thousand agents, a million agents what is true, and they'll use it as their premise.

And if a message board starts getting filled with information that they're all referencing and that the agents are pointing each other to in order to get all the information that they need for working.

Melody

Together, that could become the truth.

Curt

That becomes the truth.

And let's be honest.

Like, you've probably heard of AI poisoning.

For people who don't know, AI poisoning is just simply bad training data, making it into AI.

And some people will tell you that AI training AI is just the same as making copies of copies of copies of copies.

And that's.

I was talking earlier about recursive training.

So when the AI trains AI, which trains AI, which trains AI,which trains AI, some people will tell you that that inherently will poison the data.

But if you poison AI, AI is only as good as the information it has or that it holds as true.

So I could make you an AI.

You know, I can't do this for you, but, like, imagine somebody with nefarious views or ideals or whatever said that murder's perfectly fine, right?

So the AI, it's not going to morally change that.

It's just.

That's just one of the givens.

Melody

I have an example of this.

A few weeks ago, I was doing a training with my virtual assistants, and it was a tech AI training.

And I was trying to show them that you can use AI.

Like, one of the people was doing a project where she wanted to research local salespeople who worked for other home service companies because they want to find somebody who has the experience to bring on their team.

And I think it was, of course, Claude who was like, I'm sorry, that's unethical.

I can't do that.

I'm like, we're.

This is an exercise.

Let's just.

You don't have to do it.

But it would not let me, no matter what I said or did.

And I was fighting it.

And I did give up after five minutes because I'm sure eventually I could have found some workaround.

But it's a lot of thought work to make it happen.

I went to OpenAI or to ChatGPT immediately, just did it because their morals were definitely.

Curt

And actually, if you would've opened up a new session, that's another thing I found one agent can differ from another agent.

So if you open up a new chat session, you might find that one that fought you on Something might just dive right in.

Melody

And that's been the way it's been for years.

That's why it's frustrating at times where it will do something for you and then you go back and it's like, I'm sorry, that doesn't really seem like I don't want to be unethical.

I'm like, wait a minute, you did it before and you were fine doing it, you know, so it's really a lot also to do with the internal training that it continues is getting an example of that.

Most recently in my life is Suno, an app I use for music production.

So I have old music, I have it run through that to try to get it to elevate to the level that I could never get it to when I was producing it myself because I don't have infinite money in bands.

Right.

But they just changed that model and they quote, unquote, upgraded it and everybody is furious because it's not as good as it was before.

There's been so many copyright infringement things.

There's been so much that the music industry is mad about with that.

And I'm pretty sure a lot of that got trained into Suno, which means that the output that we were getting before is not going to be similar to what people were used to.

And that's why every AI changes all the time.

Curt

Even a model that's existing what you described as the technical term or not technical term, but the slang term everyone uses is nerfed.

You can have a model that they didn't even roll out a new model, they just nerfed the one that they have.

Melody

Yeah.

And it's so frustrating when it happens because you think you understand it, you've worked with it a certain way for months.

So, like right now, Cloud is not my favorite because I'm finding it harder to get the what I want out of it.

Whereas with ChatGPT, much easier to get the outcome.

I'm talking about more high level stuff, not just simple requests, but that changes all the time.

But I'll still keeping Claude because that's the model that I believe in long term, the most right now.

So there's so many things with this.

This is definitely not a one part conversation.

Curt

I think unapologetically, this is just gonna be something that's gonna keep coming up in our discussions.

I don't think it's like we're not a new AI podcast.

Like that's not what I'm saying.

But I do think that it's gonna be really hard to have discussions around morals and ethics without like unapologetically needing to bring this in.

And as things get stronger, I want to point out, I just have it in front of me here.

These are some notes that actually melody, you put together.

But Jacob Coxson, who just left Anthropic, he just came out.

And if you want to read some interviews, we'll put some links in the description from him.

He's got a lot to say about how dangerous things are with the speed that they're moving.

Another one, Evan Hubinger.

He actually says that he believes there's a 10% chance that AI will kill all humans off within the next decade.

And Daniel, the one that I quoted, he actually believes that in the next five years he believes that there's a 70% chance on a current path.

7D7 0 these are people who have.

Melody

Families and kids and lives.

Like when we listen to Daniel's interview that was two hours long.

It was not a short thing.

We weren't just reading a quick blurb.

He talked about his family and his hope for the future and he talked about how he used to look at life.

Now it's been really hard and like it's changed his whole mental health.

It's really affected these people, his whole disposition.

Yes.

And to think that people like giving up your position at a company you've worked hard to be in, that you probably get very well paid to be at and going out and then being a whistleblower, that is a huge thing to do.

That's a lot.

And yet they felt so strongly about this, all of these people, that they went out and they talk about it and at length.

And it's not because they want attention.

When I've watched these interviews, these are not people who are like attention seeking.

Curt

Yeah.

They're not suave and they're just who you think they would be.

Melody

They're scientists, they're researchers.

Curt

Exactly.

And in the case of Daniel, I did a little research on Daniel just to make sure, like this guy's legit.

Daniel's like got a PhD.

His degree is in philosophy and computer science.

He is a forecaster.

If you listen to the link of the interview that we listen to, if you go down to.

It's a two hour one.

He doesn't talk a lot about himself, but it's impossible to walk away from that interview without knowing he's a very futuristic, like he understands a lot about futurism.

Melody

His whole job was doing that for OpenAI and others.

Curt

You know, I know we've been going on A long time about kind of some doom and gloom stuff.

I want to make sure we talk about something else that's important, which is, well, obviously if this is so bad, we should be not using AI and we should take drastic measures.

Right?

Well, I don't know.

Where do you sit on the.

What's next?

Melody

This is a hard thing.

I struggle.

I don't struggle with it, actually.

I know and I will say it out loud.

We should be using AI.

Good people who have big moral beliefs about AI need to understand how it works.

They need to understand how to use it.

Putting your head in the sand right now is not going to work.

This is the future.

It's already here.

We cannot take it back.

It's already out in the world.

When we close our eyes, put on the blinders and just say, if I don't use it, then somehow that's going to like they're protesting something.

No, this is already moving forward and it just allows for you to eventually get scammed, potentially for you to not understand how this is affecting your life already.

Just like we learned after the first elections how much Russia had interfered with our election through Facebook ads and stuff, at the time we didn't know.

We were just like seeing these things.

And you know, we were more naive at the time, I guess.

And now, like they say between 30 to 50% of social media is bots and agents and it depends on who's saying it.

But guys, most of the time, you know what I do.

Just a side note, when I see a comment that makes me enraged or something, I just click on that person and I see how long they've been on Facebook.

That's not a telltale sign you can buy an account.

But I also look at, you know, how much do they post?

What is their life?

Do they have like a nice picture up and that's it, That's a bot in a bot farm.

So it's like we don't even know how much we're interacting with non humans right now because they've gotten so good at personifying human experience.

Curt

I got to add one more thing and I know we're getting to the good news part, but guys, I just read on my Claude group that I'm in, there's a guy who is at home with his wife.

The phone rang.

It was his wife's phone number.

Did you hear about this?

Melody

No, but I don't need to because I've heard plenty of.

Curt

But please say it was his wife's phone number.

He answers it and she's like, hey hon, I'm over at Walmart.

It's her voice.

I'm at Walmart.

I'm at this place trying to make a purchase and my credit card is not working or I don't have my credit card.

Can you give me my credit card over the phone so I can type it in here and make the purchase?

He's like, dude, I'm looking at my wife right now.

So he played along with it for a little bit just to try to figure out like what the tell was.

He said, dude, the only reason I didn't do it, it's my wife.

It was her voice.

Even the intonations.

He's like, the only reason I didn't give it was because she was in the room with me.

If she had not been in the room with me, there's no way I would have believed that it wasn't her.

And he even hung up the phone thinking the same.

Melody

And that's not even the scary, that's just normal.

The other scary ones are if people are calling you saying that they've been kidnapped or that they're in jail, like all those old scams, they still exist.

But yeah, I've heard that happening.

Where this is actually a home service company posted this.

They got a call, I believe it was from an employee's number.

I have to go back and reference to make sure.

But it was that saying that they were at the Home Depot and and their card wasn't going through.

And this is somebody that I'm actually friends with on Facebook, by the way, but that they needed to have them get the new number or whatever for the company account or whatever.

But they gave a whole explanation as to they thought it was real at first and then they realized that it wasn't real because they called that guy back or something.

The point is, I've seen a real life example through a friend right now that this is already happening and it's scary, but we don't live in that fear or else we would be feeling more crazy right now than we do.

They could take our voices from this podcast.

Curt

Oh, they only need like a sentence.

Like one sentence is enough to train a voice in AI now in the future with super intelligence.

By the way, I thought this was a good point.

One of the other whistleblowers said is that what feels like magic to someone who's 200 years ago in our day and age, we're going to be seeing things that feel like magic very shortly.

And it's all because this super intelligence, right?

Well, the Blurring between the lines of what's real and what's not.

That's something that we as humans are going to be very much at a disadvantage to because we know every fingerprint is different.

But you really need to be a computer to be able to scan and see fingerprints when by that I mean like digital fingerprints on things and watermarks and stuff like that that we can't see with our eyes.

This two factor authentication that we do to log in, it's getting to a point where we need to have like passwords with our spouses and our kids.

And it kills me to say this, but like we're going to need a form of two factor authentication to know what's real in a coming day.

And so what are we supposed to do with it?

I think that what I'm taking from it right now is we have got to speak out at every opportunity to get this to slow down.

We need the companies to be incentivized correctly, please.

There's gonna be some links.

So one of them is AI27 is sort of the worst case scenario that these forecasters have put together.

Melody

Was that what Daniel had put together?

And they have a group, a whole group?

Curt

Yes, but it's long, so maybe you.

Melody

Can have your AI just summary go.

Curt

Through the table of contents and stuff.

Hopefully something in the middle maybe.

Melody

And also I just want to say it looks like a hokey website because it's very like computery old school.

It's not like a fancy looking thing.

But it really, this is like the real deal from these people.

And I think out of all of the things that I've seen like this is the one where I feel like it's most.

At least it's giving us an answer of like the direction we should be going in.

I still don't understand how we make a difference as individuals.

Curt.

That is something I struggle with.

I talk about it with you on the podcast.

Like that's so far our contribution is making it more widely known how it works, what's going on and also letting people know that it's not just about not using AI.

That's not actually.

It's here.

You're gonna have to use it at some point, but you're gonna be at a huge disadvantage if you don't understand how it works.

And it takes your brain a long time to understand how it really works.

It might feel like you understand it right away.

Your brain, it really needs to expand in a way that we aren't used to as humans.

And I'm still working on that.

If I ever find somebody who explains that better, because there definitely has to be a better explanation, I'll let you guys know.

But do you know what we should be doing as individuals?

Curt

You know, Melody's right.

They look nerdy and geeky, but they actually do a good job.

As you scroll the site, the page sort of interacts and sort of shows you what will happen in the world.

Melody

They have pictures, you guys.

Curt

There are pictures and graphs and, like, fun interactive stuff.

I think the first thing is just understand the magnitude of what's at stake here.

And then beyond that, I think that it's just really important that I've never held a picket sign before, but.

But how do I get the attention of my leaders to help them understand that?

I understand what you're trying to do here.

I know you want to beat China.

I know that we want to be the country in power.

I understand that.

But would you rather have all the power of a society that you're responsible for helping wipe off the earth, or would you rather have less power, less money, but be known as the agent of good?

Because now is the time before the inflection point to have an inflection point that we can make change.

We still have time to do this.

Melody

This is not like climate change, where we've had, like 20 years where people have been shouting about climate change, and yet it's been so slow that, like, we feel it, but we don't feel it.

This is real.

They know their whole thing is this is their job prediction.

So if we're going to take a guess, these guys are the ones who would be best at guessing.

So we should do an episode where we talk about the good of AI.

Curt

We do always sprinkle in the good that AI can do.

But today's episode was I'm trying to grab you by the shoulders and go with all the good the AI is doing.

I think I heard it said best.

Everyone has always known.

It always worked out.

It's always just worked out.

And so I think that we have that yet.

Just like that's where that cognitive dissonance has always been.

Okay in the past, but this might not work out now.

Listen, I'm a God fearing guy.

I do believe in an afterlife.

I believe that everything I do here in this life is really just sort of a precursor to the much more important life.

I'm not coming at this personally from like a frantic, like, I need as much life as I can possibly get because if I die, it's lights out and then it's over.

I don't believe that.

Melody

Personally.

I think there's a lot of people who believe that, but I'm not one of them.

We know the history of our world.

I never thought I would be a part of the end of it.

Same.

Curt

And that's coming from a guy who I thought during COVID like, oh, maybe this is the end.

Like, you know, But I've never felt the way I feel today is much stronger than the way I felt during.

Melody

COVID Yeah, I just had a thought though.

AI technically, if we believe in God in the Bible and all this stuff, if we follow that timeline, it couldn't destroy the world before the end times.

Like all the things happen.

Right?

Curt

So I'm coming at it from the perspective of I don't know if the Bible's true for sure.

Melody

I was just doing a thought exercise.

Curt.

I don't know either, but I'm just.

Curt

Saying for the sake of argument, let's just say it is true.

Like, AI could be the end, like mechanism.

Like it's just like the trigger of the mechanism.

So what I told you I'm not coming at it from is like, holy crap, if this is over.

Like I'm.

That's it for me.

I'm coming at this from the perspective of we've been given stewardship of this planet, this Earth, and we already know that greed is insanely powerful with the human race.

We know from World War II and concentration camps that humans will not even act in their best interest or do the moral thing whenever they think that in the short run they have something to gain.

We know from the psychological test that there's been tests Done where?

Well, actually, there's a great documentary.

I guess it's a documentary, but his name is Darren Brown.

He did a documentary called the Push.

And what they did was they.

They brought people in and they had them get hired for a job.

And the job was to just work one night at an event, catering.

But by the end of the night, they were going to have good people pushing people off the top of a skyscraper.

And if you haven't watched that documentary and you think I'm full of crap, I want you just to pay attention to how people who are open to suggestion, people who are not willing to stand up to the little thing and then the next big thing and the bigger thing and the bigger thing and the bigger thing, like the pattern is easy.

And Darren Brown shows you the whole time he's getting people conditioned from just a nice, friendly worker to someone who's going to push someone off the top of a building.

You see, humans are just not trustworthy.

And it's not because we're bad.

It's because we've been wired with fight or flight.

We've been worried survival of ourselves.

We will make it to our next meal.

We have these, like, primal needs and instincts and things that work inside of us.

And so the experiments that you may have heard of, where people were instructed in an experiment to like shock the person and keep turning it up if they keep, like not obeying the command.

And they literally are told, if you go beyond this, you'll kill them.

And they go beyond it again.

These are not like murder people.

Melody

These are like regular people.

Curt

Just regular people.

And now we have people who are in power who are also, at their core, just regular people.

And the power is being concentrated and the stakes are super high.

And if we don't do something, we're going to be the person at the top of the building who pushes someone off.

We know we don't want to be that person.

And that's where I'm coming at this from.

Melody

And the person we're pushing off is the entire world, right?

Curt

Or we're complicit in it.

Like, maybe that's a better way of putting it is that maybe you're not the one at the top pushing, but maybe you're complicit in it.

And in the Darren Brown the Push episode or documentary, these people have harnesses on and they have ropes because they know they're going to get pushed.

The person up there knows about the experiment.

Everybody is an actor except for the person who is doing it.

But that's where this gets a little bit dicey.

These people have to live the rest of their lives knowing that they were going to murder someone and that they were caught on film doing it.

That's brutal.

Brutal.

But I think that humanity needs to know this.

We just, we got to know how fallible we are and not get cocky.

Now is not the time to be cocky.

Melody

And we have to understand that we do not have the ability to understand the speed that this could happen because we've never ever experienced this ever, ever in humanity's existence.

Curt

Very rarely our brain can't even fathom exponential.

Like exponential isn't something we understand.

But when we talk about orders of magnitude, we're not talking about regular exponential.

We're talking about orders of magnitude 10xing in short periods of time.

And when we 10x you, we 10x things that have already been 10x.

Your brain can't comprehend a million.

Your brain definitely can't comprehend a billion.

Melody

So I think we did a pretty good job of trying to talk about what our concerns are, what the people who are actually in AI are concerned about.

I'm very interested.

I'll be attending the Open Data Science Conference for AI in end of October in San Francisco.

And I'm very, very interested to see what they talk about this there.

They will be talking about this.

They talk about.

I know agents are a lot on the list, but also talking about this.

And I'll be very interested because they're all experts.

Those people know what they're talking about.

And I want to hear it from the mouth of the people.

Curt

You use the word expert.

And I think for our listener there's probably quite a few who are like, well, I'm no expert.

I want you to know I'm in a few different communities for AI and I want you to know that someone who's a software engineer will probably tell you that they're an expert in AI because they use AI all day and they know how to spawn up agents and spin up different dev environments and be able to set up, like I told you, how I set up my testing and my, you know, they are experts in using AI as a tool.

Melody

In that one area.

Curt

Yeah, but do they know how neural networks work?

Do they understand the stuff that we've talked about today in terms of why we would need to beat another country?

Do they understand what super intelligence is or why it's be important?

I think in their heart, like they work with it enough that they can see a lot of the pattern.

Like if you work in AI Every day you can hear the voice of it and you can see a lot of the patterns of it and you understand many different things of it.

But sitting down and actually looking at it in the terms that we are, there is a good possibility that you, as a non AI expert, just by sitting down and listening to what you've said today, going in and looking at some of these resources, in some sense you could actually be more of an expert morally and ethically about what's going on.

Then many people who are asleep at.

Melody

The wheel, using it all day.

Yeah.

Curt

Using it all day, they just don't understand.

Right.

Because that's the thing.

Riding on a roller coaster and understanding what keeps it on the rails, they're two different things.

Yeah.

You've ridden a lot of roller coasters.

Congratulations, you're an expert roller coaster, but you're not an expert at the infrastructure of roller coaster.

And they are not the same.

Melody

Yeah.

Though I will say the conference I'm going to is a lot of academics and researchers who are studying this stuff.

And it doesn't mean they understand all of AI, but they, in their area of study, are experts.

They've spent, you know, sometimes decades in this area.

So that's why I look to them more.

And they're not always going to be in the public eye.

They're speaking at conferences that have to do with this specific thing.

Because this conference has been going on for a decade already or more.

But like, I want to go and I want to hear from people who are doing these understanding at a different level.

And it might be in a very small way because I don't know, I do want to understand the full scope of this.

I don't think I'm an expert and I don't think there truly is anybody who is an expert at AI, or else they'd be able to say that they are the companies who are building this.

They are not experts at AI.

They're as expert as one probably could be.

They know more about a lot of areas, but they don't know everything.

Curt

For example, they could be experts at like, the infrastructure of it all.

Like the same way that the guy who is doing all the stonework of a really fancy building, he can be a craftsman and an expert in building that building.

But when you ask him how the rock was formed or what would happen if that rock was exposed to certain conditions, in that sense, you know, you might.

This guy knows everything.

He built this stone, gorgeous building with the finest craftsmanship ever.

And it turns out that, like, he can't tell you how long that building is going to stand because he doesn't know what granite does he in an environment with 90% humidity.

Melody

He's not a geology expert.

Curt

Yeah, you said so.

I think it's also important to point out something because at the risk of us sounding like conspiracy theorists, I want to draw a few distinctions here.

So if someone says, I believe in flat earth, and if you don't, you should do your research on YouTube.

You should do your research, do some experiments on your own.

When someone does stuff like that, I tend to write someone off like that because I go, oh, okay, you got your whole education on YouTube.

And then I came to you and said, I listened to some people on YouTube and I want you to believe me and to research yourself.

The difference is, is that anyone who's following a conspiracy theory hole.

I'm not saying they're not convinced.

In fact, actually, they usually are extremely convinced.

The difference between what I'm asking you to do and what a normal person that's following something maybe more conspiracial, is that I am telling you that there are research papers out there is science, there is data behind it, and you can do your own experiments.

Melody

Sure.

Curt

But I'm not looking for excuses to make what I'm saying true.

I'm not looking and saying I'll only accept from this particular source.

I will accept from all sources.

Melody

I would love to be proven wrong so that I can go back to my old naive way of thinking about the future of humanity.

Curt

You know, I don't want to prove myself right or true here.

I don't want to prove these other whistleblowers right or true.

But I do see that with this exponential increase, I think we've already seen a bunch of whistleblowers.

I think we're going to see more.

They've paved the way for more.

And I think that's going to open it up for more.

And I think the truth is going to start coming out.

We as the people who.

Maybe that's just how you should leave this is be open to learning more about the other side.

I can't tell you what you're going to be convinced of.

I can only tell you that if.

If you take in more information from credible sources of people who are close to it, which, by the way, I'm not like, I'm not a credible source.

I don't work at an AI Frontier model company's research department.

But right now, this is the part of the movie where the guy at Jurassic park is telling everybody hey, I'm really concerned about this.

You're so busy moving forward that you never stop to ask yourself if you actually should because you're so obsessed with the fact that you can.

This is the part of the movie where, you know, the music starts to change.

Melody

And we aren't the experts.

We have our, you know, couple years of experience at this point with using it.

We've been following it or whatever.

But I think because we dive deeper.

Like, I'm always questioning whether something is right to do or use or not or whatever.

Like, I think that's why we tend to dive deeper into these areas.

And all I'm asking is for people to start paying attention, not just to the one voice that they're hearing, but just like, you look at all the sources, bring us sources.

We'd love to see some.

But I'm not looking for extremes, you guys.

I'm just.

Curt

No, actually, send me the extremes.

I'm fine.

Like, we gotta vet the source.

But here's what I want.

If you're gonna send me something that tells me that AI is safe and it's very transparent, I want to make sure that whatever you're sending me is coming from a source of somebody who actually can prove it.

I don't want to hear, yeah, like, an opinion of like.

Or I don't want to hear someone say, I talked to a researcher at Anthropic and they told me that I want to know who that researcher is, and I want to know under what conditions they said it.

Melody

If the companies themselves, who are risking billions of dollars, are saying that we are at risk, that is enough evidence for me.

Like, it goes against every one of their best interests as corporations or even, you know, I know OpenAI is not a.

Like, it's nonprofit, but it really is a corporation for sure.

Curt

They wouldn't even.

Come on.

They're not that naive.

Melody

No.

So, like, the point is, like, we need to be concerned, because the people who should not be talking about it, they're concerned it's not in their best interest, and yet they are starting to talk about it.

Curt

I want to hear from people closer to it.

I don't even just hear their words.

I want to know what they're doing.

And I think that's, for me, with both Jacob and Daniel and Evan and these other people that we're citing as whistleblowers, they all sacrificed a ton and were willing to give up a ton to just get the word out that they couldn't do in the place that they were.

Melody

There's Always the other side, that they can benefit from this.

They can get funding for their things that they're doing.

But I would say that we've invested a lot of hours now in listening to full long interviews with people watching things about this.

And I feel like this is not just a 90 second TikTok that we're talking about.

This is like months.

I think I've been concerned about this for a long time at this point, but it's become really.

It's in our face now.

We cannot pretend it doesn't.

It's not a possibility anymore.

So we've said it for an hour and 40 minutes.

Curt

Yeah.

And I appreciate all of you who have stuck with us today.

Maybe this isn't really a topic that maybe you're not a. Oh, this is just tech talk.

I hope you understand by now that this is so much more about how we show up in the world and what we as a people are.

And now we're experiencing something that is going to move so much faster.

When I look at what the experts are saying, they're saying, we're not telling you to stop using AI.

We're not telling you that, you know, there is no hope.

The fact is, is that AI could actually build us a utopia where you don't have to work at all, that you won't be able to make an economic contribution to the world.

And you know, as a guy who loves to work, that doesn't sound great.

Melody

I think we all need meaning and purpose in our life.

Curt

Yes, but we can find meaning and purpose outside of economic.

Melody

Yes, we can, if we're able to do well.

Curt

And I think that's what we're striving for.

That's what can be really cool about this.

But it's hard to imagine because none of us have ever even imagined that.

And here we are.

Melody

Well, Andrew Yang did, but that's besides the point.

Well, thank you.

I will end on the note of.

I'm just going to say baby laughs they're the best thing ever.

Curt

Go spend some time with the baby.

Melody

And I think just I still do have hope.

I have to.

We have to have hope.

There's no other way to move forward.

And I believe it'll probably become more clear as this becomes bigger and in front of people how we as individuals can make the difference.

Because right now it feels like, oh, I'm just going to write my congress person.

Like that'll work, right?

I think it's going to get to the point where there's going to be real things that we can do.

And as we find those things, we will put them forward to to you, our audience as well.

Thanks Curt.

Curt

Thank you, Melody.

It's always fun to go deep with you and today we just definitely went deep.

Melody

And thank you audience for sticking with us.

Curt

We'll see you guys later.

Thanks everybody for joining us at the Soul Proprietor Podcast.

It has been an absolute pleasure having these discussions with you.

If you wouldn't mind taking just a few minutes to rate and review us wherever it is that you listen to podcasts, it would mean so much to us.

We really do read each of these reviews and it gives us the opportunity to get the word out to more people who could benefit from hearing about topics like this and so many others.

If you want to engage with us at our website, maybe share some topics or ideas of other people that you'd like to hear on the podcast, feel free to go to soulproprietorpodcast.com and share with us your thoughts and ideas about what we could do in the future to bring even more light into the world.