Heya and welcome back to Five Things AI!
Read on my dear, or have your agent recite this for you.
French A.I. Start-Up Mistral Raises $3.5 Billion as Part of Strategy Shift
Mistral is trying to take advantage of growing concern among many businesses and governments in Europe and elsewhere about overdependence on American technology. Companies like Anthropic and OpenAI charge customers a premium to use their A.I. systems but provide little access to the underlying data or models. President Trump’s decision in June to use an export ban to briefly limit international use of Anthropic’s most advanced model also caused alarm.
This is a big one. Even though there is some competition in Europe, Mistral is by far the continent’s most recognized LLM. It’s important that we make sure we can rely on a European stack for production.
Why some experts increasingly fear AI will take over
For years, researchers concerned about existential AI risks have argued that powerful systems could eventually act in ways that conflict with human interests. Critics often refer to them as “AI doomers”.
But as details of the OpenAI incident have emerged, those concerns have grown, including among some researchers working in AI labs.
The Silicon Valley giant’s chief scientist, Jakub Pachocki, said the risks associated with AI are “unfortunately going to grow from here” as he and others are building what he calls “an alien intellect exceeding our own”.
We need better regulation and we need to enforce it. It really is that simple. Maybe we also need to think about licensing, just like when you want to get on the airwaves. While I am certainly concerned, I am not yet alarmed.
How Would AI Actually Kill Us All? What to Know About the AI Doomsday Debate
So-called doomers aren’t usually worried that the “Terminator” movies will come true, with robots actively trying to exterminate people. Their concerns fall mostly into two buckets: loss-of-control and human misuse.
In one, highly intelligent AI agents capable of copying and improving themselves start pursuing their own goals and destroy humanity in the process. In the other, a nefarious human enlists a capable AI to do something like create novel viruses that wipe us all out.
There are also catastrophic scenarios that fall short of extinction, such as massive cyberattacks that take down power grids or financial systems and lead to a breakdown in the human social order.
Hmm, should I be more worried?
I Let an AI Agent Hack All My Gadgets—and I’d Do It Again
To get a closer look, I decided to unleash one in my own home network. Over the course of a few days, I watched as my own rogue agent found vulnerabilities in various household devices, hacked into a PC, and showed me that several vibe-coded projects were—unsurprisingly—riddled with bugs. (My wife knew what I was up to, and rolled her eyes each time I proudly announced the discovery of a new vulnerability.)
But Will, you might be thinking, giving an impish, all-powerful cybersecurity agent access to your home network is batshit. And you would be correct! Nevertheless, I believe that a good way to understand the cybersecurity hellscape in front of us is to pay it a visit.
In the end, my experiment was revealing, but oddly reassuring, too. My little network gremlin showed me how vulnerable my home life would be to AI hacking, but it also told me how to make everything a lot more secure. In the end, I discovered that the best way to deal with AI hacking may well be having your own AI hacker.
Would you do that? While I certainly think that this is an interesting experiment, I’d rather not have AI tamper with my gadgets at home.
I Worked on Safety at OpenAI. This Is What It Should Do Now.
But it’s not necessary for the A.I. industry to wait for collective action. There are simple steps any A.I. company could take, today, to reduce the danger we face.
The first step would be to commit to meaningful incident disclosure, including defining what incidents warrant bringing in third-party oversight. Like in aviation, private companies should disclose not only actual breaches of safety, but near misses as well, lest we pay for each lesson with a tragedy. If A.I. companies continue to hide their scary incidents, they are robbing us of the scientific know-how to avert more serious catastrophes.
I am sure that nothing will happen without government intervention. We need to act quickly, but it doesn’t look as if the US government will do anything reasonable anytime soon.
If you missed last week’s edition of Five Things AI, you can read it here:
That’s it for Five Things AI this week! 🤖
— Nico