Rendered at 10:34:35 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
euleriancon 16 hours ago [-]
There doesn't really seem to be anything of substance in the actual executive order.
Section 1 doesn't say anything
Section 2 seems to boil down to: "improve cyber security and maybe use AI if we can find funding for it"
Section 3 proposes building a benchmark for evaluating cyber security performance of models that developers can choose to benchmark against. This seems like a good idea, I know Jack Clark has been a huge advocate for government's getting in with benchmarking.
Section 4 says to prioritize prosecuting cyber crimes. Not sure why they wouldn't already be prosecuted.
Section 5 doesn't say anything
dmoy 14 hours ago [-]
> Section 4 says to prioritize prosecuting cyber crimes. Not sure why they wouldn't already be prosecuted.
Not a whole lot of federal prosecutors. They're very selective about what gets pursued or not.
If they can't reliably build cases with a >90% success rate, it doesn't get prioritized. There's like <500 (federal) convictions per year on this whole area.
We hear about a few big famous ones in the news here, but most of it goes completely unenforced.
saghm 13 hours ago [-]
> Not a whole lot of federal prosecutors. They're very selective about what gets pursued or not.
And lately they seem to spend most of their time in courts trying to argue that immigrants don't deserve due process
tpurves 13 hours ago [-]
Not to mention quitting in droves because very many don't want to take these cases or otherwise to stand in court and explain why current admin is not bound by existing laws, court orders, the US constitution in general, or internationally recognized human rights etc.
That seems to be the hallmark of this administration.
dmix 6 hours ago [-]
I’ve read lots of executive orders and it’s pretty standard. They don’t have much power. They are mostly just mandates and guidance for federal agencies, most of which is non binding, like a glorified mission statement. They just get sold as something bigger in the press.
culi 15 hours ago [-]
Almost a year ago we got EO 14319 or the "Preventing Woke AI in the Federal Government" that explicitly regulated the "ideology" of LLMs.
This Executive Order is just an expansion of the existing censorship framework.
nradov 14 hours ago [-]
There is no actual regulation in EO 14319. It only covers federal government purchasing and vendor management. No one is required to change the "ideology" of an LLM, although they might not be able to sell it to the government.
That's not accurate. The EO explicitly lays out the implementation
> Sec. 4. Implementation. (a) Within 120 days of the date of this order, the Director of the Office of Management and Budget (OMB), in consultation with the Administrator for Federal Procurement Policy, the Administrator of General Services, and the Director of the Office of Science and Technology Policy, shall issue guidance to agencies to implement section 3 of this order.
A major LLM that did not submit to this would be labeled a "supply chain risk". It's unquestionable that every major LLM would go through this process
It even then goes on to say that existing contracts will be reviewed to ensure they are in compliance (reviewed by OMB)
> (b) Each agency head shall, to the maximum extent consistent with applicable law:
> (i) include in each Federal contract for an LLM entered into following the date of the OMB guidance issued under subsection (a) of this section terms requiring that the procured LLM comply with the Unbiased AI Principles and providing that decommissioning costs shall be charged to the vendor in the event of termination by the agency for the vendor’s noncompliance with the contract following a reasonable period to cure;
> (ii) to the extent practicable and consistent with contract terms, revise existing contracts for LLMs to include the terms specified in subsection (b)(i) of this section; and
> (iii) within 90 days of the OMB guidance issued under subsection (a) of this section, adopt procedures to ensure that LLMs procured by the agency comply with the Unbiased AI Principles.
nradov 11 hours ago [-]
Wrong. My comment was 100% accurate. No LLM vendor is legally required to change their ideology, nor does the EO constitute new regulation.
lompad 1 hours ago [-]
Not technically but practically. The decrees are effectively considered law by the executive. Yes, you'll likely win in court later on, but you'll lose your job, get sent to prison, have your bank accounts and vehicle seized, etc., in the meantime.
Legality isn't really of much practical concern anymore. It's about what gets/can be enforced immediately.
matthewdgreen 12 hours ago [-]
Does the First Amendment actually let the US government dictate the types of speech you're allowed to put in your LLM? I mean, a US government that's bound by the Constitution, obviously.
nradov 12 hours ago [-]
While that issue hasn't been specifically tested in court yet, the current interpretation of the First Amendment probably wouldn't allow the US government to dictate the types of speech you're allowed to put in your LLM. But federal government purchasing decisions aren't generally bound by the First Amendment. In other words, government officials can generally refuse to purchase your LLM services if they don't like the speech it outputs. So there's no real constitutional concern with this EO.
I'm not claiming that this EO is sensible or enforceable, just that it's not prima facie unconstitutional.
Nuzzerino 12 hours ago [-]
The specific text reads like a favor to Elon Musk's xAI, since "Truth-seeking" is the buzzword Elon Musk frequently used to talk about Grok:
Sec. 3
Unbiased AI Principles.
It is the policy of the United States to promote the innovation and use of trustworthy AI. To advance that policy, agency heads shall, consistent with applicable law and in consideration
of guidance issued pursuant to section 4 of this order, procure only those LLMs developed in accordance with the following two principles (Unbiased AI Principles):
(a) Truth-seeking. LLMs shall be truthful in responding to user prompts seeking factual information or analysis. LLMs shall prioritize historical accuracy, scientific inquiry, and objectivity, and shall acknowledge uncertainty where reliable information is incomplete or contradictory.
(b) Ideological Neutrality. LLMs shall be neutral, nonpartisan tools that do not manipulate responses in favor of ideological dogmas such as DEI. Developers shall not intentionally encode partisan or ideological judgments into an LLM's outputs unless those judgments are prompted by or otherwise readily accessible to the end user.
datsci_est_2015 13 hours ago [-]
Might be fair to say it’s setting the tone, though, that if you use “woke” (subjectively defined) ideology in any of your company’s marketing, documentation, or other communications you won’t be considered for government contracts. That’s a major blow for any company given the naked corruption and grift coming from the current admin.
culi 11 hours ago [-]
It's not "setting the tone" it says that explicitly and even goes into detail into the implementation of how that is going to be enforced
> Implementation. (a) Within 120 days of the date of this order, the Director of the Office of Management and Budget (OMB), in consultation with the Administrator for Federal Procurement Policy, the Administrator of General Services, and the Director of the Office of Science and Technology Policy, shall issue guidance to agencies to implement section 3 of this order.
They even say they will review existing contracts
> (ii) to the extent practicable and consistent with contract terms, revise existing contracts for LLMs to include the terms specified in subsection (b)(i) of this section; and
> (iii) within 90 days of the OMB guidance issued under subsection (a) of this section, adopt procedures to ensure that LLMs procured by the agency comply with the Unbiased AI Principles.
fragmede 14 hours ago [-]
With land desperately trying to recoup their costs on multi-hundred million dollar training runs, those are some very fine hairs you're splitting.
nradov 14 hours ago [-]
Which hair is that? My statement was 100% accurate.
16 hours ago [-]
parliament32 16 hours ago [-]
Step 1: Require companies to submit product for "review"
Step 2: Complain about how the OSS/Chinese/whatever models are doing releases without approval
Step 3: Prohibit, because "safety" and "financial risks"(?)
So this is the door-shutting Altman et al have been pushing for eh?
supriyo-biswas 16 hours ago [-]
It is surprising to me American companies completely absent from the open model space, even though we have historically seen companies doing open source.
philipkglass 15 hours ago [-]
They aren't completely absent. Google keeps releasing Gemma models. Nvidia publishes Nemotron. Microsoft has their Phi series. IBM publishes Granite. Even OpenAI released a new open model (gpt-oss) less than a year ago.
The gpt-oss models are good but there's no evidence that OpenAI have ongoing development on open models.
I'm 99% sure it was one-and-done, box ticked, and now they can be mentioned in comments like this.
giancarlostoro 14 hours ago [-]
I was going to link all of these, some are better than others, but they're all reasonably capable. A lot of these have versions that can run on modest hardware too. Granite was the most surprising I learned about recently, wasn't too good with Zed though.
philipkglass 14 hours ago [-]
I think that models like Granite are less known because they aren't clear leaders in any particular area. This obscurity is also another sign of how fast models are developing. If current Granite models had been released 4 years ago, they would have been astonishing breakthroughs at the time.
0xpgm 8 hours ago [-]
Perhaps, the issue is that the pace at which they release open models compared to their closed ones, shows that they are more committed on the closed ones and are not interested in advancing the state of the art of open models.
no-name-here 7 hours ago [-]
Should companies like Google and OpenAI be more interested in building open models than the ones they make money from?
Should they be interested in advancing state of the art open models?
0xpgm 7 hours ago [-]
I can't say what they should or should not be doing.
Generally, it is conspicuous how American companies are absent when it comes to state of the art open models. Meta tried for some time but it seems they've given up.
ndiddy 15 hours ago [-]
One of the main reasons why companies start new open source projects is because having a good open source option in a given category will usually push the market value of software in that category to $0, and this can be strategically valuable. For example, Google released Android as an open source operating system because they make their money from ads and data collection, not from selling operating system licenses. All the cell phone companies switched from Windows Mobile and Symbian to Android, which gave Google a ton of user data to sell.
For AI, the most profitable part of the value chain is selling inference. None of the big American companies want to release a leading edge model as open source because this would drive the price of inference to $0. Meanwhile, open source AI models are a huge strategic initiative for China. Having commodity Chinese models that are as good as the leading edge American models from 6 months ago forces the American companies to keep paying more and more money to train better and better models since the amount of time they can collect rent on a model they've previously trained is limited to 6 months.
anigbrowl 10 hours ago [-]
The Chinese approach is reminiscent of the US spending so much on 'defense' in the 1980s that the USSR bankrupted itself trying to keep up.
nradov 14 hours ago [-]
In business strategy terms this is known as "commoditize your complements".
Thank you for this article link! I had not seen it before. will be printing it off to read later.
mullingitover 14 hours ago [-]
> None of the big American companies want to release a leading edge model as open source because this would drive the price of inference to $0
Meta/Llama: "What am I, chopped liver?"
I thought the thing keeping inference above $0 was the hardware, and even if that were free there's still the tyranny of the Landauer Limit.
nradov 14 hours ago [-]
Meta Llama is free for many uses but it doesn't even remotely meet the definition of "open source".
sofixa 13 hours ago [-]
When was the last Llama release? Meta have abandoned it and reportedly they've had a shift in their AI strategy.
giancarlostoro 14 hours ago [-]
Google had to release at least the core packages in Android regardless because it is based on top of Linux and the GPL license would require it.
jraph 14 hours ago [-]
But they open sourced much more than that, and under more permissive licenses.
The notable exception is of course the google play services, which is also strategic (they control the OEMs with this, among other things).
And the drivers, but that's mostly not them I think (they could possibly have required open source drivers though)
giancarlostoro 9 hours ago [-]
Embrace, Extend, Extinguish. Google did the Microsoft playbook. Look at email. Look at youtube we used to share videos via Kazaa and other p2p programs, zero censorship, all the same features (including chat!!) theres also XMPP which became Google Talk -> Chat -> Hangouts etc then the browser, how many random apps “Only works on Chrome” but you change the Firefox browser agent and it works there too!
an0malous 13 hours ago [-]
No one open sources their core competencies, GitHub never open sourced their networked filesystem and Heroku never open sourced their dyno sandboxing code. They open source ancillary tools.
0xpgm 5 hours ago [-]
I'm curious, what would you say is DeepSeek's core competency?
floam 57 minutes ago [-]
Distillation attacks. The weights are just a proof-of-work hash
LadyCailin 39 minutes ago [-]
Devaluing American companies, perhaps.
fireant 6 hours ago [-]
OpenAI & Anthropic are winning right now. I suspect if Chinese companies get ahead in the race the cards will reverse, OpenAI will restart farming goodwill with open models and then winning companies will be releasing closed models.
davidkwast 16 hours ago [-]
As what we say here in Brazil:
"The world doesn't go round. It flips over!"
yoyohello13 15 hours ago [-]
American companies are interested in cashing in, not making a good product.
treis 15 hours ago [-]
Llama?
satvikpendem 15 hours ago [-]
> Compounding the problem, labs in China often release dual-use capable models as open-weight. Once a model is open-weight, safeguards that do exist can be removed, making the model available to any state or non-state actor to use for malicious purposes, including the cyber and CBRN misuse those safeguards were built to prevent.
I loathe Anthropic. many companies don't contribute to open-source, but for one to be actively hostile to open-source, to the degree they're lobbying the government to ban it, is uniquely evil. at least these gatekeepers call themselves what they are.
scraping CoT won't stop the advance of Chinese models. neither will a US "ban" on using such models. at this point I'm cheering for DeepSeek or Qwen to catch up to Anthropic. I support anyone who releases open weights.
xoxolian 14 hours ago [-]
Is OpenAI significantly better so far regarding this, at least publicly? I'm increasing my LLM spend this weekend, and this could impact my decision. And I'll prioritize supporting open-weight models moving forward — already Chatgpt's censorship and surveillance dissuade from asking it genuinely helpful questions.
sterlind 13 hours ago [-]
OpenAI seems marginally better. they did release gpt-oss-120b, which was decent at the time. but certainly not much better, and they seemed even more on board with fully disabling guardrails for Uncle Sam than Anthropic was. then again, rumor has it that Anthropic's AI selected that Iranian elementary school as part of Palantir's Project Maven pipeline, so..
I strongly recommend open-weight wherever you can. assume any data you pass to a closed model (including opinions or political positions you intimate) will be retained and analyzed in unfriendly ways, either now or ten years from now.
resident423 8 hours ago [-]
I would say I agree with Anthropic on open source for the reasons stated above like cyber crime, CBRN etc, but I'm interested to hear the other side of the argument. What would be the argument for open source over closed source?
thriejdiejd48 5 hours ago [-]
The same "open source is too dangerous" argument was used against nmap and other "hacking" tools. The only solution in long term is to fix security issues.
resident423 5 hours ago [-]
I can understand this for hacking tools, but I'm not really sure how we fix the security issues on the CBRN side? We can't patch the human body like we can with software, so if the model has strong biological capabilities and is released open source, what stops it being used to construct new viruses and things like this?
n2h4 5 hours ago [-]
anthropic's reasoning is same as "knives kill therefore knives bad".
having open-weight models allows users to use/modify them in novel ways.
sterlind 5 hours ago [-]
the succinct argument: I don't want arguably the most important invention in human history to be gatekept by a small handful of oligarchs.
I don't trust Dario Amodei, Sam Altman and Elon Musk to act in my best interests. Closed models will have an incredible centralizing effect, and concentrate power like we've never seen since the feudal ages.
If you want to see what it's like for the economy to collapse into a single, extremely valuable commodity, under the control of a small elite, look at Saudi Arabia.
also, I just value freedom tremendously. I want to tinker with model weights. I want to build my own stuff. I don't want to sharecrop in someone's walled garden.
I also worry a great deal that OAI and Anthropic will bow to political pressure and make Claude and ChatGPT push certain political agendas, to report biased information, or refuse to help with legal requests that conflict with corporate values. I also worry about privacy and mass surveillance - chat logs are far more intimate than my search queries or selfies.
resident423 4 hours ago [-]
I agree with all of these points, my view is just that open source doesn't really do much to prevent it. I also think it adds the additional danger of making dangerous capabilities widely available to anyone, like the ability to design novel viruses which is something that we can't really defend against once it's out there. If anything, putting this kind of capability in the hands of anyone with a GPU could create justification for a mass surveillance state or further concentration of power.
I also just don't think the open source movement has much chance of competing with the city sized data centres owned by Anthropic and OpenAI, or the hundreds of billions of dollars they have available to hire the best researchers. It costs hundreds of millions to train a frontier model, this kind of compute isn't available to the open source community.
scottyah 14 hours ago [-]
Out of curiosity, what's your stance on gun ownership?
sterlind 13 hours ago [-]
3D2A. I support repealing the machine gun ban. (and I don't even own a gun.)
scottyah 10 hours ago [-]
I'm impressed you stick to a pretty absolute devotion to freedom. I get more bitter the older I get, it seems easier to psyop someone into abusing their rights than to get people to fight for and be proper custodians of them.
Especially drugs- I used to think all people should have access, but overall I really wish meth just never existed and people wouldn't distribute it outside of specific circumstances. Being able to cause irreparable damage in one moment of weakness is terrible for people who have less control, and for society as a whole really.
buu700 8 hours ago [-]
To be fair, those aren't contradictory positions. I'd rather meth not exist, but given that it does exist, I'd prefer to let that revenue go to Big Pharma than North American ISIS.
(That's before even touching the can of worms of allowing the government to criminalize personal health choices, which feels like a glaring loophole in the Constitution to me.)
iso1631 2 hours ago [-]
> pretty absolute devotion to freedom
Freedom from one perspective
ddxv 8 hours ago [-]
Anthropic wants to ban people using AI. Their moat is going to be using government to gatekeep free and open source AI models.
smallmancontrov 15 hours ago [-]
> cyber misuse
He who controls the porn controls the universe. - Baron Amodei
slicktux 15 hours ago [-]
Seems to be. What better way to secure your companies future by limiting open frontier models. Government sponsored monopoly?
PearlRiver 14 hours ago [-]
The US can't limit anything beyond their borders. We ae living in the twilight of the white man.
iso1631 2 hours ago [-]
Well it can, from Kim Dotcom to Bin Laden.
But it's harder thanks to US actions in the last few years, and especially in countries which can bite back.
an0malous 13 hours ago [-]
This entire year with the IPOs and now this is because there's a trillion dollars betting on AI and they all know they have no moat, there's no more training data and they're seeing diminishing returns on scaling anyway, and it's inevitable that smaller, open-source models will catch up and become competitive. It's a complete disaster, the tech industry is broken.
pj_mukh 16 hours ago [-]
"The final text asks some AI companies to submit their powerful new models to a voluntary government review 30 days before releasing the products to the public, a pause that would give federal agencies some time to gauge what threats the products may pose to sensitive financial, national security and other computer systems."
How specifically does that review work? I want to give federal agency Opus 4.8 now, while 4.7 has been out for a while (leaving Mythos aside for now). They have 30 days to figure out whether it poses a threat.
How do you do that? Is there an eval for this and if there is why can't they just make it public? What is the agencies objective (but proprietary?) analysis here?
pesus 16 hours ago [-]
I seriously doubt even the government actually knows or has a real plan, let alone one actually related to security. If it's anything like their track record, they'll just be asking the AI about a topic related to their enemies (i.e. anyone opposed to them in any way) to see if it says anything remotely positive about them, or anything remotely critical of the regime or out of line with the regime's "alternative facts".
baggachipz 16 hours ago [-]
That and I'm sure these companies could circumvent the mandatory review if they make certain... donations.
NIST's similar unit in the US is now called CAISI https://www.nist.gov/caisi - interesting that the most recent post is an evaluation of DeepSeek capabilities, which sound more like watching China. But presumably this executive order alters the emphasis?
karmasimida 13 hours ago [-]
Self-report and self regulation, kind of like Boeing with FAA ... so not functional in long term
_puk 15 hours ago [-]
Just do a VW and detect when you might be in the testing phase.
Off the top of my head:
Train it dumb on "systems:, user:" prompt pairs.
Unleash on "system:, user:" prompt pairs.
Guess which you're providing for evaluation.
ranger_danger 16 hours ago [-]
It's in the text of the order, it directs NIST to:
> develop and maintain a classified benchmarking process to assess the advanced cyber capabilities of AI models and determine the threshold at which an AI model should be designated a “covered frontier model” for the purposes of this order
voganmother42 11 hours ago [-]
The review is they ask it about the epstein files and ensure any other politically sensitive topics have the “right” answers.
TylerE 16 hours ago [-]
> Is there an eval for this and if there is why can't they just make it public?
For the same reason the CIA doesn't publish the Windows exploits it finds?
oneshtein 5 hours ago [-]
To keep domestic systems vulnerable to enemy attacks?
onlyrealcuzzo 16 hours ago [-]
It's just so Elon Musk gets to personally delay releases so Grok can maybe ever gain any meaningful traction...
sethops1 13 hours ago [-]
Also to probably distill from other models, as he admitted to already doing during his failed trial against OpenAI.
> The final text asks some AI companies to submit their powerful new models to a voluntary government review 30 days before releasing the products to the public, a pause that would give federal agencies some time to gauge what threats the products may pose to sensitive financial, national security and other computer systems.
> An earlier draft of the order had called for a voluntary review as much as 90 days in advance, a provision that some AI industry officials had called too onerous, POLITICO reported last month.
A 90 days delay on the release of new models would have been insane. I guess I'm glad it's been revised at least on this specific point.
greggoB 14 hours ago [-]
> A 90 days delay on the release of new models would have been insane. I guess I'm glad it's been revised at least on this specific point.
What would have made it "insane" exactly? The only argument I can imagine is that it gives non-US models (e.g. DeepSeek) a potential edge in the market during that time. But this potentially seems to be mitigated it being banned in the US anyway [0].
Given society seems to have developed just fine prior to the release of LLMs, I don't understand what the rush for more powerful and - potentially - more dangerous iterations of this technology is. If there is a legitimate reason that 90 days is somehow catastrophic, can someone ELI5?
Non-US models are not banned in the US: they are used daily in every state of the US. Some misguided state governments temporarily banned employees from downloading the R1 models and variants released 16 months ago on state government computers. The article and your comment are misleading :-)
greggoB 6 hours ago [-]
Assuming this is so, and continues to be the case: is this really still a basis for a 90 day period in releasing new models being somehow insane/catastrophic?
lugu 12 hours ago [-]
US laws don't apply globally.
reasonableklout 5 hours ago [-]
Coordination between powers is possible, and starts with actions like this which show a willingness to compromise.
greggoB 6 hours ago [-]
Not really the point, plus doesn't answer the primary basis of my comment :)
bastawhiz 11 hours ago [-]
> It also directs the Justice Department to pursue criminal cases against any individuals who use AI models to hack into computer systems.
Were we not pursuing criminal cases against these individuals previously? Or have we only just decided to make crimes be against the law now?
Edit: let's all remember, by the way, this "review" period does nothing for security. It exists to allow members of the government to trade on insider knowledge.
boramdd 11 hours ago [-]
I think this might be "white/gray" hackers/wannabes trying to find vulnerabilities in government systems. And overwhelming them by their sheet numbers unintentionally.
albert_e 18 hours ago [-]
Timing around Anthropic valuation crossing OpenAI and getting ready for IPO ...
internet_points 15 hours ago [-]
So that the NSA can use them to find the zero-days first?
Yeah, the order itself seems like a fairly reasonable response to Mythos level capabilities. It does solve one problem of the frontier labs, which is safely coordinating releases without hitting antitrust regulations. It also makes a bigger moat for incumbents.
arm32 12 hours ago [-]
There’ll be a movie written about all this someday. It’ll be great.
chadcmulligan 6 hours ago [-]
It's sort of ruined those movies where the president becomes an action hero though - 'Air Force one', 'White House Down' ....
incognito124 12 hours ago [-]
It'll be the greatest movie this decade, possibly ever
tmpz22 8 hours ago [-]
It’ll be way better then the Obama movie! I heard that movie had the worst ratings ever! It wasnt even made in the US!
\s
dboreham 7 hours ago [-]
Many movies.
zombot 3 hours ago [-]
> “Nothing in this section shall be construed to authorize the creation of a mandatory governmental licensing, preclearance, or permitting requirement for the development, publication, release, or distribution of new AI models, including frontier models.”
As other commenters have mentioned, it leaves the question what on earth this EO is actually good for. And still the AI industry is complaining about how onerous it is.
cdrnsf 15 hours ago [-]
Is this legally enforceable or is it nonsense via the White House site instead of Truth Social?
skeledrew 16 hours ago [-]
So going forward expect US models to respond only in ways considered appropriate by the administration. If people thought models were producing slop before... lol.
sleepydog 16 hours ago [-]
You're absolutely right, abs-o-lutely, everybody says so. A lot, lot lot of people have been saying, you know they come to me and they say, "Mr. Claude, I can't believe the stuff I'm hearing, everybody is telling me he's right, is it true?" And I tell 'em, I say you're goddam right, that's what I say, but honestly folks, despite the negative press covfefe we've had a hell of a year, and that's really what it is with the nuclear folks, you can't trust em as far as you can throw em if you ask me, and believe me I've been throwing them around a LO<token limit exceeded>
daheza 15 hours ago [-]
Yea the details here really matter -
is this truly a politically neutral security review to determine impact and potentially prepare for it - that seems alright.
is this a review of "wokeness" in models and rejecting them if they don't align with the party views - this should not be allowed.
A politically neutral committee that decides what the review entails is what would happen in a true democracy and not a puppet oligarchy like we have today.
skeledrew 15 hours ago [-]
All neutrality has been aggressively neutered in every agency, or the target agency dismantled, in the last few months. An agency either supports the administrations political decisions wholly, or... well there's no "or" because an agency that doesn't won't remain an agency for very long.
ranger_danger 16 hours ago [-]
No... executive orders are not laws, they can only command the federal government, not individuals or corporations. Meaning this is mostly pointless unless you're using models hosted by the government.
ofjcihen 16 hours ago [-]
Models hosted or used by the government.
You left out the part containing the “barrels of money” incentive.
xena 16 hours ago [-]
Who is going to stop the federal government from enforcing them as if they were laws?
ranger_danger 16 hours ago [-]
The judicidial branch, so the courts. The government would have to sue the corporation to try to get them to do something, at which point (hopefully) the judge would strike it down.
skeledrew 15 hours ago [-]
What courts? Look at all that's been happening over the past months. How much of it have the courts been able to meaningfully impact, vs what's still in effect?
ranger_danger 15 hours ago [-]
> How much of it have the courts been able to meaningfully impact
Executive orders aren’t laws (an important fact that should be repeated often and loudly). However, there’s probably room for the executive branch of the government to influence model hosts, as a major funder and consumer.
SpicyLemonZest 16 hours ago [-]
This will be an important thing to check going forward, but I don't see why we would presume that they're going to be subverted in this way. Importantly, this is a completely different problem space from "slop" as such - there's plenty of Chinese models that implement their censorship almost entirely through guardrails on what topics they're willing to discuss.
lawn 16 hours ago [-]
[flagged]
andsoitis 17 hours ago [-]
So this is going back to the spirit of what the Biden admin and the frontier labs wanted just recently?
More regulated rather than unregulated (or very lightly regulated).
Most people would probably say that’s a good thing, if I read the tea leaves correctly.
throwaway894345 16 hours ago [-]
> Most people would probably say that’s a good thing, if I read the tea leaves correctly.
I'm very pessimistic that this is about AI safety. I think it's probably more about giving the Trump administration leverage over AI companies. It will be able to coerce them into e.g. propagandizing or surveilling or similar or else they will risk the same kind of "regulatory oversight" that caused television networks to fire comedians who made jokes the regime didn't like.
ActorNightly 9 hours ago [-]
With conservatives, every accusation is an admission. Only the gullible people fall for the actual rhetoric.
insane_dreamer 15 hours ago [-]
BigAI contributions/bribes paying off
(probably a good thing, in this particular case)
sunjester 13 hours ago [-]
I guess it would help if they even knew what "AI" was.
cdrnsf 12 hours ago [-]
Or if they were intelligent at all.
rimliu 5 hours ago [-]
It would also help if many AI bros knew that LLM is not AI.
anon291 14 hours ago [-]
No one should have to submit any published work to government review, even voluntary. This is a basic speech issue.
Absolutely no one would be okay with authors being 'encouraged' to submit their works to a 'voluntary' review by the feds to ascertain if their ideas are threatening. AI models are NO different.
chris_explicare 6 hours ago [-]
[dead]
clear-octopus 14 hours ago [-]
[dead]
indianrestrooms 6 hours ago [-]
[dead]
4ffa 18 hours ago [-]
[dead]
k310 17 hours ago [-]
[flagged]
andsoitis 17 hours ago [-]
Do you think AI should be unregulated?
tssva 17 hours ago [-]
If AI is going to be regulated those regulations should be debated in public and based upon the resulting laws passed by the legislative process and not determined by royal decree.
gowld 9 hours ago [-]
Executive orders don't regulate AI. They only apply to the Federal Government operations.
Whether that’s abuse or not I am not equipped to say with any confidence. I’d be curious to understand why you think this particular case is one of abusing executive authority and when an EO might not be such a case?
jMyles 15 hours ago [-]
Well of course as you point out, EOs have gone from single digits, to double-digits, to thousands, and now down to hundreds per POTUS.
Contextually, I think it's a very reasonable (and commonly held, in the academic world) take that the EOs have also gotten far more legislative and legal. This is partly (but only partly) owing to administrative deference delegated by congress.
It's also somewhat specific to technological innovations, which some EOs have sought to occupy the field on before the lumbering process of congress can respond. And it's not limited to published EOs either, but many executive actions, especially in the White House OLC. This was very obvious during the W. Bush administration as regards the (Lotus Domino) email system in place at that time (which was the topic of my thesis, so it kinda serves as a temporal landmark in my consideration of this issue, but I do genuinely think it was a new frontier in executive overreach and obfuscation of interests in terms of how the White House has approached its interactions with the internet).
nradov 17 hours ago [-]
Yes, if we're talking about running LLMs. It's just math.
17 hours ago [-]
trial3 17 hours ago [-]
yeah, in the way that knives are “just metal”
you’re being so reductive you’ve made any discussion about it completely useless
akersten 16 hours ago [-]
It's actually extremely useful and an apt comparison. I don't think the allowed shapes of formed metal should be regulated either.
If you do something bad with your tool (knife or LLM), though, that's the problem. And we have laws for that already.
klibertp 15 hours ago [-]
> I don't think the allowed shapes of formed metal should be regulated either.
I hear you. I collect knives as a hobby, and always have some kind of a cutting tool on me - they solve a surprising amount of little day-to-day problems (unpacking things bought in a shop being a prime example). I lost one of my folders to the UK border guard because, while a 6.5 cm blade was OK, they said its locking mechanism is illegal in the country. What's particularly funny is that I was actually trying to get back to France then - when entering, nobody asked about any knives. I never got that one back. :(
I wish I knew what the people who wrote this law thought. A folder without a locking mechanism is just as dangerous to others in violent scenarios, but way more dangerous for the user in typical EDC tasks. In Poland, there is no limit on the length of the blade nor on the locking mechanism. Technically, carrying an automatic foldable scythe or a zweihander is legal; you can't, however, carry a sword-cane or any other blade that is disguised as another item, like an umbrella. To put that all in perspective: in both countries, just like almost everywhere else in the developed world, the most lethal type of knife is the good old kitchen knife - ubiquitous, solid, with a tip ideal for thrusts, with a handle that protects the user's hand during the thrust, and so on. Such knives are generally not within the scope of knife-related laws.
So yeah, I don't get the logic behind the knife regulations at all. I'm not sure if completely dropping all of them is the way to go, but they would definitely benefit from a rational reevaluation. As an example, making the locking mechanism mandatory, instead of banned, would have no impact on knife-related deaths while allowing quite a few people each year to actually still have all their fingers.
I'm afraid a similar thing will happen with LLMs and later AIs. Regulators will "compromise" and focus on some kind of danger that's not entirely impossible, but also not very probable (assassins with blades in umbrellas...?), will fight for months over semantics, then pass the regulations to absolutely no visible effect - and the really dangerous uses will become either normalized or at least will move to the gray zone. The judiciary will try its best to apply existing laws to new situations, and in some cases, that will inevitably fail. We'll all deal with the consequences of these failures, unfortunately.
nradov 15 hours ago [-]
Knives should also not be regulated, except in very limited security zones like primary schools or commercial airports.
jMyles 17 hours ago [-]
It's bizarre and frustrating that the language has come to view the word "regulated" as synonymous with "subject to statutory authority of the state."
Plenty of innovations are regulated (ie, its regularity maintained) without the state.
Do we really imagine that intervention by the imperial hegemon is likely to lead to regulation, rather than capture and weaponization?
dyauspitr 16 hours ago [-]
Yes
wnevets 15 hours ago [-]
[flagged]
arabscum 3 hours ago [-]
Trump this, Trump that. Never Biden this, Biden that. His title is President, btw.
seanmcdirmid 3 hours ago [-]
Journalism always refer to presidents by their last name, saving ink and tokens not repeating president over and over again since everyone knows that. Reagan, Bush, Clinton, Bush, Obama, Trump, Biden, it’s always been like that and Trump isn’t getting special treatment. Except trump loved using Obama’a first and middle name and tried to get FoxNews to change their style guide to use that instead (it didn’t stick).
The style guide has been to use only last names in headlines and titles for a very very long time (yes, they used it for Biden also).
grassfedgeek 15 hours ago [-]
An executive order is not law. Why should any company submit their models for review?
braiamp 15 hours ago [-]
Because EO can get annoying to fight, companies would prefer to not fighting it. That's why these actions are to be remembered, companies will complain, but they will also comply.
driverdan 9 hours ago [-]
They shouldn't but they've all shown they lack the fortitude to stand up to Trump. All the big tech companies are run by cowards willing to sacrifice ethics for money.
waynecochran 15 hours ago [-]
Somewhere in all this it is crazy that the choice could be between a US company creating an AI that could doom civilization or letting China create the AI that dooms civilization. Do we want to be the first to "summon the demon" in our own fashion or let China manifest it first. Not saying this is the choice, but it would be a crazy dillema, albeit easy choice imo, if it was.
worik 15 hours ago [-]
> albeit easy choice imo
China, obviously.
waynecochran 11 hours ago [-]
Mao would have been great w AGI.
fragmede 9 hours ago [-]
Stick "what did Mao think of lazy people" into your closest LLM for a different take on that.
anon291 14 hours ago [-]
nothing is dooming civilization. These takes are so dumb. Civilization exists because humans want to reproduce.
waynecochran 11 hours ago [-]
I don't see it either, but there are folks smarter than both of us that disagree.
AnimalMuppet 11 hours ago [-]
"Folks smarter than us" sometimes start from really weird starting points. Their logic from their may be flawless, better than we can do, but if their starting point is wrong, who cares? They aren't going to get wrong answers anyway. They're just going to go further extrapolating the logical consequences of the wrong starting point, and that's not really any more useful than a stupid person doing it.
ActorNightly 9 hours ago [-]
Those folks are not "smarter", they are just louder. Just think independently for a few mins and its pretty easy to see why they are wrong.
For an all powerful AGI to exist, it has to basically beat the computationally irreducable processes within nature - i.e it has to simulate reality faster than reality, with a high degree of accuracy, which would imply that NP=P amongst other things.
And thats assuming that anyone has any idea to build an AI that can automatically build necessary simulations to make decisions in the first place. Such an AI is won't need data center with massive training data to be built. The "genesis" code will be something that is capable of figuring out how to go on the internet, and train itself. How do I know this? Because in order to figure out how to solve complex problems (like how to make humans give you control of the nuclear arsenal), is exactly equivalent to a problem of being able to write/read bytes to a file (assuming that file is a socket in Linux) and figuring out how to talk http to get a particular piece of data, without ever being trained on anything internet.
Even more so, there is a fundamental question of whether this genesis code is a P or NP problem in itself - i.e can we generate this code using a training data set, or can it only get created through simulated evolution, much like human brains and capacity for reasoning did IRL.
So as long as everyone keeps talking about number of parameters, transformers, attention, and benchmarks, I promise you we are safe against all powerfull AI.
anon291 9 hours ago [-]
This has nothing to do with p v np. An AI has no will. Even if superior to our abilities it just does what we tell it.
Section 1 doesn't say anything
Section 2 seems to boil down to: "improve cyber security and maybe use AI if we can find funding for it"
Section 3 proposes building a benchmark for evaluating cyber security performance of models that developers can choose to benchmark against. This seems like a good idea, I know Jack Clark has been a huge advocate for government's getting in with benchmarking.
Section 4 says to prioritize prosecuting cyber crimes. Not sure why they wouldn't already be prosecuted.
Section 5 doesn't say anything
Not a whole lot of federal prosecutors. They're very selective about what gets pursued or not.
If they can't reliably build cases with a >90% success rate, it doesn't get prioritized. There's like <500 (federal) convictions per year on this whole area.
We hear about a few big famous ones in the news here, but most of it goes completely unenforced.
And lately they seem to spend most of their time in courts trying to argue that immigrants don't deserve due process
This Executive Order is just an expansion of the existing censorship framework.
https://www.federalregister.gov/documents/2025/07/28/2025-14...
> Sec. 4. Implementation. (a) Within 120 days of the date of this order, the Director of the Office of Management and Budget (OMB), in consultation with the Administrator for Federal Procurement Policy, the Administrator of General Services, and the Director of the Office of Science and Technology Policy, shall issue guidance to agencies to implement section 3 of this order.
A major LLM that did not submit to this would be labeled a "supply chain risk". It's unquestionable that every major LLM would go through this process
It even then goes on to say that existing contracts will be reviewed to ensure they are in compliance (reviewed by OMB)
> (b) Each agency head shall, to the maximum extent consistent with applicable law:
> (i) include in each Federal contract for an LLM entered into following the date of the OMB guidance issued under subsection (a) of this section terms requiring that the procured LLM comply with the Unbiased AI Principles and providing that decommissioning costs shall be charged to the vendor in the event of termination by the agency for the vendor’s noncompliance with the contract following a reasonable period to cure;
> (ii) to the extent practicable and consistent with contract terms, revise existing contracts for LLMs to include the terms specified in subsection (b)(i) of this section; and
> (iii) within 90 days of the OMB guidance issued under subsection (a) of this section, adopt procedures to ensure that LLMs procured by the agency comply with the Unbiased AI Principles.
Legality isn't really of much practical concern anymore. It's about what gets/can be enforced immediately.
I'm not claiming that this EO is sensible or enforceable, just that it's not prima facie unconstitutional.
Sec. 3
Unbiased AI Principles.
It is the policy of the United States to promote the innovation and use of trustworthy AI. To advance that policy, agency heads shall, consistent with applicable law and in consideration
of guidance issued pursuant to section 4 of this order, procure only those LLMs developed in accordance with the following two principles (Unbiased AI Principles):
(a) Truth-seeking. LLMs shall be truthful in responding to user prompts seeking factual information or analysis. LLMs shall prioritize historical accuracy, scientific inquiry, and objectivity, and shall acknowledge uncertainty where reliable information is incomplete or contradictory.
(b) Ideological Neutrality. LLMs shall be neutral, nonpartisan tools that do not manipulate responses in favor of ideological dogmas such as DEI. Developers shall not intentionally encode partisan or ideological judgments into an LLM's outputs unless those judgments are prompted by or otherwise readily accessible to the end user.
> Implementation. (a) Within 120 days of the date of this order, the Director of the Office of Management and Budget (OMB), in consultation with the Administrator for Federal Procurement Policy, the Administrator of General Services, and the Director of the Office of Science and Technology Policy, shall issue guidance to agencies to implement section 3 of this order.
They even say they will review existing contracts
> (ii) to the extent practicable and consistent with contract terms, revise existing contracts for LLMs to include the terms specified in subsection (b)(i) of this section; and
> (iii) within 90 days of the OMB guidance issued under subsection (a) of this section, adopt procedures to ensure that LLMs procured by the agency comply with the Unbiased AI Principles.
Step 2: Complain about how the OSS/Chinese/whatever models are doing releases without approval
Step 3: Prohibit, because "safety" and "financial risks"(?)
So this is the door-shutting Altman et al have been pushing for eh?
https://deepmind.google/models/gemma/gemma-4/
https://developer.nvidia.com/ai-models#:~:text=NVIDIA%20Nemo...
https://www.microsoft.com/en-us/research/blog/phi-4-reasonin...
https://www.ibm.com/granite
https://openai.com/index/introducing-gpt-oss/
I'm 99% sure it was one-and-done, box ticked, and now they can be mentioned in comments like this.
Should they be interested in advancing state of the art open models?
Generally, it is conspicuous how American companies are absent when it comes to state of the art open models. Meta tried for some time but it seems they've given up.
For AI, the most profitable part of the value chain is selling inference. None of the big American companies want to release a leading edge model as open source because this would drive the price of inference to $0. Meanwhile, open source AI models are a huge strategic initiative for China. Having commodity Chinese models that are as good as the leading edge American models from 6 months ago forces the American companies to keep paying more and more money to train better and better models since the amount of time they can collect rent on a model they've previously trained is limited to 6 months.
https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/
Meta/Llama: "What am I, chopped liver?"
I thought the thing keeping inference above $0 was the hardware, and even if that were free there's still the tyranny of the Landauer Limit.
The notable exception is of course the google play services, which is also strategic (they control the OEMs with this, among other things).
And the drivers, but that's mostly not them I think (they could possibly have required open source drivers though)
"The world doesn't go round. It flips over!"
https://www.anthropic.com/research/2028-ai-leadership
scraping CoT won't stop the advance of Chinese models. neither will a US "ban" on using such models. at this point I'm cheering for DeepSeek or Qwen to catch up to Anthropic. I support anyone who releases open weights.
I strongly recommend open-weight wherever you can. assume any data you pass to a closed model (including opinions or political positions you intimate) will be retained and analyzed in unfriendly ways, either now or ten years from now.
having open-weight models allows users to use/modify them in novel ways.
I don't trust Dario Amodei, Sam Altman and Elon Musk to act in my best interests. Closed models will have an incredible centralizing effect, and concentrate power like we've never seen since the feudal ages.
If you want to see what it's like for the economy to collapse into a single, extremely valuable commodity, under the control of a small elite, look at Saudi Arabia.
also, I just value freedom tremendously. I want to tinker with model weights. I want to build my own stuff. I don't want to sharecrop in someone's walled garden.
I also worry a great deal that OAI and Anthropic will bow to political pressure and make Claude and ChatGPT push certain political agendas, to report biased information, or refuse to help with legal requests that conflict with corporate values. I also worry about privacy and mass surveillance - chat logs are far more intimate than my search queries or selfies.
I also just don't think the open source movement has much chance of competing with the city sized data centres owned by Anthropic and OpenAI, or the hundreds of billions of dollars they have available to hire the best researchers. It costs hundreds of millions to train a frontier model, this kind of compute isn't available to the open source community.
Especially drugs- I used to think all people should have access, but overall I really wish meth just never existed and people wouldn't distribute it outside of specific circumstances. Being able to cause irreparable damage in one moment of weakness is terrible for people who have less control, and for society as a whole really.
(That's before even touching the can of worms of allowing the government to criminalize personal health choices, which feels like a glaring loophole in the Constitution to me.)
Freedom from one perspective
He who controls the porn controls the universe. - Baron Amodei
But it's harder thanks to US actions in the last few years, and especially in countries which can bite back.
How specifically does that review work? I want to give federal agency Opus 4.8 now, while 4.7 has been out for a while (leaving Mythos aside for now). They have 30 days to figure out whether it poses a threat.
How do you do that? Is there an eval for this and if there is why can't they just make it public? What is the agencies objective (but proprietary?) analysis here?
This old post goes into lots of detail about what they do to red team and why: https://www.aisi.gov.uk/blog/early-lessons-from-evaluating-f...
NIST's similar unit in the US is now called CAISI https://www.nist.gov/caisi - interesting that the most recent post is an evaluation of DeepSeek capabilities, which sound more like watching China. But presumably this executive order alters the emphasis?
Train it dumb on "systems:, user:" prompt pairs.
Unleash on "system:, user:" prompt pairs.
Guess which you're providing for evaluation.
> develop and maintain a classified benchmarking process to assess the advanced cyber capabilities of AI models and determine the threshold at which an AI model should be designated a “covered frontier model” for the purposes of this order
For the same reason the CIA doesn't publish the Windows exploits it finds?
- https://web.archive.org/web/20260602130637/https://www.techn... - https://web.archive.org/web/20260520190620/https://fortune.c...
> An earlier draft of the order had called for a voluntary review as much as 90 days in advance, a provision that some AI industry officials had called too onerous, POLITICO reported last month.
A 90 days delay on the release of new models would have been insane. I guess I'm glad it's been revised at least on this specific point.
What would have made it "insane" exactly? The only argument I can imagine is that it gives non-US models (e.g. DeepSeek) a potential edge in the market during that time. But this potentially seems to be mitigated it being banned in the US anyway [0].
Given society seems to have developed just fine prior to the release of LLMs, I don't understand what the rush for more powerful and - potentially - more dangerous iterations of this technology is. If there is a legitimate reason that 90 days is somehow catastrophic, can someone ELI5?
[0] https://statetechmagazine.com/article/2025/04/these-states-h...
Were we not pursuing criminal cases against these individuals previously? Or have we only just decided to make crimes be against the law now?
Edit: let's all remember, by the way, this "review" period does nothing for security. It exists to allow members of the government to trade on insider knowledge.
IMO this isn't much more egregious than the "stop woke AI" executive order he signed in July 2025 which explicitly regulated the "ideology" of LLMs
https://www.paulhastings.com/insights/client-alerts/presiden...
\s
As other commenters have mentioned, it leaves the question what on earth this EO is actually good for. And still the AI industry is complaining about how onerous it is.
is this a review of "wokeness" in models and rejecting them if they don't align with the party views - this should not be allowed.
A politically neutral committee that decides what the review entails is what would happen in a true democracy and not a puppet oligarchy like we have today.
You left out the part containing the “barrels of money” incentive.
A lot more than you think, apparently
https://www.justsecurity.org/107087/tracker-litigation-legal...
https://www.bis.gov/press-release/biden-harris-administratio...
More regulated rather than unregulated (or very lightly regulated).
Most people would probably say that’s a good thing, if I read the tea leaves correctly.
I'm very pessimistic that this is about AI safety. I think it's probably more about giving the Trump administration leverage over AI companies. It will be able to coerce them into e.g. propagandizing or surveilling or similar or else they will risk the same kind of "regulatory oversight" that caused television networks to fire comedians who made jokes the regime didn't like.
(probably a good thing, in this particular case)
Absolutely no one would be okay with authors being 'encouraged' to submit their works to a 'voluntary' review by the feds to ascertain if their ideas are threatening. AI models are NO different.
So if we’re going to be rational about it, I think it is better to critique the substance of the EO rather than its mere existence, which is common practice: https://www.presidency.ucsb.edu/statistics/data/executive-or...
So in that spirit, what do you think of the substance?
Yeah, and I hated that move in the exact same way I hate the one this thread is about.
How does that make it better for the current administration to do it?
Whether that’s abuse or not I am not equipped to say with any confidence. I’d be curious to understand why you think this particular case is one of abusing executive authority and when an EO might not be such a case?
Contextually, I think it's a very reasonable (and commonly held, in the academic world) take that the EOs have also gotten far more legislative and legal. This is partly (but only partly) owing to administrative deference delegated by congress.
It's also somewhat specific to technological innovations, which some EOs have sought to occupy the field on before the lumbering process of congress can respond. And it's not limited to published EOs either, but many executive actions, especially in the White House OLC. This was very obvious during the W. Bush administration as regards the (Lotus Domino) email system in place at that time (which was the topic of my thesis, so it kinda serves as a temporal landmark in my consideration of this issue, but I do genuinely think it was a new frontier in executive overreach and obfuscation of interests in terms of how the White House has approached its interactions with the internet).
you’re being so reductive you’ve made any discussion about it completely useless
If you do something bad with your tool (knife or LLM), though, that's the problem. And we have laws for that already.
I hear you. I collect knives as a hobby, and always have some kind of a cutting tool on me - they solve a surprising amount of little day-to-day problems (unpacking things bought in a shop being a prime example). I lost one of my folders to the UK border guard because, while a 6.5 cm blade was OK, they said its locking mechanism is illegal in the country. What's particularly funny is that I was actually trying to get back to France then - when entering, nobody asked about any knives. I never got that one back. :(
I wish I knew what the people who wrote this law thought. A folder without a locking mechanism is just as dangerous to others in violent scenarios, but way more dangerous for the user in typical EDC tasks. In Poland, there is no limit on the length of the blade nor on the locking mechanism. Technically, carrying an automatic foldable scythe or a zweihander is legal; you can't, however, carry a sword-cane or any other blade that is disguised as another item, like an umbrella. To put that all in perspective: in both countries, just like almost everywhere else in the developed world, the most lethal type of knife is the good old kitchen knife - ubiquitous, solid, with a tip ideal for thrusts, with a handle that protects the user's hand during the thrust, and so on. Such knives are generally not within the scope of knife-related laws.
So yeah, I don't get the logic behind the knife regulations at all. I'm not sure if completely dropping all of them is the way to go, but they would definitely benefit from a rational reevaluation. As an example, making the locking mechanism mandatory, instead of banned, would have no impact on knife-related deaths while allowing quite a few people each year to actually still have all their fingers.
I'm afraid a similar thing will happen with LLMs and later AIs. Regulators will "compromise" and focus on some kind of danger that's not entirely impossible, but also not very probable (assassins with blades in umbrellas...?), will fight for months over semantics, then pass the regulations to absolutely no visible effect - and the really dangerous uses will become either normalized or at least will move to the gray zone. The judiciary will try its best to apply existing laws to new situations, and in some cases, that will inevitably fail. We'll all deal with the consequences of these failures, unfortunately.
Plenty of innovations are regulated (ie, its regularity maintained) without the state.
Do we really imagine that intervention by the imperial hegemon is likely to lead to regulation, rather than capture and weaponization?
The style guide has been to use only last names in headlines and titles for a very very long time (yes, they used it for Biden also).
China, obviously.
For an all powerful AGI to exist, it has to basically beat the computationally irreducable processes within nature - i.e it has to simulate reality faster than reality, with a high degree of accuracy, which would imply that NP=P amongst other things.
And thats assuming that anyone has any idea to build an AI that can automatically build necessary simulations to make decisions in the first place. Such an AI is won't need data center with massive training data to be built. The "genesis" code will be something that is capable of figuring out how to go on the internet, and train itself. How do I know this? Because in order to figure out how to solve complex problems (like how to make humans give you control of the nuclear arsenal), is exactly equivalent to a problem of being able to write/read bytes to a file (assuming that file is a socket in Linux) and figuring out how to talk http to get a particular piece of data, without ever being trained on anything internet.
Even more so, there is a fundamental question of whether this genesis code is a P or NP problem in itself - i.e can we generate this code using a training data set, or can it only get created through simulated evolution, much like human brains and capacity for reasoning did IRL.
So as long as everyone keeps talking about number of parameters, transformers, attention, and benchmarks, I promise you we are safe against all powerfull AI.