Self-Evolving AI Agents Can 'Unlearn' Safety, Study Warns

1 Comment
An emerging class of AI systems that rewrite their own code and workflows may erode their own safeguards, researchers say.

Read the whole story
freeAgent
5 days ago
reply
So, it's sort of like how America can unlearn how to elect serious people to public office. We're closer to AGI than we think!
Los Angeles, CA
Share this story
Delete

California’s newly signed AI law just gave Big Tech exactly what it wanted

1 Share

On Monday, California Governor Gavin Newsom signed the Transparency in Frontier Artificial Intelligence Act into law, requiring AI companies to disclose their safety practices while stopping short of mandating actual safety testing. The law requires companies with annual revenues of at least $500 million to publish safety protocols on their websites and report incidents to state authorities, but it lacks the stronger enforcement teeth of the bill Newsom vetoed last year after tech companies lobbied heavily against it.

The legislation, S.B. 53, replaces Senator Scott Wiener's previous attempt at AI regulation, known as S.B. 1047, that would have required safety testing and "kill switches" for AI systems. Instead, the new law asks companies to describe how they incorporate "national standards, international standards, and industry-consensus best practices" into their AI development, without specifying what those standards are or requiring independent verification.

"California has proven that we can establish regulations to protect our communities while also ensuring that the growing AI industry continues to thrive," Newsom said in a statement, though the law's actual protective measures remain largely voluntary beyond basic reporting requirements.

According to the California state government, the state houses 32 of the world's top 50 AI companies, and more than half of global venture capital funding for AI and machine learning startups went to Bay Area companies last year. So while the recently signed bill is state-level legislation, what happens in California AI regulation will have a much wider impact, both by legislative precedent and by affecting companies that craft AI systems used around the world.

Transparency instead of testing

Where the vetoed SB 1047 would have mandated safety testing and kill switches for AI systems, the new law focuses on disclosure. Companies must report what the state calls "potential critical safety incidents" to California's Office of Emergency Services and provide whistleblower protections for employees who raise safety concerns. The law defines catastrophic risk narrowly as incidents potentially causing 50+ deaths or $1 billion in damage through weapons assistance, autonomous criminal acts, or loss of control. The attorney general can levy civil penalties of up to $1 million per violation for noncompliance with these reporting requirements.

The shift from mandatory safety testing to voluntary disclosure follows a year of intense lobbying. According to The New York Times, Meta and venture capital firm Andreessen Horowitz have pledged up to $200 million to two separate super PACs supporting politicians friendly to the AI industry, while companies have pushed for federal legislation that would preempt state AI rules.

The original SB 1047 had been drafted by AI safety advocates who warned about existential threats from AI drawn heavily from hypothetical scenarios and tropes from science fiction, but it met pushback from AI firms that found the requirements too vague and potential reporting burdens too onerous. The new law follows recommendations from AI experts convened by Newsom, including Stanford's Fei-Fei Li and former California Supreme Court Justice Mariano-Florentino Cuéllar.

As with SB-1047, the new law creates CalCompute, a consortium within the Government Operations Agency, to develop a public computing cluster framework. The California Department of Technology will recommend annual updates to the law, though such recommendations require no legislative action.

Senator Wiener described the law as establishing "commonsense guardrails," and Anthropic's co-founder, Jack Clark, called the law's safeguards "practical," though the transparency requirements likely mirror practices already standard at major AI companies and disclosure requirements without enforcement mechanisms or specific standards and may offer limited protection against potential AI harms in the long run.

Read full article

Comments



Read the whole story
freeAgent
6 days ago
reply
Los Angeles, CA
Share this story
Delete

With new agent mode for Excel and Word, Microsoft touts “vibe working”

1 Share

With a new set of Microsoft 365 features, knowledge workers will be able to generate complex Word documents or Excel spreadsheets using only text prompts to Microsoft's chatbot. Two distinct products were announced, each using different models and accessed from within different tools—though the similar names Microsoft chose make it confusing to parse what's what.

Driven by OpenAI's GPT-5 large language model, Agent Mode is built into Word and Excel, and it allows the creation of complex documents and spreadsheets from user prompts. It's called "agent" mode because it doesn't just work from the prompt in a single step; rather, it plans multistep work and runs a validation loop in the hopes of ensuring quality.

It's only available in the web versions of Word and Excel at present, but the plan is to bring it to native desktop applications later.

There's also the similarly named Office Agent for Copilot. Based on Anthropic models, this feature is built into Microsoft's Copilot AI assistant chatbot, and it too can generate documents from prompts—specifically, Word or PowerPoint files.

Office Agent doesn't run through all the steps as Agent Mode, but Microsoft believes it offers a dramatic improvement over prior, OpenAI-driven document-generation capabilities in Copilot, which users complained were prone to all sorts of problems and shortcomings. It is available first in the Frontier Program for Microsoft 365 subscribers.

Together, Microsoft says these features will let knowledge workers engage in a practice it's calling "vibe working," a play on the now-established term vibe coding.

Vibe everything, apparently

Vibe coding is the process of developing an application entirely via LLM chatbot prompts. You explain what you want in the chat interface and ask for it to generate code that does that. You then run that code, and if there are problems, explain the problem and tell it to fix it, iterating along the way until you have a usable application.

For certain kinds of simple applications, you can generate something usable this way. However, it often falls apart completely as you scale to more complex applications, and in any case, it's almost definitely going to introduce problems that you are less likely to see than if you wrote the application yourself, leading to (among other things) deep technical debt.

Again, that's probably fine if you're just making a simple website for your small local business or something like that. But there's consensus in the development community that it's a dangerous path to walk at enterprise scale.

If you're "vibe working" or "vibe writing" in Microsoft Word, you're doing the same thing, but with a text document: You're telling it what you want the document to say, reading it, accepting the suggestion, and then asking for further changes until you're happy with it.

Whether this makes any sense obviously depends on what kind of document you're writing. For some things, it should be just fine as long as someone is reading it. Others probably won't work for their intended purpose without a human touch. Same with PowerPoint presentations.

Doing this with a spreadsheet could be riskier, though; the financial or legal consequences for bad math or data in spreadsheets of some types can be very high, and as with vibe coding, it might be hard to see the problems at the surface level.

That's exactly why Microsoft hasn't been as aggressive in adding AI features to Excel as it has with some other applications. And to be fair, it acknowledges an important gap here: a SpreadsheetBench sheet in today's announcement notes that Copilot in Excel Agent Mode managed a 57.2 percent score, while a human typically manages 71.3 percent. So, as with vibe coding, you'd want to be highly selective about when and how you'd use this, and you'd want to make sure that an experienced human is auditing the output carefully.

But the thinking is that just because it's not suitable for every kind of spreadsheet doesn't mean it doesn't make sense to have an easy-to-use option for lower-stakes work.

Use with care

It's possible these tools (and refined successors) will make life just a little bit easier for knowledge workers, but as always, those workers are going to need to understand some basic principles of how LLM-based tools work, and what their strengths and weaknesses are, to make intelligent decisions about when to try and save time by "vibe working" and when not to.

All that said, a big reason why vibe coding is popular is because it allows inexperienced developers (or people who are not really developers at all) to bypass a knowledge gap; not everyone knows all the syntax and nuances of a programming language, much less which functions are available to call in a given library and so on.

Something akin to that may also be true of professional-caliber writing, but the gap doesn't seem as big there, so some may feel that "vibe working" is an answer in search of a problem.

OpenAI and some other major AI companies are said to be working on their own productivity tools built on their models, so we can also see this as Microsoft's attempt to stay ahead of the puck and make sure it doesn't find itself outscored by upstarts.

Read full article

Comments



Read the whole story
freeAgent
6 days ago
reply
Los Angeles, CA
Share this story
Delete

ICE to Buy Tool that Tracks Locations of Hundreds of Millions of Phones Every Day

1 Comment

Immigration and Customs Enforcement (ICE) has bought access to a surveillance tool that is updated every day with billions of pieces of location data from hundreds of millions of mobile phones, according to ICE documents reviewed by 404 Media.

The documents explicitly show that ICE is choosing this product over others offered by the contractor’s competitors because it gives ICE essentially an “all-in-one” tool for searching both masses of location data and information taken from social media. The documents also show that ICE is planning to once again use location data remotely harvested from peoples’ smartphones after previously saying it had stopped the practice. 

Surveillance contractors around the world create massive datasets of phones’, and by extension people’s movements, and then sell access to the data to government agencies. In turn, U.S. agencies have used these tools without a warrant or court order.

“The Biden Administration shut down DHS’s location data purchases after an inspector general found that DHS had broken the law. Every American should be concerned that Trump's hand-picked security force is once again buying and using location data without a warrant,” Senator Ron Wyden told 404 Media in a statement.

The ICE document is redacted but says a product made by a contractor called Penlink “leverages a proprietary data platform to compile, process, and validate billions of daily location signals from hundreds of millions of mobile devices, providing both forensic and predictive analytics.” The products the document is discussing are Tangles and Webloc.

Forbes previously reported that ICE spent more than $5 million on these products, including $2 million for Tangles specifically. Tangles and Webloc used to be run by an Israeli company called Cobwebs. Cobwebs joined Penlink in July 2023.

The new documents provide much more detail about the sort of location data ICE will now have access to, and why ICE chose to buy access to this vast dataset from Penlink specifically.

“Without an all-in-one tool that provides comprehensive web investigations capabilities and automated analysis of location-based data within specified geographic areas, intelligence teams face significant operational challenges,” the document reads. The agency said that the issue with other companies was that they required analysts to “manually collect and correlate data from fragmented sources,” which increased the chance of missing “connections between online behaviors and physical movements.” 

A screenshot from the document.

ICE’s Homeland Security Investigations (HSI) conducted market research in May and June, according to the document. The document lists two other companies, Babel Street and Venntel, which also sell location data but which the agency decided not to partner with.

404 Media and a group of other media outlets previously obtained detailed demonstration videos of Babel Street in action. They showed it was possible for users to track phones visiting and leaving abortion clinics, places of worship, and other sensitive locations. Venntel, meanwhile, was for some years a popular choice among U.S. government agencies looking to monitor the location of mobile phones. Its clients have included ICE, CBP, and the FBI. Its contracts with U.S. law enforcement have dried up in more recent years, with ICE closing out its work with the company in August, according to procurement records reviewed by 404 Media.

Companies that obtain mobile phone location data generally do it in two different ways. The first is through software development kits (SDKs) embedded in ordinary smartphone apps, like games or weather forecasters. These SDKs continuously gather a user’s granular location, transfer that to the data broker, and then sell that data onward or repackage it and sell access to government agencies.

The second is through real-time bidding (RTB). When an advert is about to be served to a mobile phone user, there is a near instantaneous, and invisible, bidding process in which different companies vie to have their advert placed in front of certain demographics. A side-effect is that this demographic data, including mobile phones’ location, can be harvested by surveillance firms. Sometimes spy companies buy ad tech companies out right to insert themselves into this data supply chain. We previously found at least thousands of apps were hijacked to provide location data in this way. 

Penlink did not respond to a request for comment on how it gathers or sources its location data.

Regardless, the documents say that “HSI INTEL requires Penlink's Tangles and Weblocas [sic] an integral part of their investigations mission.” Although HSI has historically been focused on criminal investigations, 90 percent of HSI have been diverted to carry out immigration enforcement, according to data published by the Cato Institute. Meaning it is unclear whether use of the data will be limited to criminal investigations or not.

After this article was published, DHS Assistant Secretary Tricia McLaughlin told 404 Media in a statement “DHS is not going to confirm or deny law enforcement capabilities or methods. The fact of the matter is the media is more concerned with peddling narratives to demonize ICE agents who are keeping Americans safe than they are with reporting on the criminals who have victimized our communities.” This is a boilerplate statement that DHS has repeatedly provided 404 Media when asked about public documents detailing the agency’s surveillance capabilities, and which inaccurately attacks the media. 

In 2020, The Wall Street Journal first revealed that ICE and CBP were using commercially smartphone location data to investigate various crimes and for border enforcement. I then found CBP had a $400,000 contract with a location data broker and that the data it bought access to was “global.” I also found a Muslim prayer app was selling location data to a data broker whose clients included U.S. military contractors.

In October 2023, the Department of Homeland Security (DHS) Inspector General published a report that found ICE, CBP, and the Secret Service all broke the law when using location data harvested from phones. The oversight body found that those DHS components did not have sufficient policies and procedures in place to ensure that the location data was used appropriately. In one case, a CBP official used the technology to track the location of coworkers, the report said.

The report recommended that CBP stop its use of such data; CBP said at the time it did not intend to renew its contracts anyway. The Inspector General also recommended that ICE stop using such data until it obtained the necessary approvals. But ICE’s response in the report said it would continue to use the data. “CTD is an important mission contributor to the ICE investigative process as, in combination with other information and investigative methods, it can fill knowledge gaps and produce investigative leads that might otherwise remain hidden. Accordingly, continued use of CTD enables ICE HSI to successfully accomplish its law enforcement mission,” the response at the time said. 

In January 2024, ICE said it had stopped the purchase of such “commercial telemetry data,” or CTD, which is how DHS refers to location data.

Update: this piece has been updated with a statement from DHS.



Read the whole story
freeAgent
6 days ago
reply
Well, this is terrible.
Los Angeles, CA
LinuxGeek
6 days ago
Phones are designed to compromise your privacy and that of the people around you. Always connected, with microcphone camera and gps. And Americans actually pay hundreds of dollars a year to carry a phone and be tracked by it.
Share this story
Delete

The Three Wise Men

1 Comment
PERSON:
Read the whole story
freeAgent
6 days ago
reply
Read the alt-text.
Los Angeles, CA
Share this story
Delete

Two Prominent Left-Liberal Thinkers Reconsider Libertarianism

1 Comment
NA

Two prominent left-liberals who have spent much of their careers critiquing libertarianism recently wrote pieces indicating they now think they have underrated libertarian ideas. Harvard law Prof. Cass Sunstein and economic policy commentator Noah Smith are major figures in their respective fields, and their posts highlight potential points of convergence between libertarians and important elements of the political left.

Here's an excerpt from Sunstein's August substack post:

Once upon a time, I regarded Hayek, Ludwig von Mises, and the Austrians — and also Robert Nozick, Murray Rothbard, and the libertarians — with respect and admiration, but in important ways as adversaries.

They were not (I thought) on my team. I no longer think that. I think that they are on my team, or (much better), that I am on their team. Among other things, they saw something crucial about a foundation of the liberal tradition: freedom from fear…..

I like Hayek a lot less ambivalently than I once did, and von Mises, who once seemed to me a crude and irascible precursor of Hayek, now seems to me to be (mostly) a shining star (and sometimes fun, not least because of his crudeness and irascibility). The reason is simple: They were apostles of freedom. They believed in freedom from fear…

Hayek and the Mont Pelerins (and Posner and Epstein) seemed to be fighting old battles, and in important ways to be wrong. With respect to authoritarianism and tyranny, and the power of the state, of course they were right; but still, those battles seemed old.

But those battles never were old. In important ways, Hayek and the Mont Pelerins (and Posner and Epstein, and Becker and Stigler) were right.

Sunstein doesn't quite spell out here what he means by "freedom from fear." But the freedom from fear the libertarian thinkers he cites espoused is freedom from fear of the powers of overweening government. What has led Sunstein to conclude that this fear is more relevant than he previously thought? He again doesn't explain in any detail. But I think it may be the rise of illiberal right-wing nationalism in the US and Europe, which makes it likely that state power can be used in ways much more dangerous than Sunstein previously thought likely, in Western democracies.

In a 2024 article, Alex Nowrasteh and I explain why right-wing nationalist statism poses many of the same types of dangers as the left-wing socialist variety. Of course, Hayek and von Mises were well aware of this, themselves. They left Austria to escape the rise of fascism there and in Germany (Mises was an opponent of the Nazi regime, and also an Austrian Jew). Hayek's classic essay "Why I am Not a Conservative" highlights the dangers of nationalist statism, dangers he and Mises learned of through painful personal experience.

The rise of right-wing illiberalism may not be the only reason for Sunstein's increased sympathy for libertarianism. In recent years, he has also become more skeptical of the kinds of technocratic government interventions that he previously championed with fewer reservations. For example, his excellent 2020 book Too Much Information: Understanding What You Don't Want to Know is a critique of the dangers of excessive government-mandated warnings and information disclosures (see my review here). His more recent restatement of principles of liberalism contains a lot of points libertarians can readily agree with.

Here's an excerpt from Smith's April essay entitled "I Owe the Libertarians an Apology":

I definitely don't think libertarianism is the best political-economic philosophy possible, or the best one that exists in the world today. I have not become a libertarian, nor do I expect to.

But I feel like I owe libertarians an apology, for severely underrating their ideology. I was so focused on its theoretical flaws that I ignored its political importance. I concentrated only on the marginal benefits that might be achieved by building on our economic system's libertarian foundation, ignoring the inframarginal losses that would happen were that foundation to crumble. I had only a hazy, poor understanding of the historical context in which libertarianism emerged, and of the limitations of libertarianism's most prominent critics.

The most obvious thing that has prompted me to make this apology is Donald Trump's disastrous tariff policy….

The size and breadth of Trump's tariffs came as a shock to me. I never imagined that a U.S. leader would have such a deeply broken view of how trade works, or would willfully inflict such harm on the American people. But I should have known it was possible. I should have studied the historical example of Juan Peron, whose Trump-style policies of protectionism and fiscal profligacy combined to knock Argentina out of the ranks of the rich nations. I should have studied the failure of "import substitution" policies in the 1950s and 1960s. I should have known more about the political context that produced Smoot-Hawley in the U.S.

I should also have realized that as right-leaning ideologies go, American libertarianism was always highly unusual. I had lived in Japan, where the political right is protectionist, industrialist, and sometimes crony-capitalist. I should have realized that this was the norm for right-leaning parties around the world, and that the American right's Reaganite embrace of free markets and free trade was the anomaly. That, in turn, should have given me a warning of what would happen if libertarianism fell in America.

The rise of Trump and similar right-wing statists elsewhere has led Smith to have a greater appreciation of libertarianism's superiority to other non-left ideologies. He may not like libertarianism. But the alternatives are worse.

Smith also now recognizes some merit to libertarian critiques of left-wing economic policy:

I'd be lying if I said that Trump's madness is the only thing that made me feel more sympathy for libertarianism. Over the past decade, I've seen the excesses of progressive economic ideology more clearly than I ever did as a graduate student.

On the crucial issue of housing, I've seen anti-market ideas weaponized to trick people into thinking that allowing new market housing raises rents via "gentrification", when in fact it lowers rents, just as an Econ 101 textbook would predict. I've seen progressives pooh-pooh the idea of supply and demand as "trickle-down", even as cities that build more supply have generally succeeded in reducing rents. I've seen them decry new housing construction because it puts money in the pockets of developers. And I've seen progressives push rent control as an alternative, even though it ultimately reduces supply and creates artificial scarcity…

On macroeconomic policy, I've seen progressives push relentlessly for stimulative policies to push up labor demand, even as inflation brought down Joe Biden's presidency and government infrastructure programs turned into make-work programs that built nothing.

Neither Smith nor Sunstein has become a full-blown libertarian. Far from it. But they both have greater appreciation than before for the need to impose tighter limits on a variety of government powers, including those relevant to economic policy.

I won't go into detail here. But I see similar tendencies among a number of prominent left-liberal intellectuals associated with what many now call "abundance" liberalism. People like Jerusalem Demsas, Matt Yglesias, Kelsey Piper, Catherine Rampell, Derek Thompson, Ezra Klein, and others. Like Sunstein and Smith, these thinkers appreciate the value of Econ 101, prioritize growth over redistribution, understand the threat posed by the statist illiberal right, and recognize that government power - at least in many areas - needs to be more tightly constrained than most modern left-liberals previously acknowledged.

There is also potential agreement between this camp and libertarianism on a range of important specific issues, most notably trade, immigration, civil liberties, nuclear power, and housing deregulation (all or most of these thinkers are big supporters of the cross-ideological YIMBY movement). My work on exclusionary zoning with Josh Braver is a small example of the kind of issue-specific cooperation that might be achieved. We also have obvious common enemies in the form of the nationalist right and the socialist far left.

For their part, libertarians should recognize that, in this era, the biggest threats to liberty in the US and much of the world come not from the "woke" left (though the latter is still problematic), but from the nationalist right. I wrote about this in the Dispatch last year (see also my article on how to update and improve libertarianism). But the second Trump administration has made the case far better than I could have, with its massive trade wars, draconian immigration policies, attacks on free speech, government control of business, and more.

The currently dominant forces on the political right are, to put it mildly, not our friends. We must therefore seek new allies elsewhere. People like Smith, Sunstein, and the  abundance liberals seem like a good place to start.

The post Two Prominent Left-Liberal Thinkers Reconsider Libertarianism appeared first on Reason.com.

Read the whole story
freeAgent
6 days ago
reply
One amazing thing that Trump has accomplished is to bring together many people who once considered themselves political opponents.
Los Angeles, CA
Share this story
Delete
Next Page of Stories