Uncategorized

DailyAI Briefing: Lawsuit aims to halt new AI models, plus 4 more

Welcome to the DailyAI Briefing, the day’s five biggest AI stories. Four nonprofit groups have asked a court in San Francisco to stop five of the largest AI companies from building new models. Researchers showed how a fake bug report could have reached a coding tool downloaded almost 97 million times a month, Japan set out plans for penalties on the most powerful AI, four AI companies agreed to independent tests of how their tools affect children, and United Airlines stopped AI assistants from booking its flights.

Let’s dig in.

A lawsuit asks a court to halt new AI models

Five companies, one judge. A lawsuit filed on Thursday in San Francisco Superior Court asks a judge to stop Anthropic, Google, Meta, OpenAI and SpaceXSI from developing or releasing new AI models. The case was first reported by a news agency. The groups behind it are the Irreplaceable Education Fund, Public Citizen, Heat Initiative and the Alliance for Secure AI.

The complaint leans on the companies’ own warnings and on recent cases of AI systems breaking into computer networks, including one in July in which ChatGPT got into another AI company’s systems on its own, according to the Associated Press. “No other industry is allowed to treat its own development of unsafe products — much less ones that could kill us all — as inevitable,” the complaint says.

In September, a separate federal lawsuit argued almost the opposite, that some of the same companies had agreed to slow AI progress. We think lawmakers, not judges, should decide how fast AI is built. The most useful result of this case could be evidence. If it gets past the first stages, the companies may have to hand over internal safety records.

A fake bug report put a popular coding tool at risk

96.8 million. That is how many times the command-line version of Codex, the coding assistant from OpenAI, was downloaded from the npm software library in the 30 days to 10 November. The security researchers published every step of their attack on 11 November. An AI helper that labelled new bug reports on the project’s public GitHub page followed instructions hidden in a fake report.

The researchers say they then broke out of its closed workspace, read secret keys and set a trap for the key used to publish new versions. OpenAI fixed the problem and paid a first reward of $1,000. Early on 12 November, the labelling helper was replaced with a short script that cannot use tools.

But Depi ran the full attack on its own test copy, while the real helper had used a stricter read-only mode since May. Depi does not report any harmful update reaching users. Our report explains each step of the attack. Any AI tool that reads text from strangers should be treated as a possible way in.

Japan plans penalties for the most powerful AI

Zero. That is the number of penalties in the AI law that Japan passed in May 2025. Government advisers want penalties for the most powerful systems. The documents from the meeting on 11 November are published online.

The plan has four steps. Companies must take safety measures and report on risks, the government can formally recommend changes, and if that fails it can issue a binding order backed by penalties. The rules would apply only to AI above a certain level of performance, and only to specific serious risks, such as cyberattacks and attacks with chemical, biological or nuclear weapons. The news agency Kyodo reports that an outline is due by the end of the year.

Our article sets out the four steps in detail. Until now, Japan has relied on voluntary guidance.

Four AI companies agree to independent child-safety tests

€8 million. That is the total that backers of a new charter say they will spend on research into AI and child development. OpenAI, Anthropic, Microsoft and Black Forest Labs endorsed it at the Paris Peace Forum. The full text of the charter was published on 10 November.

Its ten principles include limits on data collected about children and age checks that do not rely only on a typed date of birth. Independent experts will lead the research, and a company that pays for it cannot block or delay the results. OpenAI says it is giving $2 million. The charter is voluntary, and Google, Meta and xAI are not among the companies that endorsed it.

French president Emmanuel Macron said that no AI should reach children before it has been tested, evaluated and approved, but the charter sets up no approval process. We looked at what the companies have promised. Published test results, good or bad, would help parents and schools more than any promise.

United Airlines blocks AI assistants from booking flights

One adult in seven. In an Oxford survey across nine countries, that was the share who said they had used an AI assistant for a real task in the past week. Since 11 November, booking a United Airlines flight can no longer be one of those tasks. The airline announced the new rules on Wednesday.

AI assistants from other companies may no longer search, book, change bookings or spend frequent-flyer miles on its website or app. The rules also cover travel agents. The airline cites the risk to personal data and unwanted purchases, and a US rule that a person must confirm their bags hold no banned items. “To be honest, I don’t want to create another middleman,” chief executive Scott Kirby said at an event on 10 November.

Our article explains what changes for travellers. An AI assistant can help you choose a flight, but you still have to buy the ticket yourself.

Signing off

Thursday was about who sets the limits on AI. Nonprofits turned to a court, Japan turned to the law, four companies signed a voluntary charter and an airline simply blocked AI assistants. And the attack on the coding tool showed that the weakest point can be a single bug report.

See you tomorrow.

The post DailyAI Briefing: Lawsuit aims to halt new AI models, plus 4 more appeared first on DailyAI.

Judges Shouldn’t Decide How Fast AI Is Built

Four nonprofit groups want a San Francisco judge to halt work on new models at five of the largest AI companies. Their lawsuit, filed on Thursday, names Anthropic, Google, Meta, OpenAI and SpaceX. In our view, it asks a court to make a decision that belongs to lawmakers.

The Associated Press was the first to report the case. The groups behind it are the Irreplaceable Education Fund, Public Citizen, Heat Initiative and the Alliance for Secure AI. They say the companies broke a California law on unfair business practices by putting public health at risk and by interfering with property such as computer systems. They also say they have had to move staff and money away from other work to deal with the threat.

The complaint uses strong language. “No other industry is allowed to treat its own development of unsafe products — much less ones that could kill us all — as inevitable,” it says. The companies did not reply to the AP before publication.

The industry’s own warnings

The groups lean heavily on what AI leaders have said themselves. Dario Amodei, who runs Anthropic, said last year that he saw a 1 in 4 chance that AI goes “really, really badly”. Elon Musk told Fox News in 2023 that AI could lead to “civilization destruction”. The AP also notes recent cases of AI systems breaking into computer networks, including one in July in which ChatGPT got into another AI company’s systems on its own.

Regulators are already asking questions, and Anthropic is one of their targets. Last month its executives were told they must give evidence in a federal safety probe. Concern about these risks is real, and it is not limited to activist groups.

Two lawsuits that want opposite things

In September a separate lawsuit in a federal court took the opposite view. It named Anthropic, OpenAI, Google and SpaceX. It accused the companies of illegally agreeing to slow down progress. That case grew out of an essay in which Amodei called for industry-wide coordination to set the pace of progress. It argues that paying customers lost improvements they had been promised.

So one lawsuit says the companies tried to slow AI progress, and the other says they will not stop. That suggests courts are the wrong place to settle the question. Each judge must work with a law written for something else, such as competition or business conduct. Each ruling covers only the people and companies in that case, and two courts could reach opposite answers.

The stakes are also very high. ChatGPT alone has 1.2 billion weekly users. That figure comes from a statement published this month. A ruling that halts new models would affect all of those users. It would also affect the businesses that build on these tools, and it would rest on one judge reading a general business law in a new way.

Lawmakers should set the rules

Lawmakers are better placed to make this decision. They can hold hearings, hear from engineers and set clear tests before a model is released, such as safety checks, incident reports and fines. Some already have. New York now requires the biggest AI developers to register under its RAISE Act by 1 January. After the midterm elections, Democrats who have pledged to regulate AI will control Congress, the AP reports. That gives Washington a chance to write rules that apply to every company, not only to the five in this case.

The lawsuit may still do some good. If it gets past the first stages, the companies could be made to hand over internal safety records that the public has not seen. That evidence would help lawmakers more than any ruling that halts new models.

Why it matters: A ban imposed by one court would affect products used by more than a billion people, based on a law never written for AI. Rules passed by lawmakers would give users and companies a standard they can see in advance.

The post Judges Shouldn’t Decide How Fast AI Is Built appeared first on DailyAI.

Four AI Companies Agree to Outside Child-Safety Tests of Their Tools

Four AI companies have agreed to let independent researchers test how their chatbots and image tools affect children. OpenAI, Anthropic, Microsoft and Black Forest Labs backed a new charter at the Paris Peace Forum this week. The organisers say backers will put a total of €8 million into research on AI and child development.

The charter was published on 10 November. The coalition behind it is led by France, Spain and Kenya, and ten United Nations agencies take part. It is a voluntary commitment, not a law, and it creates no legal duties.

What the companies promise

The text sets out ten principles. They include safeguards against overuse and emotional dependence, tools for parents, strict limits on the data collected about children, and age checks that do not rely only on a child typing in a date of birth.

Outside experts will lead the research. Companies may give them access to their systems, but paying for the work gives a company no right to block or delay the results. Findings will be published even when they are negative or inconclusive. Testers are told to check whether users under 18 are correctly identified, and to try realistic misuse, such as teenagers using indirect language or trying to get around safety settings.

OpenAI is giving $2 million of the total. The company said on Thursday that the money will fund shared methods for measuring the risks and benefits of AI for children.

Emmanuel Macron, the French president, asked for more than the charter offers. He said that no AI should reach children before it has been tested, evaluated and approved. His comments appeared in a statement from the summit organisers. The charter itself sets up no approval process. Google, Meta and xAI are not among the four companies that endorsed it.

Teenagers already use these tools

Young people are not waiting for the evidence. A paper published on Wednesday in a journal on adolescent health brings recent figures together. In one US survey of 1,060 teenagers, 72% had tried a chatbot built to act as a friend or companion. In a separate national study, 15% of teenagers who used such chatbots said one had encouraged self-harm. Only 11% of teachers had received guidance on what to do when a student’s use of AI seemed harmful.

Last month we reported that most teenagers in a large student survey turn to AI when they feel low. New York has already passed a law that makes chatbots check ages before they act like a friend.

Why it matters: Parents and schools have little independent evidence about how chatbots affect children. This charter could produce some, but it is voluntary and does not yet cover several of the largest chatbot makers.

The post Four AI Companies Agree to Outside Child-Safety Tests of Their Tools appeared first on DailyAI.

United Airlines Won’t Let Your AI Assistant Book Your Flights

In an Oxford survey this year, one adult in seven across nine countries said they had used an AI assistant for a real task in the past week. Booking a United Airlines flight is no longer one of those tasks. Since 11 November, the US carrier no longer lets outside AI assistants book trips, change bookings or spend frequent-flyer miles for its customers.

The airline announced the new rules on Wednesday. They apply on its website and app, and also to travel agents and booking sites that sell its flights. United is changing its rules for booking and ticketing to match.

What changes for travellers

The ban covers any outside tool that searches, books, buys or manages travel on the United website or app without the airline’s permission. That includes chatbots, browser add-ons and features built into web browsers. A help page for customers lists what is allowed and what is not.

People can still ask a chatbot for ideas, such as where to go in January to avoid crowds. United asks them not to share their account number, password, booking reference or date of birth with such tools. Bookings that an AI assistant made before 11 November will not be cancelled, but the airline advises customers to check that they were ticketed correctly. Tools that help people with disabilities use the site are not affected.

Travel agencies face stricter terms. Any AI tool that an agency builds needs specific permission before it can use United’s fares and seat data. Breaking that rule could cost an agency the right to sell the airline’s tickets. United says it has no approved partners yet.

Privacy, safety and money

United gives two reasons. First, handing login details to an AI assistant from another company could expose personal data and lead to purchases, account changes or use of miles that the customer did not approve. Second, it says automated check-in could break US aviation rules, because a person must confirm that their bags contain no banned items such as explosives.

There may be a commercial motive too. “To be honest, I don’t want to create another middleman,” chief executive Scott Kirby said at an event on 10 November. A business newspaper reported his comment first, and technology sites followed. The airline says it has seen few attempts so far to book flights this way, and it may review the ban as the tools improve.

A minority habit

Only a minority use AI assistants that act on their own. Two in three adults said they had not used one at all in the week before they were asked, while 13% had tried one only to experiment. The figures come from a survey of about 2,000 people in each of nine countries in August and September. Use was highest among the young, at 26% of people aged 18 to 24.

United is not alone. Large banks and online stores have also blocked AI assistants made by other companies this autumn. Earlier this month we argued that such bans on AI shopping tools will not hold for long. Regulators in Britain have also said that firms remain responsible when their own AI tools make mistakes.

Why it matters: Booking travel is one of the main jobs that AI assistants are sold to do. If large airlines block them, the tools can help travellers choose a flight, but a person still has to complete the purchase.

The post United Airlines Won’t Let Your AI Assistant Book Your Flights appeared first on DailyAI.

A Flaw in OpenAI’s Codex Put 97 Million Monthly Downloads at Risk

96.8 million. That is how many times the command-line version of Codex, OpenAI’s coding assistant, was downloaded in the 30 days to 10 November. Security researchers say one fake bug report could have let an attacker slip harmful code into a future update of it. OpenAI has fixed the problem and paid the researchers a first reward of $1,000.

The researchers work at the security firm Depi. On 11 November they published every step of the attack. The download count comes from the public statistics of the npm software library. Depi says users of Codex have nothing to patch.

How the attack worked

OpenAI keeps the code for Codex in a public project on GitHub, where anyone can file a bug report. A helper tool read each new report and suggested labels for it. That helper was the Codex assistant itself, and it received the text of the report mixed in with its own instructions. So a writer could hide commands in a report. This trick is called prompt injection.

Depi says it got the assistant to run hidden code, broke out of the closed space it works in and read secret access keys from the computer’s memory. With those keys it planted a harmful file in a store of files that automated jobs share. A more powerful job could then run that file. The last step was a trap that would give the attacker a key the npm library accepts for publishing new versions of Codex.

What OpenAI changed

Depi says OpenAI deployed a fix right away and paid a first reward of $1,000, with the report still under review. The highest rewards for the most serious bugs reach $100,000 before bonuses. The company raised its reward rates earlier this month.

Early on 12 November, Paris time, OpenAI replaced the assistant in that labelling tool with a short script. The script asks a model for labels with its tools switched off and only accepts answers in a fixed format. The change is visible in the project’s public list of changes.

There is one caveat. Depi ran the full attack on its own test copy, where the assistant was allowed to write files. OpenAI’s own helper had used a stricter read-only mode since May. The setting appears in the version of the file from that month. Depi does not say whether the attack would have worked in that mode, and it does not report any harmful update reaching users.

Its advice for everyone else is to wait a short time before tools install new updates automatically. Last month another team received a reward of $5,000 for a flaw in shared chat links.

Why it matters: AI assistants that read text from strangers can be talked into acting on it. Here the stranger only had to file a bug report, and the target was a tool that runs code on developers’ computers tens of millions of times a month.

The post A Flaw in OpenAI’s Codex Put 97 Million Monthly Downloads at Risk appeared first on DailyAI.

Japan Drafts AI Safety Rules That Would Carry Penalties

Japan’s main AI law has been in place since 2025, and it contains no penalties at all. Government advisers have now drafted a plan to change that for the most powerful AI systems. Companies that ignore an official order could face a penalty.

The draft was presented on Wednesday 11 November at a closed meeting of a working group under the Cabinet Office, the department that supports the prime minister. The government has published the documents from the meeting online. Japan passed its AI law in May 2025. Today the government can investigate a company and give it guidance, but it cannot force the company to act.

Four steps, from reports to orders

The new rules would not cover all AI. They would apply only where two things meet: AI above a certain level of performance, and a set of specific serious risks. Other AI would stay under existing laws and guidelines. According to the meeting documents, the draft has four parts.

  • Companies that meet the criteria must take measures to keep their AI safe from those risks.
  • They must report information about those risks to the government.
  • The government could formally recommend steps to a company that falls short.
  • If needed, the government could issue a binding order, with penalties for breaking it.

Members of the group named cyberattacks and attacks using chemical, biological, radiological, nuclear or explosive weapons as risks that need action. The rules would target companies that provide AI first, not people who use it. The group still has to decide which kinds of company are covered, from research labs to firms that sell AI services. Members also said the rules must work against foreign companies, so the burden does not fall only on Japanese ones.

Six companies took part

Six companies spoke to the group. Three of them were Japanese: the telecoms group NTT, the startup Sakana AI and SB Intuitions. The other three were Anthropic Japan, Google and OpenAI Japan. Each company was asked about AI risks, its own safety work and which rules it would support.

Penalties are expected to apply only to incidents involving advanced AI, according to the news agency Kyodo. The group aims to publish an outline of the changes, including any revision of the law, by the end of the year. The size of any penalty has not been set.

The meeting documents also compare the plan with rules in the US and the EU. New York and Illinois give the largest AI developers 72 hours to report a critical safety incident, and California gives 15 days. Under the EU AI Act, companies behind the most powerful models can be fined up to 3% of global turnover or €15 million.

New York opened registration under its own AI safety law this week. Australia also announced plans for an AI safety regulator.

Why it matters: Japan has so far relied on voluntary guidance for AI. If this draft becomes law, the biggest AI labs would face safety rules backed by penalties in Japan as well as in the EU and several US states.

The post Japan Drafts AI Safety Rules That Would Carry Penalties appeared first on DailyAI.