Meta's New Commercial Terms: What Its Crawling Rights Mean

By Jaspal Singh October 17, 2026 Updated: October 17, 2026

From October 30, 2026, Meta's revised Commercial Terms let it crawl any website tied to a business that uses Facebook, Instagram, WhatsApp or Messenger for commercial purposes. It can then use what it collects to build and improve its products, AI included. The terms offer no separate opt-out. A business that rejects them has to stop using Meta's products for business.

What Happened

Meta has published a preview of its updated Commercial Terms, marked "Effective Date: October 30, 2026." The terms cover anyone who uses Meta's products "for a business or commercial purpose." That definition is broad. It includes running ads or marketing messages, selling or listing products, developing apps, managing a Facebook Page or a business Group, and using tools such as Meta Business Suite.

The new clause sits in the section on licences that customers grant Meta. It authorises Meta, or "any entity acting on our behalf," to use automated software, listed as "web scrapers, bots, robots, spiders, crawlers, and user agents," on the domains, pages and URLs associated with a customer's products or services. It also lets Meta "create test credentials" to access, index, cache, analyse, collect, retrieve or crawl those sites. Meta may then use the content it gathers to "provide, develop, improve, research, and maintain" its products. The Register's reading of the clause is that it covers training AI models.

Nieman Lab first reported on October 16 that the terms could reach news publishers, since almost every news organisation runs at least one Meta business account. The same day, the News/Media Alliance, which represents US publishers, called the terms "outrageous and probably illegal." Its president and CEO, Danielle Coffey, said they "cannot be allowed to be implemented on October 30." Ars Technica reported that the clause could save Meta many millions of dollars in content-licensing fees.

Meta's spokesperson Andy Stone replied on X, as reported by Gizmodo. He called one summary of the reporting "false" and said Meta's crawling has not changed. He added that content owners "can use robots.txt flags to indicate their preferences," and that the update is meant to pull accurate product information for ads and recommendations, not to scrape news sites. A Meta spokesperson also told Ars Technica that the "test credentials" wording already appeared in Meta's Seller Agreement.

Why It Matters

Most AI companies that want large amounts of professional content either license it or crawl the open web and accept the legal risk. Meta's approach is a third route: it makes crawling a condition of a business relationship that millions of companies already depend on. A restaurant that runs Instagram ads, a shop that lists products on Facebook Marketplace, and a newspaper that manages a Facebook Page all fall inside the definition of a commercial user.

The scale is large. The Register notes that at least a million businesses use Meta's AI agents in WhatsApp and Messenger alone. Advertisers, sellers and Page managers add many more. For most of them, leaving Meta's platforms is not a realistic choice, because Facebook, Instagram and WhatsApp are where their customers are. Critics say this is why "accept or leave" is not real consent. Eric Goldman, a law professor at Santa Clara University, told The Register that users should get a genuine option to reject amended terms, or they face a Hobson's choice.

Publishers have the most at stake. Their archives have licensing value, and many have signed paid deals with AI companies. If the clause holds up, Meta could get comparable material for free from any publisher with a Facebook Page. Matt Topic, a lawyer at Loevy & Loevy, said it is unclear whether courts would enforce the provision, but publishers should assume they would.

The change is also not entirely new. Search Engine Journal noted that Meta's existing terms already allowed it to scrape advertisers' websites, including for training. The October update extends that permission to businesses that never buy an ad.

How It Works

Three pieces fit together here: the contract, the crawlers that do the work, and robots.txt, which is how site owners usually control crawlers.

The contract: consent by continued use

Under the update section of the Commercial Terms, a customer who does not accept new terms may "stop all access or use of our Products for business or commercial purposes." There is no option to keep using Meta's business tools while declining the crawling clause. Keep running ads or managing a Page after October 30 and you have accepted it. The clause covers sites "associated with" your products or services, so it is not limited to the one URL you list in an ad. Meta does not define how far that association reaches.

The crawlers: five user agents

Meta's developer documentation on its web crawlers lists five user agents, each with its own job:

FacebookExternalHit fetches titles, descriptions and thumbnails when someone shares a link. Meta says it "might bypass robots.txt" during security or integrity checks.
Meta-ExternalAgent crawls the web "to train foundation AI models" or to improve products by indexing content directly. This is the crawler that matters most for AI training.
Meta-ExternalAds crawls to support advertising and other business products.
Meta-WebIndexer builds the index behind Meta AI's search answers, so that Meta can cite and link to pages.
Meta-ExternalFetcher fetches pages when a user asks an AI agent to do something. Meta says it "may bypass robots.txt rules" because those fetches are user-initiated.

Because the jobs are split across separate user agents, a site can make finer choices. It can block the training crawler and still allow the link-preview bot, so shared links keep their thumbnails.

Robots.txt: a request, not a lock

Robots.txt is a plain text file at the root of a site that tells crawlers which paths they may fetch. Its rules are standardised in RFC 9309, the Robots Exclusion Protocol. The RFC states plainly that these rules are not a form of access authorisation. A site that wants to keep Meta's training crawler out while still allowing link previews would add:

User-agent: meta-externalagent
Disallow: /

User-agent: facebookexternalhit
Allow: /

Meta's spokesperson has said that robots.txt preferences still apply. The terms themselves do not say that, though. Goldman pointed out that the law does not necessarily require anyone to honour robots.txt. Once a business has agreed by contract to automated access, a robots.txt line that says otherwise may carry little legal weight.

Test credentials: the paywall question

The clause that worries publishers most is the one about "test credentials." James Grimmelmann, a Cornell law professor, said this wording could let Meta register accounts on a site and reach gated content, possibly including paywalled articles. Meta says the language came from its Seller Agreement. Ars Technica noted that the Seller Agreement version appeared to apply only to URLs that sellers submitted for listing optimisation. In the new terms, the same words apply to every commercial user.

What's Still Unknown

Several key questions remain open less than two weeks before the deadline:

Robots.txt in writing. Meta's spokesperson says robots.txt preferences apply, but the terms do not say so. It is not known whether Meta will add that commitment before October 30.
How far "associated with" reaches. The terms do not say whether a publisher's entire domain counts as associated with its Facebook Page, or whether a brand's separate microsites are included.
Use of credentials. Meta has not said whether, or how, it would use test credentials to get past logins or paywalls.
Enforceability. Courts have not ruled on whether a platform can win scraping rights over third-party websites through a terms update. Copyright claims are another open question.
Regulators. No competition or data-protection authority has commented publicly so far. Meta did not respond to The Register's request for comment.

Frequently Asked Questions

When do Meta's new Commercial Terms take effect?

The revised Commercial Terms take effect on October 30, 2026, according to the preview Meta has published. From that date, continuing to use Meta's products for business or commercial purposes counts as accepting the new terms, including the clause that authorises crawling of websites associated with your business. Until then, the current version of the terms still applies to existing customers.

Does this apply to my personal Facebook or Instagram account?

Not directly. The Commercial Terms cover use of Meta's products for a business or commercial purpose, such as running ads, selling or listing products, developing apps, managing a Page or business Group, or using Meta Business Suite. A purely personal account falls under Meta's ordinary terms. A creator who sells products or manages a business Page could be covered.

Can I opt out of Meta crawling my website?

The terms contain no opt-out. The only alternative they offer is to stop using Meta's products for business. Meta's spokesperson says site owners can use robots.txt to state preferences, and Meta's crawler documentation explains how to block Meta-ExternalAgent, the crawler it uses for AI training. However, the terms do not promise that Meta will honour those robots.txt directives.

Which robots.txt rule blocks Meta's AI training crawler?

Add a group for the user agent meta-externalagent with "Disallow: /" to your robots.txt file. That asks Meta's AI-training crawler to skip the whole site. Leave facebookexternalhit allowed if you want shared links to keep showing previews. Remember that Meta-ExternalFetcher, which serves user-initiated agent requests, may bypass robots.txt rules according to Meta's own documentation.

Could Meta get past paywalls under these terms?

Possibly. The clause lets Meta create test credentials to access, index and crawl associated sites. Cornell law professor James Grimmelmann said that wording could let Meta register on a site and reach gated content, possibly including paywalled material. Meta says the language comes from its existing Seller Agreement. It has not said how it would use credentials on publishers' sites.

Why are news publishers objecting?

Almost every news organisation has a Meta business account to manage Facebook Pages or run ads. Publishers say the terms would let Meta take content they would otherwise license to AI companies, in exchange for nothing more than continued access to Meta's audiences. The News/Media Alliance called the terms "outrageous and probably illegal" and wants them stopped before October 30.

Is this the first time Meta has claimed scraping rights?

No. Search Engine Journal noted that Meta's existing terms already let it scrape advertisers' websites, including for training. A Meta spokesperson said the test-credentials wording already appeared in its Seller Agreement. The October update extends similar rights to every commercial user, including businesses that never buy ads but simply manage a Page or list products.