Crawler guide
PetalBot: what it is and how to block it.
Who runs Huawei's Petal crawler, what it does with your pages, how to spot its user agent in your logs, and what blocking it costs you.
Short answer
What is Huawei's Petal crawler?
PetalBot is the automatic crawler of Petal Search, Huawei's search engine. It indexes PC and mobile pages so people can search your site in Petal. That content also feeds recommendations in Huawei Assistant and AI Search. You control it in robots.txt.
In brief
Four things to know first.
A search crawler
It builds the index behind Petal Search. Its documentation describes no model training use.
Easy to recognise
Every request names the crawler in its user agent string. A reverse and forward DNS check confirms it.
It obeys robots.txt
A User-agent group naming the crawler, with Disallow: /, blocks it. You can also allow some paths only.
Blocking has a cost
Your pages become unsearchable in Petal and its search services. Removal of indexed pages can take months.
Who and why
What PetalBot does with your content.
Petal's webmaster site calls its crawler the automatic program of the Petal search engine. It accesses PC and mobile websites to build an index database, so users can search your content in Petal.
Petal says crawled content also powers recommendations in Huawei Assistant and AI Search. Both services run on Petal Search.
| Item | Detail |
|---|---|
| Operator | Petal Search (Huawei) |
| Purpose | Indexing for Petal search |
| Also feeds | Huawei Assistant and AI Search |
| Respects robots.txt | Yes, per Petal |
| Contact | Petal support mailbox at huawei.com |
Recognise it
PetalBot user agent and verification.
Petal says you can identify its crawling by the User-agent field. It publishes two strings: one for PC and one for mobile. Match both in your logs.
A user agent can be faked, so check the IP. Petal says to run a reverse lookup on the IP in your logs, then a forward lookup on the returned name. Both addresses must match.
| Version | User agent |
|---|---|
| PC | Mozilla/5.0 (compatible;PetalBot;+https://webmaster.petalsearch.com/site/petalbot) |
| Mobile | Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot) |
Control it
Block this crawler in robots.txt.
Petal says its crawler complies with the Internet robots protocol. You can block it from your whole site or from selected files.
Choose your goal
Decide whether to block Petal's crawler everywhere, block some paths, or keep it fully allowed.Block everything
Name Petal's crawler in a User-agent line, then add Disallow: / on the next line to keep it off your whole site.Or limit it by path
Petal's example group for its crawler: Allow: /w/api/, Disallow: /trap/.Verify the visits
Check log entries with the reverse and forward DNS test, so you act only on real Petal traffic.Report problems
If crawling looks unreasonable, send your concerns to Petal's support address, listed on its webmaster site.
Not sure which crawlers to allow?
We check crawler access on your site as part of Technical GEO, so the pages you want cited can be reached and read.
The trade-off
Should you block this crawler?
Petal says banning this crawler makes your pages, and all search services Petal provides, unsearchable in Petal search. If Huawei users are not your audience, the loss may be small.
Blocking does not clear the index at once. Petal says removing information already indexed may take several months. For urgent requests, email Petal's support address and recheck your robots configuration.
| Allow Petal's crawler | Block Petal's crawler | |
|---|---|---|
| Petal Search | Your pages can be searched | Your pages become unsearchable |
| Huawei services | Content can feed recommendations | Petal's search services lose your pages |
| Server load | Petal adjusts crawl volume to capacity | No requests from this crawler after it picks up the rule |
| Reversal | Nothing to undo | Index clearing can take months |
FAQ
This crawler: common questions.
What is Huawei's Petal crawler?
Petal's crawler is the automatic crawler program of the Petal search engine, run by Huawei. It visits PC and mobile websites to build an index, so users can search your content in Petal. Petal also says that content supports recommendations in Huawei Assistant and AI Search.
What is the user agent of Huawei's Petal crawler?
Petal documents two user agents on its webmaster site: a PC string and an Android 7.0 Mobile Safari string. Both include the crawler's name and a link to Petal's webmaster page, so match either one in your logs.
How do I block Petal's crawler in robots.txt?
Add two lines to robots.txt: a User-agent line naming Petal's crawler, then Disallow: /. Petal says its crawler complies with the robots protocol. For partial blocking, Petal's example allows /w/api/ and disallows /trap/ in that group.
How do I verify a request really comes from Petal's crawler?
Verification uses DNS. Run a reverse lookup on the IP in your logs, then a forward lookup on the returned name. Petal says the forward result must match the original IP. Its example resolves 114.119.128.10 to a hostname ending in petalsearch.com.
Does blocking Petal's crawler remove my pages from Petal at once?
Blocking this crawler does not clear Petal's index immediately. Petal says it may take several months to remove page information already established in its database. For urgent cases, Petal asks you to email its support address and confirm your robots configuration is correct.
Does Petal's crawler overload my server?
Petal says its crawler tries not to burden sites. It adjusts crawl volume based on server capacity, website quality and website updates. If you see unreasonable crawling, Petal asks you to send your concerns to the contact address it publishes: [[huawei.com mailbox]] (the Petal support address on its webmaster site).
Check which crawlers can read your site.
We run your buyers' questions through ChatGPT, Claude, Perplexity and Gemini and show you the gap.