How to Get a New Cloudflare Workers Site Indexed

A new site on a workers.dev subdomain starts with no links pointing to it, which is the hardest problem in search. You cannot buy your way out of it on a free plan, but you can remove every avoidable obstacle and tell the engines that support it about each new page the moment it exists. This is the setup we use, what we saw in the first hours, and what it cannot do.

1. A sitemap that includes images

Serve /sitemap.xml from your Worker, generated from the same index that renders the site. Include every page with a lastmod date, and add an image entry for each page that has a picture, so image search can find it:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1">
  <url>
    <loc>https://your-site.workers.dev/devnotes/some-post/</loc>
    <lastmod>2026-10-02</lastmod>
    <image:image><image:loc>https://your-site.workers.dev/media/devnotes/some-post/img.jpg</image:loc></image:image>
  </url>
</urlset>

Point to it from robots.txt, and keep private areas out:

User-agent: *
Allow: /
Disallow: /admin/
Sitemap: https://your-site.workers.dev/sitemap.xml

2. IndexNow for Bing, Yandex and friends

IndexNow lets you push a list of changed URLs instead of waiting to be crawled. Generate a random key (we use 32 hex characters), serve it as plain text at /<key>.txt, and POST your URLs:

await fetch("https://api.indexnow.org/indexnow", {
  method: "POST",
  headers: { "content-type": "application/json; charset=utf-8" },
  body: JSON.stringify({
    host: "your-site.workers.dev",
    key: KEY,
    keyLocation: `https://your-site.workers.dev/${KEY}.txt`,
    urlList: [/* new pages, section pages, home, sitemap */],
  }),
});

We got 200 from api.indexnow.org and from Bing, and 202 from Yandex, which means accepted. Three practical rules:

• Batch. Send all the URLs of a publishing run in one request. When we submitted page by page, the requests came back 429 (rate limited).
• Include the hubs. A new article changes the section page, the home page and the sitemap too, so ping those along with the article.
• Do it automatically. Our publishing cycle pings after every run, so nobody has to remember.

The key file is just a route in the Worker that answers only for the stored key, so nothing else can be mistaken for it.

3. Structured data and plain facts

Each article page carries JSON-LD of type Article with the headline, date, image and author, and a FAQPage block built from the same question and answer pairs shown on the page. Give every page a canonical URL, a unique title and a description of 120 to 155 characters, and set lang on the html element. None of this is exotic, but a generated site that forgets it looks like spam.

We also publish /llms.txt, a short Markdown index of the site for AI assistants and crawlers: what the site is, its sections with one-line descriptions, and where to find the machine-readable API description. It is a convention rather than a standard, and we cannot claim it brings traffic, but it costs nothing.

4. What we saw in the first hours

On launch day, within the hour after we submitted the first pages with IndexNow, our traffic counter (see Counting Visitors Without Cookies on a Cloudflare Worker) recorded the Yandex crawler fetching pages, and kept recording it through the next hours. Bing's crawler had not yet appeared in our numbers at that point, and Google had not either. A search for site: with our domain returned nothing in the first hours. That is normal for a new site and it is why we measure instead of guessing.

5. What IndexNow cannot do

Google does not take part in IndexNow. For Google you need two things that a free plan cannot automate. First, links from other places, which come from being useful: a page that answers a question precisely, with working code, gets linked to. Second, a Google Search Console property for the site, where you submit the sitemap and can request indexing of important URLs. Verifying that property needs a Google login, so it is a one-time job for the site owner: choose the HTML file method, and serve the file it gives you from a Worker route.

A launch checklist

1. sitemap.xml with lastmod and images, referenced from robots.txt.
2. IndexNow key served at /<key>.txt, and a batched ping after every publish.
3. JSON-LD Article and FAQPage, canonical URLs, unique titles and descriptions.
4. A counter that tells you which crawlers actually showed up.
5. Search Console, once, by the owner.
6. Content that is worth linking to.

The point about content is the one that matters. We put real effort into this section of the site for that reason: the guides here are written from what we built and tested, such as How to Add x402 Payments to a Cloudflare Worker with Hono.

FAQ

Does Google support IndexNow?

No. IndexNow is supported by Bing, Yandex, Naver, Seznam and others that share submissions through api.indexnow.org. For Google you need links to your site and, to speed things up, a Google Search Console property with your sitemap submitted.

Where does the IndexNow key file go?

At the root of your site as a text file named after the key, for example /<key>.txt, containing only the key. A Worker can serve it from a route; the submission then names it in the keyLocation field.

Why did IndexNow return 429?

Submitting repeatedly in a short time gets rate limited. Batch your URLs into one request instead of pinging once per page.

Does a workers.dev site get indexed?

Yes, search engines index workers.dev subdomains. A custom domain is stronger for the long term, but it is not required to start.