robots.txt in Next.js App Router, Crawl Budget and Disallow Pitfalls (2026)

TL;DR

robots.txt tells crawlers which paths they may fetch, it is not a privacy tool (use noindex + auth for that). In the Next.js App Router you generate it with app/robots.ts (MetadataRoute.Robots), point to your sitemap, and deliberately limit /api/, staging, and admin areas. A mistaken Disallow: / in production or blocking CSS/JS breaks indexing and how Google renders your pages. Below: crawl budget for service sites, safe vs dangerous rules, staging vs production on DevStudioIT Cloud, and a Search Console checklist.

Who is this for

• Next.js (App Router) developers owning technical SEO
• Companies after migration/redesign or stuck on “Discovered, not indexed”
• Teams with separate staging and production
• SEOs managing crawl budget as the blog/catalog grows
• Anyone who left a tutorial’s Disallow: / on a live site

Keyword

robots.txt nextjs app router, crawl budget google 2026, disallow robots.txt mistakes, metadataroute robots, search console robots.txt, staging noindex

What robots.txt is, and is not

A URL can still appear in the index despite Disallow (e.g. from backlinks), often without a snippet. To drop it from results: noindex and optionally GSC Removals.

Crawl budget, should a service firm care?

For 50–500 URLs (offer + blog + case studies) crawl budget is rarely the bottleneck. It matters when you have:

• Query parameters (?utm_, filters, sort) as separate URLs
• Faceted navigation across services / cities
• CMS publishing thousands of tags and paginated archives
• Staging/preview leaking to the public internet
• APIs and assets crawled as if they were HTML pages

Goal of robots.txt: stop wasting crawls on /api/, preview, admin, parameter duplicates so Google returns more often to offer pages, blog posts, and landings.

Dynamic content (case studies from PostgreSQL in Branchly) still belongs in the sitemap, robots will not invent those URLs.

Next.js App Router, app/robots.ts

// app/robots.ts
import type { MetadataRoute } from 'next';

const BASE = 'https://devstudioit.com';
const allowIndexing = process.env.ALLOW_INDEXING === 'true';

export default function robots(): MetadataRoute.Robots {
  if (!allowIndexing) {
    return {
      rules: { userAgent: '*', disallow: '/' },
      // no sitemap on staging
    };
  }

  return {
    rules: [
      {
        userAgent: '*',
        allow: '/',
        disallow: ['/api/', '/admin/', '/draft/', '/preview/'],
      },
    ],
    sitemap: `${BASE}/sitemap.xml`,
    host: BASE,
  };
}

Set ALLOW_INDEXING=true only in production (DevStudioIT Cloud). Staging: Disallow: / and robots: { index: false } in metadata, defense in depth.

Disallow rules, safe vs traps

Safe (typical corporate site)

User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /draft/
Sitemap: https://devstudioit.com/sitemap.xml

Dangerous

Next.js serves JS/CSS under /_next/static/.... Do not block /_next/.

Trailing slash and locales

If the canonical is https://example.com/en/services (no trailing slash), do not invent Disallow-only logic for slash variants, use 301 + canonical (see the canonical article). robots.txt does not fix PL/EN/DE duplicates.

robots.txt and the sitemap

A Sitemap: line speeds discovery, but:

1. Sitemap URLs do not override Disallow, blocked URLs must not appear in the sitemap
2. Audit that staging sitemaps are not linked from production
3. After a big republish: GSC → Sitemaps → check discovered URLs

Verify in Google Search Console

1. Check indexing reasons for “Blocked by robots.txt”
2. URL Inspection → crawled page / robots line
3. Test paths: /, /api/health, /en/blog/...
4. After rule changes, allow re-crawl time; request indexing for key URLs only

robots.txt, middleware, and locales

With [locale] middleware that redirects / → /en, make sure:

• robots.txt is reachable at the domain root (https://domain/robots.txt), not only under /en/robots.txt
• Next.js app/robots.ts serves the file outside the locale prefix, that is how MetadataRoute works
• You do not Disallow locale paths you want indexed

If the CDN on DevStudioIT Cloud caches robots.txt, purge after rule changes, otherwise Googlebot may keep the old file for a long time.

Example: staging subdomain rules

# staging.example.com/robots.txt
User-agent: *
Disallow: /

Never copy that file 1:1 onto production. ALLOW_INDEXING plus a separate staging domain is the pattern that stops index leaks after redesigns.

Production checklist

1. ALLOW_INDEXING=true only on prod
2. app/robots.ts allows site, disallows /api/, admin, draft
3. No Disallow on /_next/
4. Sitemap: points to production /sitemap.xml
5. Staging: Disallow all + noindex + separate domain
6. No blocked URLs inside the sitemap
7. GSC: no critical robots blocks on offer/blog
8. After deploy: curl https://domain/robots.txt and review by hand
9. Purge CDN cache for /robots.txt after changes
10. Confirm /pl, /en, /de are Allow

FAQ

Will robots.txt hide the client portal?

Not enough. Portal needs auth + noindex + Disallow. Secrets must never live in HTML.

Should I block UTM URLs in robots.txt?

Prefer canonicalizing UTMs (canonical without params) and avoid internal links with UTMs. Disallow: /*?* is often too aggressive.

Googlebot vs other bots

You can split User-agent: Googlebot rules, but * is usually enough for a corporate site. Over-splitting makes debugging harder.

What about AI crawlers (GPTBot, etc.)?

A business policy choice: Allow/Disallow separately. Do not confuse that with Google Search crawling.

CTA

Need an audit of robots.txt, sitemap, and Next.js indexing in production?

• Book a crawl / GSC audit, robots.ts, staging vs prod, DevStudioIT Cloud