robots.txt availability
Serve `/robots.txt` from the `/robots.txt` path as `text/plain` (not HTML) with a 2xx status; never let it return 5xx/429 or redirect to the homepage or another host. Add `User-agent` rules for generic and AI crawler user agents and an absolute `Sitemap:` URL (optional but recommended). Drop, or keep as advisory only, the directives major crawlers ignore (`crawl-delay`, `host`, `request-rate`, `visit-time`, `clean-param`), strip any leading BOM, and keep the file within a sane size.
robots.txt availability
- Category
- Discoverability
- Remediation key
- robots_txt_redirected
- URL
- https://tisland.travel/
A public `/robots.txt` is reachable at the `/robots.txt` path and returns either HTTP 2xx with a parseable `text/plain` body (ideally with `User-agent` rules and an absolute `Sitemap` directive) or a 4xx, which agents read as 'no restrictions'. It must not return 5xx/429 (which forces crawlers to assume a full disallow), redirect off `/robots.txt` or to another host, be served as HTML, or exceed the parse limit.
Serve `/robots.txt` from the `/robots.txt` path as `text/plain` (not HTML) with a 2xx status; never let it return 5xx/429 or redirect to the homepage or another host. Add `User-agent` rules for generic and AI crawler user agents and an absolute `Sitemap:` URL (optional but recommended). Drop, or keep as advisory only, the directives major crawlers ignore (`crawl-delay`, `host`, `request-rate`, `visit-time`, `clean-param`), strip any leading BOM, and keep the file within a sane size.
AI-agent prompt
You are a senior web developer. Improve tisland.travel for AI agents. Task: add or repair support for robots.txt availability (robots_txt). Current check result:
{"evidence":{"final_url":"https://www.tisland.travel/robots.txt","headers":{"content-type":"text/plain; charset=utf-8","vary":"Accept-Encoding"},"redirects":[{"location":"https://www.tisland.travel/robots.txt","status_code":301,"url":"https://tisland.travel/robots.txt"}],"requested_url":"https://tisland.travel/robots.txt","status_code":200},"outputs":{"final_url":"https://www.tisland.travel/robots.txt","http_status":200},"remediation_key":"robots_txt_redirected","scanner_message":"robots.txt redirected off /robots.txt to https://www.tisland.travel/robots.txt.","status":"partial"}Expected result: A public `/robots.txt` is reachable at the `/robots.txt` path and returns either HTTP 2xx with a parseable `text/plain` body (ideally with `User-agent` rules and an absolute `Sitemap` directive) or a 4xx, which agents read as 'no restrictions'. It must not return 5xx/429 (which forces crawlers to assume a full disallow), redirect off `/robots.txt` or to another host, be served as HTML, or exceed the parse limit. Fix guidance: Serve `/robots.txt` from the `/robots.txt` path as `text/plain` (not HTML) with a 2xx status; never let it return 5xx/429 or redirect to the homepage or another host. Add `User-agent` rules for generic and AI crawler user agents and an absolute `Sitemap:` URL (optional but recommended). Drop, or keep as advisory only, the directives major crawlers ignore (`crawl-delay`, `host`, `request-rate`, `visit-time`, `clean-param`), strip any leading BOM, and keep the file within a sane size. Stack templates: - Generic web server: serve `/robots.txt` from the public web root or a route, with the correct status, `Content-Type`, and cache policy. - Nginx: add a `location`/`add_header` for `/robots.txt`; verify `try_files`, redirects, and that private upstreams are not exposed. - Apache: configure `Alias`, `Header set`, or a rewrite rule for `/robots.txt`; check `.htaccess` and virtual host precedence. - Next.js: use `public/`, route handlers, Metadata API, or middleware for `/robots.txt`; do not disable SSR/SSG for public content. - Nuxt: use `public/`, server routes, `useHead`/Nitro middleware for `/robots.txt`; verify prerender/SSR output. - Laravel: add a route/controller, Blade metadata, or middleware headers for `/robots.txt`; do not bypass auth/authorization for private routes. - Django: add a `urls.py` route/view, template metadata, or middleware headers for `/robots.txt`; verify `ALLOWED_HOSTS` and cache behavior. - WordPress: implement `/robots.txt` via theme/plugin hooks, SEO plugin settings, or a static file; do not edit core files. - Bitrix: add `/robots.txt` via the site template, component, or web server config; verify composite cache and multi-site rules. - Static hosting: add `/robots.txt` as a static asset or platform headers file (`_headers`, `vercel.json`, `netlify.toml`). - Headless CMS: generate `/robots.txt` in the delivery layer/edge app from published content; do not expose draft or admin APIs. Constraints: do not break SEO; do not expose private endpoints; preserve SSRF/redirect safety; do not add remote JS execution to the scanner; add tests. Return a patch/diff and a short explanation.
Evidence JSON
{ "headers": { "vary": "Accept-Encoding", "content-type": "text/plain; charset=utf-8" }, "final_url": "https://www.tisland.travel/robots.txt", "redirects": [ { "url": "https://tisland.travel/robots.txt", "location": "https://www.tisland.travel/robots.txt", "status_code": 301 } ], "status_code": 200, "requested_url": "https://tisland.travel/robots.txt" }{ "final_url": "https://www.tisland.travel/robots.txt", "http_status": 200 }"robots_txt_redirected""robots.txt redirected off /robots.txt to https://www.tisland.travel/robots.txt.""partial"