INETDEX Bot
Crawler identification and webmaster information
INETDEX-BOT/2.0 is the crawler used to discover, fetch, and index publicly accessible web pages for INETDEX.
Identification
| Bot name | INETDEX-BOT |
|---|---|
| Version | 2.0 |
| User-Agent | INETDEX-BOT/2.0 |
| Purpose | Indexing publicly available web content for niche search engines, curated collections, and ranking/top lists. |
Robots and Crawl Control
INETDEX-BOT respects standard crawling controls:
- robots.txt rules (
Disallow,Allow, etc.) - Common noindex directives where applicable (page-level directives and headers)
- Site stability and rate limiting to avoid overloading servers
Recommended
Please provide a /robots.txt file if you want to control crawling behavior.
What We Crawl
- Public pages reachable without authentication
- HTML documents and linked resources that are relevant for indexing
- Content intended for public discovery and access
What We Avoid
- Pages behind logins, paywalls, or requiring personal access
- Content explicitly disallowed by
robots.txt - Pages that may cause harm to server stability (excessive errors, loops, unstable endpoints)
Contact
If you have questions about crawling, indexing, or want to request adjustments, please contact us:
- Contact form: /contact.php
Administrator Notes
For best interoperability, keep your site’s canonical URLs consistent, avoid infinite calendars, and ensure your robots rules reflect your intent. If your site is experiencing unexpected load, contact us and include timestamps and sample URLs.