INETDEX-Bot 2.0

Our new generation crawler is designed to be fast, respectful, and transparent. inetdex Bot 2.0 indexes websites efficiently while giving site owners full control.

🤖 What is inetdex Bot 2.0?

inetdex Bot 2.0 is the web crawler behind inetdex.com. It scans websites to keep our search index fresh and relevant. The bot respects industry standards, uses polite crawling rates, and identifies itself with the user agent string:

User-Agent: inetdex-bot/2.0 (+https://inetdex.com/bot.php)

🔒 How to Exclude the Bot

If you don’t want your site (or certain parts of it) to be crawled, you can exclude inetdex Bot 2.0 using standard methods:

• robots.txt file in the root of your domain
• Meta tag in the page header: <meta name="robots" content="noindex, nofollow">
• X-Robots-Tag HTTP header

📂 Example robots.txt Configuration

Allow or block inetdex Bot 2.0 with simple rules:

# Block the entire site
User-agent: inetdex-bot
Disallow: /

# Allow entire site
User-agent: inetdex-bot
Allow: /

# Block only a directory
User-agent: inetdex-bot
Disallow: /private/

# Allow specific directory while blocking others
User-agent: inetdex-bot
Disallow: /
Allow: /public/
    

✅ Best Practices

• Always place robots.txt in the root of your domain (e.g., https://example.com/robots.txt).
• Use Allow and Disallow to fine-tune crawling.
• Changes may take some time to be recognized as the bot re-crawls.

📩 Contact

If you have questions or need special crawl settings, reach out at bot@inetdex.com.