Web4Agents

Make the web agent-ready. Open source.

Crawl4AI extracts data FROM websites for your agents.
Web4Agents makes websites ready FOR any agent to use.
Complementary tools. Same mission.
⭐ Star on GitHub See Demo
0

Average website Agent Readiness Score

πŸ“„

llms.txt

AI content guide β€” tells agents what matters on your site

15 points
πŸ“‹

llms-full.txt

Full clean markdown of site content for direct AI consumption

10 points
πŸ€–

robots.txt AI Policy

Are AI bots (GPTBot, ClaudeBot, etc.) allowed or blocked?

15 points
πŸ—ΊοΈ

sitemap.xml

Discoverable page index for agents to navigate

5 points
🏷️

Schema.org / JSON-LD

Structured data agents can parse without guessing

15 points
🧩

Semantic HTML

Proper labels, ARIA, nav structure for agent navigation

10 points
πŸ”Œ

WebMCP / Tool Contract

Structured actions agents can call directly

15 points
πŸ›‘οΈ

Bot/CAPTCHA Blocking

Does the site actively block AI agents?

10 points
πŸ“–

Content Extractability

Can agents extract clean, meaningful content?

5 points

πŸ”­ Watch It Scan

Real checks against accenture.com

web4agents β€” scanner
$ node src/scanner.js https://www.accenture.com

Roadmap

v0.1 β€” Now

Scanner

Check any site's Agent Readiness Score

v0.2 β€” Next

Generator

Auto-create llms.txt, Schema.org, WebMCP

v0.3 β€” Future

Dashboard

Web UI, batch scanning, trend tracking