# As a condition of accessing this website, you agree to abide by the following # content signals: # (a) If a content-signal = yes, you may collect content for the corresponding # use. # (b) If a content-signal = no, you may not collect content for the # corresponding use. # (c) If the website operator does not include a content signal for a # corresponding use, the website operator neither grants nor restricts # permission via content signal with respect to the corresponding use. # The content signals and their meanings are: # search: building a search index and providing search results (e.g., returning # hyperlinks and short excerpts from your website's contents). Search does not # include providing AI-generated search summaries. # ai-input: inputting content into one or more AI models (e.g., retrieval # augmented generation, grounding, or other real-time taking of content for # generative AI search answers). # ai-train: training or fine-tuning AI models. # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF # RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET. # BEGIN Cloudflare Managed content User-Agent: * Content-signal: search=yes,ai-train=no Allow: / User-agent: Amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: GPTBot Disallow: / User-agent: meta-externalagent Disallow: / # END Cloudflare Managed Content # ============================================================================== # Enterprise-Grade Robots.txt for Roniversal.co.il (FIXED VERSION) # Strategy: Maximize SEO ROI, protect crawl budget, block AI training, allow AI traffic # Updated: 2025-11-20 | Corrected REST API & Asset Handling # ============================================================================== # --- Section 1: Sitemap Declaration --- Sitemap: https://roniversal.co.il/sitemap_index.xml # Add separate sitemaps if you have them: # Sitemap: https://roniversal.co.il/product-sitemap.xml # Sitemap: https://roniversal.co.il/category-sitemap.xml # ============================================================================== # SECTION 2: AI SEARCH BOTS (ALLOW - Drive Traffic & Sales) # ============================================================================== # These bots power AI search engines that send qualified traffic User-agent: ChatGPT-User Allow: / Crawl-delay: 1 User-agent: Claude-SearchBot Allow: / Crawl-delay: 1 User-agent: OAI-SearchBot Allow: / Crawl-delay: 1 User-agent: PerplexityBot Allow: / Crawl-delay: 1 User-agent: YouBot Allow: / Crawl-delay: 1 User-agent: Google-InspectionTool Allow: / # ============================================================================== # SECTION 3: AI TRAINING SCRAPERS (BLOCK - Protect IP) # ============================================================================== User-agent: CCBot Disallow: / User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Bytespider Disallow: / User-agent: FacebookBot Disallow: / User-agent: Diffbot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Omgilibot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: cohere-ai Disallow: / User-agent: PerplexityBot-AI Disallow: / # ============================================================================== # SECTION 4: SEO TOOL CRAWLERS (BLOCK - Save Crawl Budget) # ============================================================================== User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: SemrushBot-BA Disallow: / User-agent: SemrushBot-SI Disallow: / User-agent: SemrushBot-SA Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: Scrapy Disallow: / User-agent: serpstatbot Disallow: / User-agent: linkdexbot Disallow: / User-agent: spbot Disallow: / User-agent: MegaIndex Disallow: / User-agent: SERankingBacklinksBot Disallow: / User-agent: MozDotBot Disallow: / User-agent: sistrix User-agent: SISTRIX User-agent: SISTRIX Crawler Disallow: / User-agent: SiteCheckerBot Disallow: / User-agent: SEOkicks User-agent: SEOkicks-Robot Disallow: / User-agent: python-requests Disallow: / User-agent: aiohttp Disallow: / User-agent: httpx Disallow: / User-agent: Go-http-client Disallow: / User-agent: HeadlessChrome Disallow: / User-agent: Puppeteer Disallow: / User-agent: Playwright Disallow: / User-agent: BuiltWith Disallow: / User-agent: ZoominfoBot Disallow: / User-agent: CapterraBot Disallow: / User-agent: ImageBot Disallow: / User-agent: Pinterestbot Disallow: / User-agent: SEBot-WA Disallow: / # ============================================================================== # SECTION 5: AGGRESSIVE/SPAM CRAWLERS (BLOCK) # ============================================================================== User-agent: PetalBot Disallow: / User-agent: Nutch Disallow: / User-agent: Heritrix Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: WebBandit Disallow: / User-agent: WebCopier Disallow: / User-agent: HTTrack Disallow: / User-agent: ia_archiver Disallow: / # ============================================================================== # SECTION 6: INTERNATIONAL SEARCH ENGINES (Selective) # ============================================================================== # Yandex (Russia) - Block if not targeting Russian market User-agent: Yandex Disallow: / # Seznam (Czech Republic) User-agent: SeznamBot Disallow: / # Baidu (China) - Block if not targeting Chinese market User-agent: Baiduspider Disallow: / # To enable for specific markets, change to: # User-agent: Yandex # Allow: / # Crawl-delay: 2 # ============================================================================== # SECTION 7: MAIN SEARCH ENGINES (Google, Bing, etc.) # ============================================================================== User-agent: * # --- 7A: CRITICAL FIX - Allow Assets & REST API --- # Modern WordPress/WooCommerce needs these for proper rendering # Allow ALL plugin assets (CSS, JS, fonts, images, etc.) Allow: /wp-content/plugins/ # Allow ALL theme assets Allow: /wp-content/themes/ # Allow uploads (product images, etc.) Allow: /wp-content/uploads/ # CRITICAL: Allow REST API for Gutenberg/WooCommerce dynamic content Allow: /wp-json/ # Allow admin-ajax for AJAX functionality Allow: /wp-admin/admin-ajax.php # Allow core WordPress includes Allow: /wp-includes/ # --- 7B: WordPress Admin & System (Block) --- Disallow: /wp-admin/ Disallow: /wp-login.php Disallow: /wp-register.php Disallow: /wp-signup.php Disallow: /wp-activate.php Disallow: /xmlrpc.php Disallow: /trackback/ Disallow: /cgi-bin/ # --- 7C: WooCommerce User Pages (Block - No SEO Value) --- Disallow: /cart/ Disallow: /checkout/ Disallow: /my-account/ Disallow: /orders/ Disallow: /order-received/ Disallow: /order-tracking/ Disallow: /wishlist/ Disallow: /downloads/ Disallow: /addons/ # --- 7D: Search & Filter Parameters (Block Duplicate Content) --- # Internal search Disallow: /search/ Disallow: /*?s= Disallow: /*&s= # Product filtering/sorting Disallow: /*?*filter Disallow: /*?*orderby= Disallow: /*?*min_price= Disallow: /*?*max_price= Disallow: /*?*product_cat= Disallow: /*?*product_tag= Disallow: /*?*rating_filter= # Cart/wishlist actions Disallow: /*?*add-to-cart= Disallow: /*?*added-to-cart= Disallow: /*?*add_to_wishlist= Disallow: /*?*removed_item= Disallow: /*?*undo_item= # Tracking parameters Disallow: /*?*utm_ Disallow: /*&*utm_ Disallow: /*?*gclid= Disallow: /*?*fbclid= Disallow: /*?*mc_cid= Disallow: /*?*mc_eid= Disallow: /*?*cst= Disallow: /*?*ref= Disallow: /*?*source= # --- 7E: WordPress Duplicate Routes --- Disallow: /*?page_id= Disallow: /*?cat= Disallow: /*?tag= Disallow: /*?attachment_id= Disallow: /*?p= # IMPROVED: Block thin content archives (saves crawl budget) Disallow: /author/ Disallow: /date/ # --- 7F: Page Builder Previews (Block) --- Disallow: /*?*elementor_library= Disallow: /*?*elementor-preview= Disallow: /*?*elementor_editor= Disallow: /*?*preview=true Disallow: /*?*preview_id= # --- 7G: JetEngine/Elementor Dynamic Queries --- Disallow: /*?*query-*-page= # --- 7H: Feeds & Pagination --- # RSS feeds don't need indexing Disallow: /feed/ Disallow: /*/feed/ Disallow: /*?feed= Disallow: /comments/feed/ Disallow: /*/*/feed/ # Paginated archives (canonical tags handle this) Disallow: /page/ Disallow: /*/page/ # --- 7I: Security - System Files --- Disallow: /readme.html Disallow: /license.txt Disallow: /wp-config.php Disallow: /.htaccess Disallow: /error_log Disallow: /.env Disallow: /composer.json Disallow: /package.json Disallow: /wp-content/cache/ Disallow: /wp-content/backup*/ # ============================================================================== # SECTION 8: ADVANCED E-COMMERCE OPTIMIZATION # ============================================================================== # Block parameter combinations (faceted navigation duplicates) Disallow: /*?*&* # Block session IDs Disallow: /*?*PHPSESSID= Disallow: /*?*sid= Disallow: /*?*session=