{"id":8876,"date":"2019-01-31T14:35:41","date_gmt":"2019-01-31T14:35:41","guid":{"rendered":"https:\/\/www.7boats.com\/academy\/?p=8876"},"modified":"2019-10-30T18:15:12","modified_gmt":"2019-10-30T18:15:12","slug":"can-anyone-explain-x-robots-tag-http-header","status":"publish","type":"post","link":"https:\/\/www.7boats.com\/academy\/can-anyone-explain-x-robots-tag-http-header\/","title":{"rendered":"&#8216;X-Robots-Tag&#8217; http header"},"content":{"rendered":"\n<p> X Robots Tag is used to control more data then\u00a0<strong>Robots<\/strong>.<strong>txt<\/strong>\u00a0file and\u00a0<strong>Robots<\/strong>\u00a0Meta\u00a0<strong>Tag<\/strong>. The\u00a0<code>X-Robots-Tag<\/code>\u00a0can be used as an element of the HTTP header response for a given URL. Any directive that can be used in an robots meta tag can also be specified as an\u00a0<code>X-Robots-Tag<\/code>. Here&#8217;s an example of an HTTP response with an\u00a0<code>X-Robots-Tag<\/code>\u00a0instructing crawlers not to index a page:<\/p>\n\n\n\n<div class=\"devsite-code-button-wrapper\">\n<div class=\"devsite-code-button gc-analytics-event material-icons devsite-dark-code-button\" data-category=\"Site-Wide Custom Events\" data-label=\"Dark Code Toggle\" data-tooltip-align=\"b,c\" data-tooltip=\"Dark code theme\" aria-label=\"Dark code theme\" data-title=\"Dark code theme\"><\/div>\n<\/div>\n\n\n\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_82_2 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Page Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.7boats.com\/academy\/can-anyone-explain-x-robots-tag-http-header\/#X-Robots-Tag_noindex\" >X-Robots-Tag: noindex<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.7boats.com\/academy\/can-anyone-explain-x-robots-tag-http-header\/#X-Robots-Tag_noarchive_unavailable_after\" >X-Robots-Tag: noarchive &amp; unavailable_after<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.7boats.com\/academy\/can-anyone-explain-x-robots-tag-http-header\/#X-Robots-Tag_googlebot\" >X-Robots-Tag: googlebot<\/a><\/li><\/ul><\/nav><\/div>\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"X-Robots-Tag_noindex\"><\/span><strong>X-Robots-Tag: noindex<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<pre class=\"wp-block-preformatted\">HTTP\/1.1 200 OK\nDate: Tue, 25 May 2010 21:42:43 GMT\n<em>(\u2026)<\/em>\n<strong>X-Robots-Tag: noindex<\/strong>\n<em>(\u2026)<\/em>\n<\/pre>\n\n\n\n<p>Multiple&nbsp;<code>X-Robots-Tag<\/code>&nbsp;headers can be combined within the HTTP response, or you can specify a comma-separated list of directives. Here&#8217;s an example of an HTTP header response which has a&nbsp;<code>noarchive<\/code>&nbsp;<code>X-Robots-Tag<\/code>&nbsp;combined with an&nbsp;<code>unavailable_after<\/code>&nbsp;<code>X-Robots-Tag<\/code>.<\/p>\n\n\n\n<div class=\"devsite-code-button-wrapper\">\n<div class=\"devsite-code-button gc-analytics-event material-icons devsite-dark-code-button\" data-category=\"Site-Wide Custom Events\" data-label=\"Dark Code Toggle\" data-tooltip-align=\"b,c\" data-tooltip=\"Dark code theme\" aria-label=\"Dark code theme\" data-title=\"Dark code theme\"><\/div>\n<\/div>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"X-Robots-Tag_noarchive_unavailable_after\"><\/span><strong>X-Robots-Tag: noarchive &amp; unavailable_after<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<pre class=\"wp-block-preformatted\">HTTP\/1.1 200 OK\nDate: Tue, 25 May 2010 21:42:43 GMT\n<em>(\u2026)<\/em>\n<strong>X-Robots-Tag: noarchive\nX-Robots-Tag: unavailable_after: 25 Jun 2010 15:00:00 PST<\/strong>\n<em>(\u2026)<\/em>\n<\/pre>\n\n\n\n<p>The&nbsp;<code>X-Robots-Tag<\/code>&nbsp;may optionally specify a user-agent before the directives. For instance, the following set of&nbsp;<code>X-Robots-Tag<\/code>&nbsp;HTTP headers can be used to conditionally allow showing of a page in search results for different search engines:<\/p>\n\n\n\n<div class=\"devsite-code-button-wrapper\">\n<div class=\"devsite-code-button gc-analytics-event material-icons devsite-dark-code-button\" data-category=\"Site-Wide Custom Events\" data-label=\"Dark Code Toggle\" data-tooltip-align=\"b,c\" data-tooltip=\"Dark code theme\" aria-label=\"Dark code theme\" data-title=\"Dark code theme\"><\/div>\n<\/div>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"X-Robots-Tag_googlebot\"><\/span><strong>X-Robots-Tag: googlebot<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<pre class=\"wp-block-preformatted\">HTTP\/1.1 200 OK\nDate: Tue, 25 May 2010 21:42:43 GMT\n<em>(\u2026)<\/em>\n<strong>X-Robots-Tag: googlebot: nofollow\nX-Robots-Tag: otherbot: noindex, nofollow<\/strong>\n<em>(\u2026)<\/em>\n<\/pre>\n\n\n\n<p>Directives specified without a user-agent are valid for all crawlers. The section below demonstrates how to handle combined directives. Both the name and the specified values are not case sensitive.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>X Robots Tag is used to control more data then\u00a0Robots.txt\u00a0file and\u00a0Robots\u00a0Meta\u00a0Tag. The\u00a0X-Robots-Tag\u00a0can be used as an element of the HTTP &hellip;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_vibebp_attr":"","_vibebp_dimensions":"","_vibebp_responsive_height":"","_vibebp_accordion_ie_support":"","footnotes":""},"categories":[315],"tags":[454],"class_list":["post-8876","post","type-post","status-publish","format-standard","hentry","category-question-answers","tag-x-robots-tag"],"_links":{"self":[{"href":"https:\/\/www.7boats.com\/academy\/wp-json\/wp\/v2\/posts\/8876","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.7boats.com\/academy\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.7boats.com\/academy\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.7boats.com\/academy\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.7boats.com\/academy\/wp-json\/wp\/v2\/comments?post=8876"}],"version-history":[{"count":0,"href":"https:\/\/www.7boats.com\/academy\/wp-json\/wp\/v2\/posts\/8876\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.7boats.com\/academy\/wp-json\/wp\/v2\/media?parent=8876"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.7boats.com\/academy\/wp-json\/wp\/v2\/categories?post=8876"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.7boats.com\/academy\/wp-json\/wp\/v2\/tags?post=8876"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}