It depends on what you want. For the page to stop existing, delete it and make it return 404 (or 410). For the page to stay but be out of Google, use the noindex tag. To hide something right now, urgently, use Search Console’s removals tool, but only as a temporary measure. And for something truly private, use a password. The robots.txt file is not for this.
Which one to pick
| Your situation |
What to do |
| The page no longer makes sense |
Delete it. The server then returns 404; over time Google drops it. If it has links, redirect to the closest match (see 301 redirects). |
| The page stays, but you do not want it to show (thank-you, drafts, test pages) |
Set noindex. On WordPress, the SEO plugin has the option on every page. See SEO plugins. |
| The whole site should not be on Google yet |
In Settings, Reading, tick the option asking search engines not to index. And remember to untick it when you open. |
| A page with data that should not have been public |
Take it off the site first (or put a password on it), and only then ask for removal. If you do not remove it at the source, it returns. |
| Private content (customers, files) |
A password or a sign-in area. In cPanel, the directory privacy option protects a folder. |
Setting noindex, step by step
| 1 |
Add the tag in the page’s <head>: <meta name="robots" content="noindex">. In a plugin, it is a box per page; on a hand-made site, you write it.
|
|
| 2 |
Let the robot visit the page. If robots.txt forbids it, Google never gets to read the tag, and the page may keep appearing. This is the commonest confusion. See robots.txt and sitemap.xml.
|
|
| 3 |
Take the page out of the sitemap, so you do not send opposite signals.
|
|
| 4 |
Wait for Google’s next visit. You can request a new crawl in URL Inspection. The page leaves when Google reads it again and sees the tag.
|
|
| 5 |
Check by searching site:yourcompany.com/page-address on Google.
|
|
If it is urgent: the removals tool
In Search Console, under Removals, you can ask for an address to stop appearing. It is a temporary measure: it hides, it does not delete. It must be followed by the lasting fix (404, noindex or password), or the page comes back. For something that changed but Google still shows the old version, the same section has the “outdated content” request.
|
robots.txt does not take pages out of Google. It asks the robot not to visit them, but if other pages link to it, the address can stay in the results, with no description. And the file is public: a list of “forbidden” folders is a map for the curious. For secrets, a password.
|
|
Even after it leaves, copies may remain elsewhere (archives, third-party copies, the memory of whoever shared it). If the problem is exposed personal data, see personal data protection. If Google shows an unsafe warning, that is a different case: the red Google warning.
|
|
Need to protect a folder with a password, or a rule that returns 404 or 410? Tell us the address and what you want to achieve.
Open a support ticket
|
RECOMMENDED PRODUCT Web hosting with cPanel Domain and SSL included, daily backups and the panel you already know. from $6.60/mo (3-year plan, with coupon) See plans |
Angola
Brasil
Kenya
Moçambique
Nigeria
Portugal
South Africa
United Kingdom
United States
International
