Login | Register | FAQ
Anonymous

robots.txt

+ Post a reply

6 posts Page 1 of 1


robots.txt

by iDana » Fri May 20, 2011 4:04 pm

Hello,

I know about robots.txt but can anyone give me any more information on it?

Thanks.
User avatar

iDana

  • Posts: 60
  • Joined: Tue Oct 26, 2010 10:05 am
--
iDana

Re: robots.txt

by iDana » Fri May 20, 2011 4:28 pm

Thank you!:)
User avatar

iDana

  • Posts: 60
  • Joined: Tue Oct 26, 2010 10:05 am
--
iDana

Re: robots.txt

by nigerianprince » Tue May 24, 2011 7:53 pm

thank you as well! I was looking for robot.txt info before but the linked site is definitely a better resource than what i was referring to before
User avatar

nigerianprince

  • Posts: 1
  • Joined: Tue May 24, 2011 7:44 pm

Re: robots.txt

by markomako79 » Fri Aug 05, 2011 4:30 am

I added a sitemap to my wordpress blog and when I ran an page analyzer it told me that robots.txt was now present. Does this term have something to do with the sitemap, or is it totally unrelated. It would be great if someone could tell me if having robots.txt benefits my website when it comes to SEO?
User avatar

markomako79

  • Posts: 11
  • Joined: Tue Aug 02, 2011 6:08 pm
--
Creating a survival seeds website, and need some basic HTML help.

Re: robots.txt

by HTML.net » Fri Aug 05, 2011 6:48 am

Robots.txt is a file you can upload to your web site to guide search bots (Google etc.) how to index your site.

For example, if you write in a file robot.txt:

User-agent: *
Disallow: /

no page will be indexed.

You can read more at the link above: http://www.robotstxt.org

- Andreas, HTML.net
User avatar

HTML.net
Site Admin

  • Posts: 1194
  • Joined: Sat Sep 11, 2010 11:24 am
--
Show some love for HTML.net on Twitter, Facebook and Google: Use the buttons on top of all pages.

Re: robots.txt

by Preeju M S » Fri Jun 09, 2017 4:22 am

Robots.txt file is a text file for restricting bots (robots, search engine crawlers ) from a website or certain pages on the website. Using a robots.txt file and with a disallow direction, we can restrict bots or search engine crawling program from websites and or from certain folders and files.
User avatar

Preeju M S

  • Posts: 6
  • Joined: Thu Jun 08, 2017 7:40 am
--
Data Entry Outsourcing Company - http://www.onlinedataentryoutsourcing.com


+ Post a reply

Page 1 of 1