A Web Crawler is a program that crawls through the sites in the Web and indexes those URL‘s. Search Engines uses a crawler to index URL’s on the Web. Google uses a crawler written in Python. There are other search engines that uses different types of crawlers.

In this post I’m going to tell you how to create a simple Web Crawler in PHP.

The codes shown here was created by me. It took me 2 days to create a simple crawler. Then How much time would it take to create a perfect crawler ? Creating a Crawler is a very hard task. It’s like creating a Robot. Let’s start building a crawler.