We all have been hearing about Global Warming and it’s consequences. The studies tell us that the effects of it will soon be experienced by us. In my place (Kerala), I say it’s already started. Millions of Trees are going down each day and people push out harmful Carbon compunds into the atmosphere without thinking. On a cruel world like this, I feel really sad.

Kerala faces heavy summer and heavy monsoon. On summer days, rain is very often. But, now in the year 2014, it rains like monsoon in the middle of summer. Throughout Kerala, it’s raining heavily. The cost of the damage is estimated to be 70 crore rupees and may increase more.

Crawlers are everywhere. They move on and on to many webpages each second. The most biggest of them are Google’s. It already crawled almost 90% of the web and is still crawling. If you’re like me and want to create a more advanced crawler with options and features, this post will help you.

When I created my Search Engine test project, it needed an awesome crawler. I knew that developing one on my own will be hard especially when you have to follow robots.txt and other rules. So, I googled and found this great library which is 10 years old !

In a post, I told how to create a localhost website using Apache server in Ubuntu. It worked on all versions up to Ubuntu’s 10.10th version. But on Ubuntu versions 11.04 and the versions released after it, there were some changes to the Apache server.

So, by this post I’m going to update the way to create a localhost hosted website in Ubuntu versions 11.04 and the versions that came after 11.04 or 2011 April. Some of them are :

In the recent post, I gave the default TTL value of different devices. As you can see, when you ping a host you can roughly identify the operating system the remote server’s running. I saw a question on Stack Overflow about this. Sadly it was down voted and was removed later. I tested if it is possible and I found that it is possible.

TTL (Time To Live) is a timer value included in packets sent over networks that tells the recipient how long to hold or use the packet before discarding and expiring the data (packet). TTL values are different for different Operating Systems. So, you can determine the OS based on the TTL value. You can get the TTL value by pinging an address. Here is the output got by pinging “subinsb.com” on my system :

Inkscape is a beautiful Open Source Image editing software. It can be used to create PNG images, JPG images and even SVG images. But, when you export an SVG image, it will have some kind of transform style in it such as translate. This will make the SVG image code so long. To make it short, it’s better to remove the transform on the image and save it as it should be.

In PHP, you can directly remove an item from an array using unset function. But there is no unset function in JavaScript. But there is another function that will remove a variable. It’s called delete. Removing variables isn’t the only thing it does. It can also remove items from a JSON array in JavaScript.

It’s usage is simple. Just pass the array variable with the name of the key. Like this :

If you google something that has spelling mistakes, Google will automatically correct it for you or if the bot isn’t sure, it will give the Did You Mean ? clause. If you’re developing a search engine for your site, this spell check will come in handy. If the user had made a mistake and he was shown a “No Results Found” page, user will be annoyed. So, it’s better to have a spell check for your site.

This is the Part 3 of “How To Create A Search Engine In PHP”. In this part, We make additional adjustments and make the search engine more awesome and cool.

Bot Page

Our crawler leaves a mark on every pages it visit. In other words a custom User Agent String goes with each requests to web pages. In the previous part, we set the user agent string as :

DingoBot (http://search.subinsb.com/about/bot.php)

We add a URL with it, because if the site owner sees this in their stats, they would think “what’s that ?”. So, to answer that question we add our bot info link. Also to promote our site.

This is the Part 2 of “How To Create A Search Engine In PHP”. In this part, We’ll add code to files that are being displayed to the users. We make the database and the table in this part.

index.php

The main page of our Web Search is index.php. It has a simple design and is not very advanced as Yahoo or something else.

<?include("inc/functions.php");?>
<html>
 <head>
  <?head("", array("index"));?>
 </head>
 <body>
  <?headerElem();?>
  <div class="container">
   <center>
    <h1>Web Search</h1>
    <form class="searchForm" action="search.php" method="GET">
     <input type="text" autocomplete="off" name="q" id="query"/>
     <div>
      <button>
       <svg class='shape-search' viewBox="0 0 100 100" class='shape-search'><use xlink:href='#shape-search'></use></svg>
      </button>
     </div>
     <p>Free, Open Source & Anonymous</p>
    </form>
   </center>
 </div>
 <?footer();?>
 </body>
</html>

We use simple functions for every files. head() function accepts two parameters. The first parameter is the title of the web page. The main site title will automatically be appended to the title you give in the first parameter when it’s displayed as .