Optimising for search engines

Search engine optimisation (SEO) is the process of ensuring that your website is properly indexed by search engines and consequently that it has the best chance of ranking well on relevant keywords. The goal is to have your site listed high in search results so that it is seen by more people, thus driving more traffic to your site.

It is beyond the scope of this site to go deep into SEO strategies, but there are a few basic principles that will help your site to play nicely with search engines. Many of these are covered elsewhere on this site as they also improve the user experience for your visitors which should always be your primary goal when building a website.


Menu & breadcrumbs

Use a good, clear menu structure to help visitors and search engines to find their way around your site easily. In addition, breadcrumbs can help visitors to quickly go to the section's parent or root page.

Broken links

It should go without saying that all links on your site should work. Broken links cause frustration for visitors and block their access to information, and hinder search engines from crawling all the pages in your site.

A tool we use to check for broken links is Xenu Link Sleuth.

Descriptive link text

You might be tempted to use 'click here' links or simply display the URL. However, it is far more helpful for both visitors and search engines to use link text which describes where the link is going.

Meta tags

Use the description meta tag as it gives search engines a summary of what the page is about, and the summary might be shown to potential visitors in the search results.

Page titles

Use unique, descriptive page titles (via <title> tags) to tell both visitors and search engines the topic of the page. Page titles are shown prominently in search results, so using an appropriate title will make it easier for people to know whether your page will contain the information they are looking for.


Use heading tags to show how the content on your page is structured, and the hierarchy and relative importance.


Use alt attributes to describe any images which contain content. This will help screen reader users as well as search engines to understand the non-text content of your site.


Use descriptive URLs rather than generic page names, and structure your content into directories which reflect the hierarchy of your site. It is also best practice to gather similar types of files, such as images, together in the same directory.

Restricting crawler access with robots.txt

If you have content that you do not want indexed, use a robots.txt file to tell search engines (including Funnelback, the ANU search engine) which parts of your site they should not crawl. While this can be particularly useful for development sites, be aware that robots.txt can be ignored by disreputable bots, so you should not rely on it in place of password protection.

ANU only sites

If your site is already restricted to ANU only (or similar) then search engines external to ANU (such as Google) will not have access to your pages.

Restricting access to all crawler bots

If you do not want any robots to visit your site (including Funnelback):

  1. Create a file called robots.txt
  2. Insert the following text:
    	User agent: *
    	Disallow: /
  3. Save the file at the top (root) level of your website

Allowing access only to the ANU search engine

If you only want the ANU search engine to index your pages, and not other search engines:

  1. Create a file called robots.txt
  2. Insert the following text:
    	User-agent: anu-gsa-prodprimary-crawler
    	User-agent: *
    	Disallow: /
  3. Save the file at the top (root) level of your website

In this example the ANU search engine (User-agent: anu-gsa-prodprimary-crawler) is allowed to search your whole site (because there are no values after Disallow). All search engines (User-agent: *) are not allowed to search anything on your site from the root folder downwards (Disallow: /).

Further information

There are more options for disallowing search engines such as specifying particular search engines, or only disallowing certain folders. There are a number of tutorials on this topic available on the web.

For example:

Updated:  7 June 2021/ Responsible Officer:  Director Marketing/ Page Contact:  Webstyle