Kortaga's crawler

If you run a website, this page tells you whether we visit it, what we take, and how to stop us.

We do not crawl your site. Kortaga's index holds only pages we publish ourselves. When it grows past them, it will read public web archives that other people have already collected. Either way, your server does not receive requests from us.

Where the pages come from

Our own sites, today. Pages we publish on luyml.com and kortaga.com.
Public archives, next, principally Common Crawl, which crawls the web openly and publishes the result. We will read their archive rather than knock on your door a second time for a page somebody already fetched.

Reading an archive instead of crawling is a deliberate choice. It costs your server nothing at all, which is a better answer than crawling politely.

If we ever do crawl

It would identify itself as KortagaBot with a link to this page, read robots.txt first and obey it including Crawl-delay, take one request at a time, and never sign in or fill a form. This page would say so before the first request rather than after. Right now it does not apply, because we do not crawl.

Keeping us out anyway

Nothing visits you today, so there is nothing to block yet. If we ever crawl, this keeps us out:

User-agent: KortagaBot
Disallow: /

Or narrow it to the parts you want left alone:

User-agent: KortagaBot
Disallow: /private/
Crawl-delay: 10

Something wrong?

Write to [email protected] with your domain. If a page of yours ever reaches our index and you want it out, say so and it goes. If you are seeing traffic claiming to be us, tell us, because it is not.