# KosoBot, Koso's web crawler | Koso

> What KosoBot is, what it reads on your site, how often, what Koso does with it, and how to block it with robots.txt.

[Skip to content](#main)

[](/)

[Services](/services)[AI OS](/ai-os)[Training](/training)[Education](/education)[Agents](/agents)[Learn](/learn)

…

[Book a call ](/book)

# KosoBot

Last updated 28 October 2026

KosoBot is Koso's web crawler. It reads public homepages to work out what a website is about: its industry, niche, language and audience. That's what powers Koso's website research tools.

## User agent

KosoBot/1.0 (+https://koso.ai/bot)

## What it reads

Your homepage and a few public files that exist for machines: robots.txt, which it checks first, sitemap.xml, which it uses to count pages rather than fetch them, and ads.txt. Sometimes it also reads your blog's index page, or an advertise or media-kit page your homepage links to.

It reads at most about 800 KB of your homepage and gives up after 15 to 20 seconds. It doesn't log in, submit forms, add things to a basket, crawl through the rest of your site or read anything behind a login.

If your homepage needs JavaScript to show its text, and your robots.txt allows KosoBot, we may load it through a rendering service instead: Jina Reader, or a headless browser we run ourselves. Jina Reader sends its own requests, so you may see those in your logs rather than KosoBot.

## How often

Rarely. A site is read once, and again only if someone looks it up later. Most of the time KosoBot doesn't visit you at all. It first looks for a copy of your homepage in Common Crawl's public archive, and if it finds one your server never sees a request from us. Only when there's no archived copy does it make one request to your homepage. If that fails too, it may use the Internet Archive's Wayback Machine copy.

## What we do with it

We turn the page into a few labels, such as "Real Estate, property listings, English, India". The text is used to work out those labels, and that includes training Koso's own classification model. We don't republish your content or sell it.

## How to block it

Add this to your robots.txt:

User-agent: KosoBot
Disallow: /

To keep it out of just one part of your site, such as your blog:

User-agent: KosoBot
Disallow: /blog/

KosoBot checks robots.txt before every visit to your site, so the change works straight away. If there's no KosoBot group, it follows your rules for `User-agent: *`. If your robots.txt says no, it stops. It doesn't retry through another service or a browser, and it never tries to get past a CAPTCHA or bot check.

Blocking KosoBot doesn't remove copies of your homepage that Common Crawl or the Wayback Machine already hold. Their crawlers follow their own rules. To have us stop using those copies too, email us (see below).

## Contact

To have your site removed from Koso's data, or to ask about KosoBot, write to aashish \[at\] koso.ai. Tell us the domain and we'll take it out.

A custom AI development agency. Bespoke agents, copilots and products engineered for teams that want a moat, not a subscription.

Ideanab Private Ltd is a company registered in England and Wales, no. [16344048](https://find-and-update.company-information.service.gov.uk/company/16344048). Registered office: 124 City Road, London, EC1V 2NX, United Kingdom.

### Company

* [Services](/services)
* [Custom AI Agency](/custom-ai-agency)
* [AI Product Development](/ai-product-development)
* [AI OS](/ai-os)
* [AI Engineering](/ai-engineering-company)
* [AI Consulting](/ai-consulting-company)
* [AI Consulting Services](/artificial-intelligence-consulting-services)
* [Training](/training)
* [Education](/education)
* [Free AI Agents](/agents)
* [Learn AI](/learn)
* [Fractional CAIO](/caio)
* [About Us](/about)
* [The Koso Deck](/deck)
* [Careers](/careers)
* [Docs for agents](/docs)

### Contact

* [Contact us](/contact)
* [Book a call](/book)
* [LinkedIn](https://www.linkedin.com/company/koso)
* [Instagram](https://www.instagram.com/okaashish)

### Legal

* [Terms of Service](/terms)
* [Privacy Policy](/privacy)
* [Cookie Policy](/cookies)
* [Acceptable Use Policy](/acceptable-use)
* [KosoBot](/bot)

© 2026 Koso, a brand of Ideanab Private Ltd. All rights reserved.