Skip to content
Definition

User agent

A user agent is the identifier a client sends in the User-Agent header to describe itself, such as a browser or a search engine crawler. Servers and firewalls use it to tailor responses, and misreading it can accidentally block legitimate crawlers.

Related terms

robots.txt

A file telling crawlers which parts of a site they may or may not crawl.

HTTP status code

A three-digit code a server returns to summarize how a request went.

Crawl budget

How many of a site's URLs a search engine will crawl in a given period.

← All terms