A
Crawler access
robots.txt directives per agent, a maintained llms.txt, firewall and bot-management rules that block fetchers by accident, and rate limits that make a crawl time out before it completes.
Technical GEO
An answer engine cannot cite a page it cannot fetch.
Most GEO problems are not writing problems. They are access, markup, and rendering problems, sitting one layer underneath. We find them and we fix them.
crawl-reportspecimen
robots.txt reachablePass
GPTBot allowedFail
ClaudeBot allowedFail
PerplexityBot allowedPass
/llms.txt presentFail
Server-rendered HTMLPartial
schema:OrganizationPass
schema:FAQPageFail
Canonical consistencyPartial
9 checks shown of 140. Illustrative output, not a live scan.
01 / Access
Who is allowed to read you
Every answer engine reaches you through a named agent, and each one obeys a different directive. Blocking the wrong line removes you from a corpus for months, because a model trained without your pages does not learn about them later. Most of the blocks we find were never decided. They were inherited from a robots.txt someone copied in 2021.
| Agent | Purpose | Status | Finding |
|---|---|---|---|
| GPTBot | OpenAI corpus collection | Fail | Disallowed in robots.txt. No pages available to the index. |
| OAI-SearchBot | ChatGPT search surface | Fail | Caught by the same wildcard rule as GPTBot. |
| ChatGPT-User | Live fetch on user request | Pass | Reachable. Returns full markup. |
| ClaudeBot | Anthropic corpus collection | Fail | Disallowed. Common copy-paste from an old robots template. |
| PerplexityBot | Perplexity index | Pass | Reachable. |
| Google-Extended | Gemini grounding control | Partial | Allowed, but AI Overviews still exclude the docs subdomain. |
| Googlebot | Search and AI Overviews | Pass | Reachable. Render budget is the constraint, not access. |
| Bingbot | Bing and Copilot | Pass | Reachable. |
| Applebot-Extended | Apple Intelligence | Partial | No directive present. Defaults vary by surface. |
| Amazonbot | Alexa answers | Pass | Reachable. |
| CCBot | Common Crawl, upstream of many corpora | Fail | Disallowed. Removes you from a widely reused dataset. |
Status is read from live directives, not assumed. Pass, partial, and fail are labelled in text as well as colour, so the table survives print and colour-blind reading.
02 / Retrieval
Whether the page survives parsing
Access only gets the fetcher to the door. Retrieval readiness scores what happens next: is the answer in the served HTML or injected by script, is the claim inside a heading a model can anchor to, does the page state its own facts or defer them to a component that never renders for a bot.
Retrieval readiness, 0 to 100, by surface. Illustrative figures. The pattern is the ordinary one: the pages marketing owns score well, and the pages that answer purchase questions score worst.
03 / Scope
Four things we touch
We do not write your content and we do not run your ads. The boundary is deliberate.
B
Structured data
Organization, Product, FAQPage, HowTo, and Article markup that validates and matches what the page actually says. Markup that contradicts the visible text is worse than none.
C
Retrieval architecture
Server-rendered answers, heading structure a model can anchor to, canonical discipline, internal linking that carries topical relationships, and pagination that does not hide the useful part.
D
Entity consistency
One spelling of your name, one description, one set of facts, repeated identically across your site and the third-party sources models corroborate against.
04 / Method
Crawl, diff, patch, re-crawl
Crawl
We fetch your site as each agent, from a clean IP, and record what every one of them actually receives. Not what your CDN reports.
Diff
We compare served HTML against rendered DOM. The gap between them is where most retrieval failures live.
Patch
Findings arrive as a ranked ticket list with the exact directive, template, or schema block to change. Written for your engineers, not for a slide.
Re-crawl
We re-run the crawl after deployment and confirm each fix landed. Anything still failing goes back on the list at no charge.
05 / Spec
Terms of engagement
- Audit window
- Two weeks from access to report.
- Agents tracked
- Eleven as standard. More on request.
- Checks per audit
- Around 140, run per template rather than per URL.
- Deliverable
- Ranked ticket list, directive diffs, and a re-crawl after your deploy.
- Access needed
- Read access to robots.txt, a staging URL, and one engineer for questions.
- What we avoid
- Content production, link buying, and any promise about ranking position.
- Fees
- Fixed per audit, quoted after a first call.
We report what a crawl shows. Where a finding is inference rather than measurement, the report says so in the finding itself.
Find out which engines are already locked out.
Two weeks, fixed fee, no content work. If the crawl comes back clean, the report says clean and the invoice is smaller than the retainer you were about to sign.
shaban@inkinex.com