AI crawlers read your HTML, not your JavaScript
Before you measure whether an AI assistant cites you, check whether it can read you at all. The test is one curl command, and the result is often uncomfortable.
- ai-visibility
- crawling
- geo
News
Build rank tracking that reports something true: pin location and device, diff sets instead of averaging positions, and know why keyword volumes never agree.
Rank tracking looks like the easiest thing in SEO to automate. Search a keyword, find your domain, store the number, chart it. Every team that builds it discovers the same three problems in the same order: the number moves when nothing changed, the average across keywords means nothing, and two tools report different volumes for the same word.
None of those are bugs. They are properties of what is being measured, and a tracker that does not account for them produces charts that are worse than no charts — because people act on them.
This article is how to build one that reports something true. The API calls are ours; the methodology is not specific to any provider.
A ranking is not a stored value you look up. It is generated per request, influenced by at minimum:
So "we rank 4th" is shorthand for "in one sample, with one configuration, at one moment, this URL was 4th". Every design decision below follows from taking that literally.
Pin every variable you can, and hold them constant for the life of the series. Location, language, device, depth. A tracker that sends no location does not get a neutral result — it gets some location, chosen for it, possibly a different one next week. When your chart moves, you will not be able to tell whether the ranking changed or the sample did.
Two different quantities get called "position", and mixing them is a classic source of phantom movement:
They diverge, and the divergence changes when Google adds or removes a feature from the SERP — which means your "ranking" can move while your organic position is identical. Whichever you use, use one, and label your charts with which.
POST /v1/serp-diff compares the organic block only, keyed on rank_absolute within those results. That is a deliberate narrowing: it makes the series stable against SERP-feature churn, at the cost of not telling you how far down the page you actually are. If page geography is what you care about, capture the full SERP with /v1/search and compute your own.
The single worst rank-tracking metric is the average position across a keyword set. It hides everything that matters: a keyword falling off the first page entirely and another climbing two spots can leave the average unchanged, and one keyword you do not care about can swing the whole line.
Track movements as a set: what entered, what left, what moved and by how much. That is what /v1/serp-diff returns:
curl -X POST https://api.dataswap.io/v1/serp-diff \
-H "Authorization: Bearer $DATASWAP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"keyword": "postgres connection pooling",
"location_name": "London,England,United Kingdom",
"language_code": "en",
"device": "desktop",
"depth": 100
}'import { Dataswap } from 'dataswap';
const dataswap = new Dataswap({ apiKey: process.env.DATASWAP_API_KEY });
// No dedicated SDK method for this route yet, so use the escape hatch. `request<T>` types the
// response; declare the fields you actually read.
type SerpDiff = {
baseline: boolean;
captured_at: string;
previous_captured_at: string | null;
changes: {
added: { url: string }[];
removed: { url: string }[];
moved_up: { url: string }[];
moved_down: { url: string }[];
unchanged: number;
};
};
const diff = await dataswap.request<SerpDiff>('POST', '/v1/serp-diff', {
keyword: 'postgres connection pooling',
location_name: 'London,England,United Kingdom',
language_code: 'en',
device: 'desktop',
depth: 100,
});
if (diff.baseline) {
// First capture for this keyword+config. There is nothing to compare against, so every result
// is reported under `added` — branch here rather than reading the diff as real movement.
console.log('baseline stored at', diff.captured_at);
} else {
const { added, removed, moved_up, moved_down, unchanged } = diff.changes;
console.log(
`since ${diff.previous_captured_at}: +${added.length} -${removed.length} ` +
`↑${moved_up.length} ↓${moved_down.length} =${unchanged}`,
);
}The endpoint is stateful: it captures the current ranking and compares it against your previous capture for the same keyword and configuration. The first call returns baseline: true, and because there is no earlier capture to compare against, every result comes back under added — that is not a page of new entrants, it is the baseline being stored. Branch on baseline before you read changes, or your first data point will show a hundred arrivals that never arrived. previous_captured_at tells you what the comparison window actually was, which matters when a scheduled run was missed and today's "daily" diff spans three days.
current carries the full captured ranking, so you can store the raw list alongside the diff.
Sampling more often does not make the number more true, it makes it more expensive and noisier. Choose by what you would actually do with the alert:
Cost scales with keywords × frequency. The freshness parameter — cached at half price, live at 1.5× — applies to /v1/search, /v1/news and /v1/maps, not to the endpoints in this article: /v1/serp-diff always captures fresh, which is what you want when you are measuring movement. Sending freshness to it returns a 400.
Tracking keywords you chose by intuition is how most sets go stale. Two calls fix it.
What you already rank for, which is usually broader than the list you were tracking:
import os
from dataswap import Dataswap
client = Dataswap(api_key=os.environ["DATASWAP_API_KEY"])
ranked = client.request("POST", "/v1/labs/ranked-keywords", {
"target": "example.com",
"gl": "us",
"limit": 200,
})What those keywords are worth, for up to 100 at a time:
volumes = client.request("POST", "/v1/keywords", {
"keywords": ["postgres connection pooling", "pgbouncer vs pgpool"],
"gl": "us",
"hl": "en",
})Together: discover what you rank for, size it, track the subset that matters, and revisit quarterly. /v1/labs/keyword-suggestions and /v1/labs/related-keywords extend the set; /v1/labs/search-intent classifies intent for up to a thousand keywords, which is how you avoid tracking informational queries with a commercial page.
Base costs: serp-diff 2 credits, keywords 3, labs/ranked-keywords 4. A credit is $0.002, and the exact charge comes back on every response.
Search volumes are estimates, and they disagree wildly between tools. This is not a minor discrepancy. Practical Ecommerce reported the same keyword at 880,000 in Google Ads, 590,000 in Semrush and 100,000 in Ahrefs. Their measurement, not ours — and the lesson is structural: volumes come from different models over different panels. Never mix sources in one table, never compare a volume from one tool against a volume from another, and treat volume as an order of magnitude for prioritisation rather than a forecast.
The diff is keyed on URL. If a page changes URL — a redirect, a slug edit, a trailing-slash change, http to https — it appears as one removed and one added, not as a move. Before investigating a "lost ranking", check whether the URL simply changed. Normalising URLs on your side before storing helps; it does not eliminate this.
Only organic results are compared. An AI Overview appearing above you does not register as a change here, even though it may change your traffic substantially. Track that surface separately — see reading AI Overviews programmatically.
A missed run silently widens the window. If your scheduler skips a day, the next diff covers two. Read previous_captured_at rather than assuming your interval held, and label charts with real intervals.
Position is not traffic. Click-through rate varies enormously by query type and by how much of the page sits above you. Rising from 5 to 3 on a SERP that gained an AI Overview and a local pack can lose you clicks. If traffic is the goal, join rankings against Search Console rather than inferring.
Depth costs and truncates. depth: 100 captures more and costs more than depth: 10; a URL below your depth is indistinguishable from a URL that is gone. Keep depth constant across a series, or your "removed" counts are an artefact of your own configuration.
Zero-click is not measured here. Ranking first for a question answered on the results page is a ranking success and a traffic non-event. Rank data cannot see that; only your analytics can.
cached and previous_captured_at with every point.Do that and the chart means something. The docs cover the credit envelope, freshness and idempotency; the API reference carries the published schemas; and AI search visibility covers the surfaces that classic rank tracking cannot see.
Before you measure whether an AI assistant cites you, check whether it can read you at all. The test is one curl command, and the result is often uncomfortable.
Search results are the wrong shape for a prompt. How to turn a SERP into grounding context an agent can use: dedupe, rerank, budget tokens, keep citations.