Research · build guide
Competitive Intelligence Engine
Weekly gap analyses, keyword angles, and positioning briefs from public data only.
What it does
The Competitive Intelligence Engine is a structured research-to-analysis pipeline that runs entirely on public data. You feed it competitor URLs, ad library creatives, job postings, and landing pages; it returns three artifacts on a defined weekly cadence: a gap analysis against your own positioning, keyword angle opportunities, and a creative positioning brief your PPC, SEO, and content teams can act on. No proprietary connections, no access to anyone's ad accounts.
The archive is what compounds. Every weekly capture is kept, so after a month you are not reading isolated pages — you are diffing them. New offers, messaging pivots, landing page tests, and hiring signals show up as changes against a baseline, and that is where the real intelligence lives.
You run a weekly capture pass by hand, log everything in a spreadsheet, and push the captures through a fixed set of Claude prompts. No APIs, no connectors — just browser, screenshots, and a sheet. This suits a solo marketer or small team that wants the discipline of a weekly cadence without any infrastructure.
The collection and analysis loop runs on a schedule: read-scoped web access snapshots your tracked sources, Claude diffs them against prior weeks and drafts the three artifacts, and a human approves before anything circulates. This suits a team that wants the weekly cadence guaranteed rather than dependent on someone's calendar.
Before you start
Prerequisites.
Path A
Build it with manual data extracts.
You run a weekly capture pass by hand, log everything in a spreadsheet, and push the captures through a fixed set of Claude prompts. No APIs, no connectors — just browser, screenshots, and a sheet. This suits a solo marketer or small team that wants the discipline of a weekly cadence without any infrastructure.
Build the competitor tracker sheet
Create a Google Sheet with one row per tracked source: competitor, URL, source type (landing page, ad library, job board), capture date, and notes. Add a second tab as an evidence log where every observation gets a source link and date. This sheet is the memory of the whole system.
Run the weekly capture pass
Visit each bookmarked source in order. Screenshot active ads in the Meta Ad Library and Google Ads Transparency Center, copy key page text from landing and pricing pages, and note new job postings. Log each capture in the tracker with the date. Keep the pass mechanical — analysis comes next, not during.
Run the fixed Claude prompt pack
Paste the week's captures into three saved prompts: one that produces a gap analysis against your positioning statement, one that extracts keyword angle opportunities, and one that drafts a creative positioning brief. Using the same prompts every week is what makes outputs comparable over time.
Diff against last week
Give Claude this week's captures alongside last week's summary and ask for what changed: new offers, new landing pages, pulled ads, fresh hiring signals. Changes are the highest-value output — flag them at the top of the brief.
Review, then distribute
Spot-check every claim in the brief against the logged source before anyone acts on it. Anything you cannot trace to a capture gets cut or marked unverified. Then share the brief with your PPC, SEO, and creative owners and file it with the date.
Cadence: Weekly; expect roughly 60–90 minutes per cycle once the tracker and prompt pack exist.
Path B
Integrate it with MCP connections.
The collection and analysis loop runs on a schedule: read-scoped web access snapshots your tracked sources, Claude diffs them against prior weeks and drafts the three artifacts, and a human approves before anything circulates. This suits a team that wants the weekly cadence guaranteed rather than dependent on someone's calendar.
Stand up the evidence store
Create a Google Sheet or BigQuery table that holds one row per snapshot: source URL, capture timestamp, extracted text, and a content hash for change detection. Every downstream claim will point back to a row here, so get this in place before anything else.
Connect read-scoped collection
Give Claude an MCP server for web fetch and search so it can read tracked competitor pages and public ad library listings — read access only, nothing that touches your own ad accounts. Schedule a weekly n8n, Make, or Zapier run that triggers the collection pass; a Screaming Frog crawl of tracked pages can feed the store as well.
Configure the weekly agent loop
On each run the agent fetches the tracked sources, writes snapshots to the evidence store, and diffs them against the previous week. From the diffs it drafts the gap analysis, keyword angle opportunities, and creative positioning brief, with every claim carrying a source URL and capture date. The source list itself is fixed config the agent cannot edit.
Add the approval gate
Drafts land in a Google Doc or Notion page and a summary posts to Slack or email for review. A named owner approves before the brief circulates to the team — the agent recommends, it does not publish. Requested additions to the competitor list also route through this gate.
Log every run and audit monthly
The run log records what was fetched, what changed, what was flagged, and what failed — blocked pages and stale sources surface to the owner instead of silently disappearing. Once a month, review the log and a sample of briefs to confirm claims still trace cleanly to sources.
Tools & guardrails
Run it safely.
- Public data only: the pipeline reads competitor sites, ad libraries, and job boards — it never connects to your ad accounts and never makes campaign changes
- Every claim in a brief carries a source URL and capture date in the evidence log; anything unverifiable is flagged, not published
- The tracked-source list is fixed configuration — the agent proposes additions, only a human applies them, and fetch frequency stays capped at the weekly schedule
- Briefs are drafts until a named owner approves them; nothing auto-distributes to the team, and any brief can be retracted since the evidence trail shows exactly what it was based on
Want new build guides as they ship?
Optional — every guide on this site is free either way. Leave your details and we'll email you when new automations join the library, nothing else.
Prefer it built for you?
We stand this automation up as a governed sprint on the stack you already run — scoped, built, and handed over with the guardrails configured. No new platform, no migration.