Most projects start with a form. Somebody on the team drafts a brief: list your competitors, describe the pages you need, explain how your filters should work, tell us about your catalog structure. The client spends an evening on it, answers as best they can, and half the answers come back approximate. Not because they do not know their business, but because the questions are about the web, not about their business. They are being asked to make technical judgments they were never supposed to make.
We stopped doing that. The rule we follow now is recorded directly in our process specification: questions go to the client only when the answer is something only they can know, or when the decision would meaningfully change the architecture, the scope, or the estimate. Everything else, we find ourselves first.
What “research first” actually means in practice
On a recent e-commerce project, the research phase covered seven areas before the client got our questions.
We crawled the old site: every page, the URL structure, the content blocks, and the addresses that needed to survive the migration. We mapped the current storefront: product categories, pricing, service conditions, the things the site was already saying publicly. We pulled Google Search Console data to understand which pages and queries were actually driving traffic right now. We ran keyword research to understand demand in the niche and which competitor pages were capturing it. We reviewed competitor catalogs, product cards, filter structures, and service pages to see what patterns buyers in this category already recognized. We obtained supplier data exports and checked their completeness, their internal contradictions, and where two sources overlapped. We read the official documentation for each payment and logistics integration we expected to use, deeply enough to inform a real estimate.
Each source went into a registry: what it is, what it is for, what state it is in, when it was obtained or last verified. A separate list tracked what was still missing from the client and how many times it had been requested. Nothing sits in someone’s head or inbox.
The result of all that work is not a report. It is a draft. We build the first version of the relevant specification blocks ourselves, grounded in what we found, and hand that to the client alongside a short list of targeted questions. Their job becomes: look at what we found and tell us where we are wrong. That is a fundamentally different task from filling out a blank form. Answering “yes, correct, except these two points” is faster, more accurate, and produces better input than writing from zero.
Scripts, not manual review
Almost all of this research is done with reproducible scripts that we wrote together with AI. There are more than a dozen of them: one that crawls and exports the old site structure, one that normalizes supplier product exports, one that generates filter-completeness reports, one that maps products against the new category tree, one that cross-references the new tree against old categories, one that builds the brand registry, one that drafts the redirect map for old URLs.
That last one matters for a specific reason. Google’s guidance on site migrations is explicit: map every old URL to its new destination, configure permanent redirects, and keep them active for at least a year. On a large catalog, a script drafts that map from current data, and people review it.
Every script in our set has three properties, and we do not ship one without all three. First, documentation sitting next to the script: what it does, how to run it, what comes out. Second, tests that confirm the same inputs always produce the same outputs. Third, a strict read-and-report constraint: scripts read source data and produce findings. They do not write to any database, do not modify source files, and do not make decisions. Decisions based on the output go into a separate decision log.
Results are saved as dated files. If the category tree changed between two runs, you can see exactly how and when. If we need to rerun a report against an updated supplier export a month later, that is a single command, not a week of manual reconciliation.
Crawling is done carefully. The client’s old site was treated with the same courtesy as anyone else’s: single-threaded, with pauses between requests, respecting the crawl delay in robots.txt even when it was slow. We did not bulk-download images, video, or documents. We did not attempt to access anything behind authentication or a CAPTCHA. For competitor sites, we took a limited sample of public pages, enough to understand catalog structure, card layout, and service presentation. The goal is to understand what solutions buyers already recognize in this category, not to copy content.
Candidates, not decisions
The most important discipline in this entire process is one that sounds small but matters enormously when you are working with large product datasets: nothing the script finds is treated as a decision. It is a candidate.
When a script identifies that the same product appears to exist in two supplier feeds, based on matching SKU and brand, that match is flagged as a candidate for human review. Even an exact match on SKU and brand stays a candidate and goes into a spot check. Ambiguous cases, like a duplicate SKU within a single source, cannot be merged at all without a human decision. That decision gets recorded in the log, with the reasoning.
The same principle applies to categories. The script proposes where a source category should map in the new tree. That proposal stays a proposal until it has been verified against the actual products in that category and confirmed by the client. Every report includes checksums of the input files, so it is impossible to accidentally apply a report to a different version of the data than the one it was generated from.
This is slower than letting AI consolidate everything automatically. That is the point. Errors that get automated into your data at the research stage are the hardest kind to find later, because they look like normal records. The tradeoff is deliberate: a bit more time now, and mistakes don’t hide inside the data.
We wrote about how AI agents need defined boundaries to function correctly in a previous post on rules, skills, and docs, and this research process is a direct application of that principle. The model reasons, finds, and proposes. The human decides, and the decision is recorded.
Where AI works, and where it does not
AI in this research phase acts as a fast researcher and programmer. It writes the crawl and analysis scripts. It consolidates findings into readable summaries. It surfaces contradictions between data sources. It drafts the specification blocks that the research supports.
Without AI, a research phase of this depth would not pay off at the specification stage. The volume of work is too large to do by hand: an old site audit, a full competitor review, keyword analysis, supplier data normalization, integration documentation review, and a redirect map. With AI doing the mechanical work, it does.
What AI does not do: define what to research or how deep to go. The process specification includes a rule about this. The audit is not meant to be exhaustive. It is meant to find the things that could change the scope, the architecture, or the estimate. Everything else waits for development. That judgment call, knowing when enough is enough, stays with the person running the process.
What the client receives, in the end, is not a questionnaire. It is a short list of questions where each question explains why it is being asked and what the answer will affect, alongside a proposed answer or direction that we have already thought through. And next to that list: a proposed site structure, a suggested category tree, a service and conditions summary pulled from their existing sites, a competitor analysis. Their actual task is review and correction, not creation.
That distinction is what this whole methodology is built around. The technical specification becomes a working tool read by three parties: the client, the team, and the AI agents that will execute work later. For it to function that way, the information inside it has to be grounded in real data, not in approximations from a questionnaire nobody was equipped to answer well.
For the client, this means spending time only on what they know best: their business, priorities, and decisions. The decisions in the specification rest on facts about their site and market, not on guesses. And the research does not end with the spec: the category tree, the decisions on duplicate products, and the redirect map go straight into development.
The research comes first. The questions come after. That order changes what gets built.