How I Built a Dubai Business Directory on 1,400 SearchApi Requests

I live in Dubai. When I want a plumber, a nursery, or a lab that runs one specific blood test, the search experience is worse than it should be for a city this size. Google Maps knows all of it. There is just no way to browse it, filter it, or ask a structural question of it.

That gap is the idea, and it was mine before it was a project. I wanted something that could answer questions about a city rather than about one business at a time.

So I built it. Directory from Scratch is an MIT-licensed toolkit that turns a city into a browsable business directory, and directory.pooyagolchian.com is a live Dubai deployment standing on 1,400 requests to SearchApi's Google Maps engine. I am writing it up here as part of their developer ambassador programme, which backs developers who ship something real on their APIs and leaves the work in your name.

Most articles about scraping Google Maps stop at one API call and a console.log. That is the part that already works. Everything expensive happens after it.

Why build against a city you already know?

Directory from Scratch had a real target from day one. Not "build a directory" in the abstract, but this city, where I could check the output against a place I actually know. When the crawler came back with a couple of hundred nurseries, I could tell whether that smelled right.

That turned out to matter more than any architectural decision. Every headline defect in this repository was found by reading real output while the test suite sat green.

What is a business directory actually for?

Four things, and only one of them is the website.

The obvious one is the browsable surface. One crawl produced 14,981 business pages, 782 neighbourhood-by-category landing pages, 81 category pages and 40 neighbourhood hubs. That is 15,887 URLs in the sitemap, generated from a single dataset.

The second is sales prospecting, and this is the one that pays. The same crawl that produces a public directory also tells you which businesses have no website, which have never published opening hours, which are rated badly despite real trade, and which have almost no reviews. Those are four different pitches to four different buyers.

The third is market research. The corpus answers questions a consultant charges for. Which of 40 neighbourhoods concentrate which of 81 categories. What the median review count looks like at each rating band. How many businesses in a city of this size hold an exact five stars.

The fourth is boring and useful. Reverse phone lookup. Numbers get normalised to E.164 at load time, so the live search matches a +971 number however someone types it.

Why use SearchApi instead of scraping Google Maps?

A single Google Maps query cannot enumerate a city. It stops at roughly 200 results, and page 11 does not come back as an empty list, it comes back with no local_results key at all. I committed that probe to the repository as a test fixture so nobody has to spend a credit rediscovering it.

That one measurement decides the entire architecture. Tiling is mandatory, not an optimisation. Dubai is 44 geographic squares crossed with 40 categories, and the only dial that moves the bill is how deep each pair is allowed to paginate.

SearchApi fits that shape better than the alternatives for reasons that are structural rather than promotional.

SearchApi takes Google's own location format straight through, as ll=@lat,lng,zoom. Tiling a city means expressing a coordinate and a zoom level, which is exactly the parameter I already had, so no geocoding service sits in the middle.

One SearchApi request costs one credit. Because the unit of billing is the unit of work, the whole cost model collapses into a 3x3 lookup table, and pnpm plan prints the invoice before anything is spent. Budget guards are only possible because the arithmetic stays that simple.

The response carries fields that replace other services. Roughly 99.8% of results arrive with country_code and city, which drives the in-city filter with no geocoder. Every single result carries gps_coordinates, which is how the pipeline reassigns a business to the tile it actually sits in rather than the one whose query found it. About half of them sat somewhere other than where I looked, because Google answers from a radius.

place_id arrives on 100% of results and stays stable. It is the dedup key, the database partition key, and the only value the takedown suppression list is allowed to hold.

Failure is predictable enough that the retry logic is two lines. Retry on 429 and on anything at 500 or above. A 400 throws immediately, which is correct, because a malformed query does not improve on the fourth attempt.

The last one I did not expect. google_autocomplete costs one credit per category and returns suggestions ordered by real query popularity. That turns "which pages should I build" from a guess into a measurement.

How do you crawl an entire city, step by step?

Ten commands. The required spine is five of them, and only one spends money on the corpus.

Plan the crawl. Costs nothing, issues nothing, writes nothing.

It prints 1,250 first-page jobs against a 3,170 worst case and a 2,000 default budget, plus a warning that full depth overruns by 1,170. Of 1,760 possible tile-and-category pairs, 510 are dropped before a single credit is committed, because crawling law firms in the desert spends money to find nothing.

Crawl. The only irreversible spend in the project, so it refuses to run without --yes.

Four guards stack here. The mandatory flag, the free dry run, a hard budget stop, and the plan-time drops. A page is only bought if the previous page came back full and at least 30% of it was new.

Every raw response gets written to disk before anything parses it. That habit is the single best decision in the project, and I will come back to why.

Seed the taxonomy. No model call, no network, no credits.

132 ordered keyword rules classify the head of the category vocabulary.

Classify the tail. This spends Anthropic tokens and never SearchApi credits.

Here is the idea worth stealing. The obvious approach sends every business to a model, which scales with your dataset forever. But category vocabulary saturates while business count does not. My 15,246 businesses contained only 1,787 distinct category strings, which is 8.5 times fewer items. The keyword rules handle the head, so the model only ever sees a tail of 537 strings, once, across every crawl this project will ever run. A re-crawl that introduces no new strings prints a marginal cost of zero and exits.

Load. Dedupe, drop suppressed listings, assign each business to its nearest tile by coordinates, then normalise.

Five acceptance gates print on every run. At least 10,000 unique businesses, 90% phone coverage, 99% taxonomy coverage, unique slugs, and zero rows outside the country. My crawl passes at 92.2% phone coverage and 99.5% taxonomy coverage.

Two of those thresholds started higher and came down on evidence. I wanted 95% phone coverage. A probe of the detail endpoint across 15 businesses recovered exactly one extra number, and it was an Indian number correctly rejected. So 92.2% is a ceiling, not a shortfall, and the gate now says so.

Then the optional passes. pnpm demand asks autocomplete what people actually search. pnpm leads scores prospects. pnpm export writes CSV, JSON or NDJSON for a CRM.

Why archive every raw response before parsing it?

Because every response is stored untouched before parsing, everything downstream is free to re-run.

No network, no credits, full rebuild. I changed the taxonomy, the ranking constants and the normalisation repeatedly over weeks, all against one paid crawl. The 573-test suite runs in 3.5 seconds against committed fixtures and spends nothing.

That archive also produced the ugliest moment in the project. A single unresolvable path in the data loader made Next's file tracer give up and wildcard the directory, which put 1,400 raw crawl files carrying 20,226 verbatim Google review snippets inside the deployment bundle. Some of them named individual employees. Nothing read them and all of them would have shipped.

The related bug is worse and more interesting. Review theme extraction rewards terms frequent for one business and rare everywhere else, which is precisely the shape of a staff member's name. The first live run produced a Sofitel listing tagged with three employees' first names. A blocklist would have been endless. The fix is a property instead, since a real theme recurs across many businesses while a person's name belongs to one.

What does the live demo actually do?

directory.pooyagolchian.com is live and prerendered on CloudFront.

Search works. A query for dentist returns 512 matches. Typing a +971 number returns the business that owns it. Typeahead answers from a JSON endpoint in one round trip. Business pages carry LocalBusiness structured data, a canonical, both phone formats, and an honest label saying the rating came from Google rather than from me.

The homepage renders 14,981 businesses across 81 categories and 40 neighbourhoods, with 92% carrying a phone number.

It also carries a chart I did not expect to build. Median review count climbs with rating exactly as you would hope, running 12, 18, 48, 93, 139, and then collapses to 11 at exactly 5.0. There are 2,283 Dubai businesses holding a perfect score, and the typical one has eleven reviews behind it. A perfect score is the bottom of the evidence, not the top of the scale, which is why listings rank on a credibility-weighted mean instead of a raw average.

Some things genuinely do not exist yet. There is no pagination, so large categories cap at 120 rendered rows and disclose it in plain text. Sorting filters what is already on the page rather than querying the server. The DynamoDB path is written and nothing reads it. I would rather say that than let you find out by clicking.

How does a directory become a lead generation engine?

This is where a directory stops being a content project.

One crawl produces two products. The 1,400 requests that built a public website also built four prospect lists, offline, at zero marginal cost. That is one acquisition cost against two outputs, and it is the only version of the growth-engine claim I am willing to defend.

The scoring took two attempts. My first version ranked leads by the same credibility-weighted rating the directory sorts by, which sounds sensible and inverts itself in practice. Across 641 real leads the score correlated with review count at negative 0.28, so the more trade a business had, the further down my call sheet it went. The list was topped by bank ATMs. A 3.6-rated hospital with 5,562 reviews sat at number 522 of 641.

The fix generalises into one rule worth stealing. The health term must never be a function of the quantity the signal measures. Score on establishment instead, and the correlation flips to positive 0.08.

What comes out is a callable list. On my Dubai corpus, 3,820 reachable prospects have no website and 892 have never published hours. Compose the filters and you get 321 restaurants with 20 or more reviews and no website, which is a web agency's afternoon.

Two caveats I will not bury. Those are reachable prospects, meaning a phone number is required, so they run smaller than the raw count of businesses with the gap. And a lead list is research, not permission to contact anyone. The CLI prints that on every run.

Why is categorisation the real Yellow Pages problem?

The reason this reaches directory scale rather than staying a scrape is the taxonomy pass, and it costs almost nothing.

Classify the vocabulary instead of the businesses and you put 8.5 times fewer items in front of a model. Handle the head with deterministic rules and the model never sees 99.5% of your corpus. What you get back is a two-level browse taxonomy that supports 782 neighbourhood-by-category pages, and it is permanent, because the committed map always wins over both the rules and the model. The next crawl re-applies it for free. So does the next city.

Thin pages stay out of it. A category needs at least three businesses in a neighbourhood before that page exists at all, which keeps the surface large without turning it into spam.

How do you port the crawl to another city?

The crawl is genuinely city-agnostic. Tiles, categories, bounding boxes, country code and phone region all live in one JSON file, and adding a city requires no code change.

That reads OpenStreetMap and writes the config at zero SearchApi cost. It refuses to fabricate, so a city with too few candidate centres throws rather than falling back to an even grid, and every generated config is stamped unverified until somebody actually crawls it. Verification stores evidence rather than a boolean, because a bare verified: true invites someone to flip it and producing a false one costs a real crawl.

The web layer still says Dubai in a few places. Porting the crawl is a data file. Porting the site is still a small patch, and I would rather write that down than imply otherwise.

What I am not claiming

The toolkit has been proven on exactly one city, and that run stopped after 18 of the 44 configured tiles, so even Dubai is not finished. I have taken no latency, Core Web Vitals, ranking or traffic numbers, so I am quoting none. The repository ships the pipeline and never the corpus, which means both the raw archive and the built dataset are git-ignored and CI fails the build if either gets committed. Running this requires your own key and your own crawl.

That last constraint is deliberate, and it is the part I find most interesting commercially. A toolkit that does nothing without a SearchApi key means every serious user opens an account. Shipping a hosted API or a paid dataset would have inverted that, because then I would hold the only key that mattered.

The code is on GitHub. Take the ranking, the phone parsing, or the taxonomy pass on its own if that is all you need, since packages/core is pure and has no idea what a city is.