EMON: Automating Policy Monitoring for Public Affairs Consultants

EMON is a daily policy-monitoring system for public affairs consultants that replaced manual searching of government websites with automated, 100% coverage of 18 Singapore agencies, cutting the client’s external vendor spend by 80% and saving the team around 12 hours a week.

Problem

Public affairs consultants live or die on knowing what government agencies announce: enforcement actions, public consultations, press releases, ministerial speeches. Their clients depend on catching the relevant ones early and missing none.

The team at Emergent Public Affairs did this by hand. Four consultants ran Google searches, checked government websites and news pages, and compiled reports from what they found. For markets they could not cover themselves, they paid external vendors, whose monitoring effort ran to roughly 23 people spending about a day a week each. The process was slow, labour-intensive, and easy to drop things from.

A client commissioned Emergent to build an internal tool and answer one question: can AI automate this monitoring work? The brief was deliberately exploratory, so the budget was minimal and the bet was staged. I had eight weeks to ship a proof of concept for Singapore only: a front-end dashboard, daily email digests, and crawlers for the local agencies. If it worked, 14 more regional markets would follow.

I owned the whole thing: product management, architecture, development, and QA.

Approach

The monitoring workflow has three parts: data capture, report writing, and editorial judgment. Phase one tackled only the first. The logic was simple: if capture could be automated, the team could lean off their external vendors and see savings immediately, and that early win would justify the rest.

So the job to be done was narrow and clear: surface every relevant government announcement, every day, without a human checking sites by hand, and push the high-value items to the top.

What I scoped in: scrapers for 18 Singapore agencies, a PostgreSQL store, a Next.js dashboard with text search and filtering, daily email digests, and an LLM layer that classifies each announcement by theme and type and writes a three-bullet summary.

What I deliberately left out: comments, sophisticated tagging, project management, and the entire report-writing and editorial layer. The client was explicit that editorial judgment should stay with their consultants, so automating it was never the goal. The point was the opposite: strip out the labour-intensive capture work so consultants could spend their time on the judgment that clients actually pay for. Filtering carries this through. Enforcement actions and consultations rise to the top automatically, where previously someone had to track them manually.

Key decisions and tradeoffs

Intercept internal APIs instead of parsing the DOM. Government sites vary wildly. No two agencies run the same infrastructure, and there is no consistency to parse against. Singapore’s sites are modern and JavaScript-driven, loading content dynamically from internal JSON APIs. Rather than parse rendered HTML, which breaks the moment a layout changes, the base scraper hooks Playwright’s network events and captures those JSON responses directly. That gives cleaner data and far more resilience. The cost is that each source still needs adaptation from a baseline configuration, so I kept DOM parsing as a fallback and used RSS where international sources offered it. The payoff was bounding a maintenance problem that would otherwise grow without limit as agencies were added.

Decouple processing from scraping. The scrapers are insert-only and run in their own container. A separate service polls every 30 minutes for unclassified rows and runs the theme, type, and summary passes. This was a reliability decision. A scraper failure cannot take down classification, and the expensive, slower LLM work is isolated where it can be restarted or scaled on its own. The tradeoff is more moving parts and classification that is eventually consistent rather than instant. For a product that publishes once a day, 30 minutes of latency does not matter, and keeping the scrape path simple, capture and store, nothing else, made the whole system easier to reason about.

Buy the high-volume news layer instead of building it. Custom Playwright scrapers earn their keep on structured government sources where coverage and precision matter. News is different: the volume is high and the goal is to feed it to an LLM efficiently. Firecrawl is expensive, but it handles bot-protected and JavaScript-rendered pages out of the box and removed a large slice of implementation work. The call was to spend money where it buys speed and robustness and build custom only where the data demands it. This is the same lesson I would push further with hindsight.

Architecture

Three layers, deployed as separate Railway services: Playwright scrapers feed PostgreSQL, which a Next.js dashboard reads from.

The scrapers extend a shared base class with the network interception described above and write insert-only into PostgreSQL. A standalone data-processing service polls for unclassified announcements and runs them through theme classification (12 top-level themes, 40 sub-themes), type classification, and summarisation via the Anthropic API. Results are cached, so the work happens once per announcement.

A separate lightweight service runs a daily cron for the news track, executing per-market boolean searches through Firecrawl. The leverage point here is that every search parameter lives in YAML config. Adding a market, adding a topic, or editing a query is a config change with no code touched, which is what lets the system scale to 15 markets without rewriting Python. Daily email digests push the surfaced items to the team.

The failure modes are contained by design. Scraping is isolated from classification, so an API problem cannot block capture, and a broken scraper for one agency does not affect the others. The main external dependency and cost driver is Firecrawl credits, budgeted against the plan ceiling.

Results

From a standing start in an eight-week pilot:

• 18 Singapore agencies covered, 100%, monitored daily.
• About 3,000 government actions captured into a proprietary index the client now owns.
• High-value items surfaced automatically instead of tracked by hand.
• Around 12 hours a week saved across the internal team of four.
• External vendor spend reduced by 80%.

The clearest result is what the pilot earned. Scope expanded from a handful of Singapore agencies to all 15 regional markets at roughly 20 agencies each, and the client commissioned phase two: automated report writing, with editorial control kept in-house. The proof of concept did its job.

What I’d do differently

I would build the report writing first. It is narrower in scope, it would have saved the team time sooner, and it can ship as a standalone script or skill rather than the full database-and-server stack the monitoring layer requires. Starting with the broad, infrastructure-heavy piece delayed the payoff the client most wanted to see.

I would also reach for Firecrawl and similar tools earlier. The client could comfortably cover the cost, and the out-of-the-box robustness would have compressed development considerably. Custom Playwright scrapers are still the right answer for specific government sites, but most of this was a buy, not a build.

Tech stack

Python Playwright PostgreSQL Next.js TypeScript Firecrawl Anthropic API (Claude) Railway Docker YAML