Challenge

CIAL Dun & Bradstreet runs hundreds of data scrapers that pull bulk commercial data into the database underpinning their credit and risk products. Before Silk, monitoring those scrapers meant opening each one individually — no aggregated view, no shared health indicators, no way to see the system’s state at a glance.

Two user types were operating in the same undifferentiated tooling. Developers needed precision controls: logs, run schedules, failure states, the ability to intervene at process level. Analysts needed health overviews: is the data coming in, where are the gaps, what needs attention. The existing tooling forced one to do the other’s job. Neither could work efficiently.

‍

The Architecture

Silk’s design problem wasn’t a features problem. It was a cognitive load problem. One team, one system, two completely different relationships with the same data.

The decision was to stop trying to serve both in one view — and build two interfaces over one truth state instead. One data layer. Two surfaces, each sized to a different job.

Every screen in Silk maps to one of two modes: developer depth or analyst overview. The role-based permission model isn’t a security feature — it’s the structural expression of that decision. What each user sees is determined by what their job actually requires them to do with the data.

‍

Discovery

I ran discovery sessions with developers and data analysts who worked with the scrapers daily. Three things became clear.

Monitoring was manual and sequential — users examined scrapers one at a time to check for failures. There was no way to see system health without doing the work of checking.

The team had no shared language for process status. Developers described failure states in technical terms analysts couldn’t act on. Analysts described data gaps developers couldn’t locate without log access.

Both user types needed flexibility — but in opposite directions. Developers wanted more control and detail. Analysts wanted less noise and faster orientation.

‍

The System

Search and filter

Scrapers are locatable across multiple parameters: source type, geographic scope, status. The search surface is the entry point; its job is to get users to the right process without scrolling through unrelated data.

‍

Two views, one data layer

The Metric View gives analysts a consolidated snapshot of process health across the full system. The Table and Grid Views give developers sortable, filterable detail — status, run history, failure counts — without the summary layer getting in the way. The decision to build two modes rather than one unified view came directly from discovery: a single view would have served neither job.

‍

Developer depth

The Job Details and Logs panel is where the architectural decision is most visible. Real replica counts, real run schedules, real stack-trace output. Developers can see exactly what ran, when, and why it failed — without an analyst-facing summary layer obscuring the signal. This is the screen that makes the two-interface decision defensible.

‍

Testing

I ran usability sessions with internal developers and analysts. Three changes came directly from that feedback: colour coding and alert states adjusted for clearer scraper status visibility at a glance; navigation between views simplified with icons and a sticky menu; tooltips added so analysts could interpret technical metrics without needing developer context.

‍

Result

After Silk shipped, the developer team’s feedback was consistent: monitoring was faster. Scraper-by-scraper manual checking was replaced by one view across all processes. Issues that previously required individual investigation surfaced in the dashboard before anyone had to go looking.

Analysts got health visibility without navigating developer tooling. Developers got precision controls without a summary layer in the way. The overlap that was forcing both to work inefficiently disappeared.