Why Large-Scale Process Tracking Required a Specialized System

Silk is an internal data source and process management tool designed to give our developers and data teams full oversight of data collection processes (or "scrapers") that pull bulk data into our database. These processes are crucial for generating data for company credit and risk assessments. Silk enables users to monitor, manage, and troubleshoot these processes in real time, providing both high-level and detailed views of system status. My role as the product designer was to make this complex data infrastructure accessible and actionable for both developers and data analysts.

‍

The Challenge of Balancing Depth for Developers and Ease for Analysts

The main challenge was to design a single management platform that could serve both developers, who needed advanced process controls, and data analysts, who required easy-to-understand process summaries and health indicators. The platform needed to display vast amounts of data in a digestible way and be flexible enough for different use cases and technical skill levels.Why Large-Scale Process Tracking Required a Specialized System

‍

Discovering the Decisions Behind Data Management

To tailor the platform’s design to the specific needs of Silk's users, I conducted discovery sessions with our internal developers and data team members who actively work with data scrapers. My aim was to understand their daily workflows, challenges, and needs related to data process management.

Key insights from discovery:

• Manual Process Monitoring: Users described their current monitoring process as time-consuming, requiring them to examine each scraper individually to check for failures or issues.‍
• Inconsistent Health Monitoring: The team had no unified system for quickly gauging the overall health of the processes, leading to delayed responses to issues.‍
• Need for Flexibility: Both developers and data analysts needed flexible views—developers wanted detailed logs and controls, while analysts preferred quick overviews of data health without the technical complexity.
  ‍

Personas That Guided the Design

From my research, I developed two primary personas that informed the user experience design:

‍Devon, the Developer

‍Frustrations: Lack of centralized process monitoring, manually examining scrapers for issues, complex scheduling needs.

Primary Goals: Control and manage scrapers with high precision, troubleshoot quickly, access detailed logs.

‍

‍Jamie, the Data Analyst

‍Frustrations: Navigating overly technical interfaces, limited visibility into system health, difficulty comparing multiple processes.

Primary Goals: Monitor overall scraper health, view high-level summaries, identify problem areas without needing technical controls.

‍

Solution That Made Complexity Clear and Actionable

Based on the diverse needs identified in discovery, I designed Silk around three primary features that addressed the main user requirements:

1. Powerful Scraper Search and Filter Tools

Silk provides a robust search functionality that allows users to easily locate scrapers across multiple parameters, including source type and geographic scope. This flexibility lets users pinpoint specific scrapers relevant to their current tasks or region, streamlining the initial stages of data review.

‍

2. Centralised Data Collection and Evaluation

To provide a unified overview of process health, Silk integrates two distinct views:

• Metric View: Displays high-level process health metrics across scrapers, offering users a consolidated snapshot of each scraper’s status and effectiveness.
• Table and Grid Views: These allow users to dive into the details with sortable lists that can be filtered and compared. In-depth statuses and statistics are accessible, enabling both quick assessments and detailed evaluations, while the grid view serves as a more visual way to monitor multiple scrapers at once.

These views provide essential data for both monitoring and analysis, allowing users to identify trends, detect issues, and maintain process stability with a blend of granular and holistic data perspectives.

‍

3. Decision-Making with Role-Based Permissions

To support efficient, collaborative workflows, Silk integrates role-based permissions, ensuring that users have appropriate access levels for their responsibilities. Data analysts, for example, can review metrics and reports, while developers have full access to manage and adjust scraper settings. This permissions-based structure enhances accountability and enables a seamless, secure collaboration among users with varying technical backgrounds.

Small Adjustments, Big Impact

I conducted usability testing with internal teams, focusing on developers and data analysts who directly interact with the platform. Through this iterative testing process, we received valuable feedback, leading to key refinements:

• Enhanced Data Clarity: Adjustments to color coding and alerts provided clearer visibility into the real-time status of scrapers, enabling users to immediately identify and address problem areas.

• Streamlined Navigation: Initial feedback revealed a need for simplified navigation between the table and grid views, so I implemented intuitive icons and a sticky menu, improving accessibility.

• Contextual Tooltips for Analysts: To support non-technical users, I incorporated tooltips that explain complex metrics, empowering analysts to interpret data insights effectively without extensive technical knowledge.

‍

From Daily Friction to Measurable Progress

Silk significantly improved our internal team’s efficiency by centralising data scraper management and providing actionable insights in one accessible interface. The platform minimised manual tracking and reduced errors, allowing developers to troubleshoot faster and data analysts to monitor scraper performance effortlessly. This unified system not only increased the accuracy of our data processing but also saved time, making data management less cumbersome and more collaborative.By blending intuitive design with robust data functionality, Silk transformed the team’s approach to data monitoring and provided a tool that is both powerful and easy to use.

Key Insights: Technical Depth, Human Understanding

In designing Silk, I aimed to balance functionality and usability, creating a system that could adapt to the needs of both technical and non-technical users. This project underscored the value of thorough research, iterative testing, and flexibility in UX design, resulting in a platform that empowers teams to efficiently manage and optimise their data processes. Silk has become an essential tool for our teams, demonstrating how thoughtful design can enhance even the most complex internal systems.