01 / The context
The problem worth solving.
Building reliable analytics workflows often requires stitching together data extraction scripts, spreadsheet-based transformations, scheduled jobs, and separate visualization tools. These disconnected processes make transformations difficult to reuse, pipeline failures harder to investigate, and data access more difficult to govern across organizations. IntroZen BI explores a unified approach in which ingestion, transformation, execution history, and analytics share a consistent workflow.
The product
Designed IntroZen BI as a multi-tenant analytics platform that connects supported data sources to declarative transformation pipelines and materialized datasets. The application brings pipeline configuration, execution monitoring, dataset exploration, interactive dashboards, reporting, workspace management, and platform administration into one product. The backend architecture is prepared separately, while the deployed public demo uses a seeded in-browser API adapter to reproduce product workflows without requiring a live backend connection or persistent storage.
02 / My contribution
Where I was involved.
Product Engineer responsible for the product concept, engineering foundation, architecture, frontend, backend implementation, data-processing workflows, analytics experience, and operational design. Established the core application before incorporating AI coding agents into the workflow, then used AI-assisted development to accelerate implementation, refine existing systems, resolve defects, improve the interface, and complete additional product workflows. The public demo showcases the frontend independently of the prepared backend.
03 / System approach
How the pieces fit.
IntroZen BI is organized into distinct presentation, interface, domain, execution, and persistence layers. The frontend uses Next.js 15, React 19, and TypeScript for the product interface, chart engine, and data exploration workflows. The prepared backend uses Django REST Framework for API handling and Django Channels for WebSocket communication, with workspace resolution, validation, and authorization at the interface boundary. Celery workers use Redis for asynchronous execution and process data with Pandas and PyArrow before materializing Parquet datasets. PostgreSQL or SQLite supports application metadata, while configurable object storage handles raw files. Docker Compose defines the containerized runtime, and Prometheus and Grafana support operational observability. The deployed demo uses a separate in-memory API adapter with seeded fixtures; it does not currently connect to the backend runtime.
04 / Engineering details
Inside the implementation.
01Separating the Demo from the Backend
- Requirement
- A portfolio demo needs to remain accessible without requiring visitors to configure an account or depend on a separately running backend and its infrastructure.
- Implementation
- Implemented a browser-based demo API adapter that follows the application's API contract, supplies seeded workspace data, and supports interactive state changes for data sources, pipelines, executions, and analytics workflows.
- Result
- Created an account-free product demonstration that can be deployed independently on the frontend hosting platform. Demo state resets on reload, while the separately prepared backend architecture remains available for future integration.
02Declarative ETL & Asynchronous Execution
- Requirement
- Data transformation can be resource-intensive, and arbitrary processing logic is difficult to validate, maintain, and govern in a multi-tenant application.
- Implementation
- Designed pipelines around a constrained vocabulary of declarative transformation steps and prepared an asynchronous execution architecture using Celery and Redis, separating pipeline definitions from background processing.
- Result
- Established a reusable processing model with explicit execution states, transformation definitions, and a foundation for handling long-running data workflows independently of application requests.
03Dataset Materialization & Efficient Reads
- Requirement
- Repeatedly reading raw files or calling upstream APIs for every chart can introduce unnecessary processing overhead and make analytics dependent on source availability.
- Implementation
- Designed the processing workflow to materialize successful pipeline outputs into columnar Parquet datasets. The backend architecture supports windowed dataset reads, while the analytics layer operates on prepared data rather than rebuilding the entire ingestion workflow for every visualization.
- Result
- Established a separation between data preparation and analytical queries, creating a foundation for reusable datasets and more efficient downstream exploration.
04Reliable Analytics & Chart Aggregation
- Requirement
- Charts can communicate misleading results when categorical data is not aggregated correctly, fields are inconsistent, or a failed pipeline run makes a previously available dataset disappear.
- Implementation
- Refined the chart engine to group records by the selected dimension, apply configurable aggregations, sort results, and limit displayed categories. Improved dataset availability handling and centralized chart configuration, styling, and rendering behavior.
- Result
- Made chart configuration more consistent and aligned the analytics experience with business questions such as revenue by region, category comparisons, and time-series exploration.
05Workspace Isolation & Access Control
- Requirement
- A shared analytics platform needs to keep each organization's data, files, pipeline executions, and logs within its authorized workspace.
- Implementation
- Designed the backend around workspace-scoped data ownership, membership and role checks, tenant resolution, and object-level authorization. Authentication uses cookie-based JWT sessions, with explicit checks for execution and dataset access.
- Result
- Established a centralized authorization model for the multi-tenant backend, with workspace boundaries applied across the documented application workflows.
06Resilient Execution Telemetry
- Requirement
- Real-time WebSocket connections can fail independently of the underlying pipeline. Treating a telemetry outage as a pipeline failure can misrepresent execution status and hide useful diagnostic information.
- Implementation
- Refined the execution-monitoring workflow to distinguish transport health from pipeline state. The prepared application supports WebSocket telemetry, polling fallback, explicit connection states, cancellation workflows, and workspace-scoped execution logs.
- Result
- Established clearer failure semantics and a more resilient monitoring experience, so a telemetry connection problem does not automatically mean the underlying execution failed.
07Operational Visibility & Error Handling
- Requirement
- A data-processing platform needs consistent error responses and operational signals to help diagnose failures across API requests, background workers, and supporting services.
- Implementation
- Designed a consistent API error envelope with stable error codes and request identifiers, structured logging, dependency-outage classification, and recovery-oriented frontend error states. Prepared Prometheus metrics and Grafana dashboards for operational monitoring.
- Result
- Created a consistent approach to error reporting, troubleshooting, and infrastructure observability across the documented backend architecture.
Product capabilities
What the system brings together.
- CSV, JSON, and REST-based data ingestion workflows
- Declarative transformations for filtering, renaming, type conversion, and sorting
- Asynchronous pipeline execution architecture using Celery and Redis
- Parquet-based dataset materialization
- Execution history, status tracking, and diagnostic logs
- Searchable and sortable dataset exploration
- Interactive bar, line, pie, donut, and candlestick visualizations
- Chart aggregation, sorting, and configurable data selection
- Reusable chart components and consistent visualization styling
- Responsive dashboards and report-building workflows
- Workspace-oriented data ownership and role-based membership
- Platform control center for workspace and account administration
- Authentication and workspace-scoped authorization in the backend design
- Structured API errors and request correlation
- Resilient execution telemetry with WebSocket and polling fallback
- Prometheus metrics and Grafana-based operational monitoring
- Account-free interactive demo with seeded data and resettable state
Interface studies
A closer look.
Development interface previews. Open a view to inspect the details.