labs.rahulsheelavantar.in — fusedata.mdx
home.md projects/ × tools/ devlog/ articles/ now.md about.md
// project

# FuseData

Data automation platform — workflow orchestration at Noesys

live React ·TypeScript ·.NET Core ·gRPC ·DuckDB ·Apache Arrow ·BigQuery ·Athena ·Databricks ·Snowflake ·Docker ·AI Agents ·ML.NET ·React Flow ·AG Grid ·CodeMirror ·ECharts

FuseData is a data automation and engineering platform I help build at Noesys Software. The public product site at fusedata.cloud is in stealth while general availability is prepared; workflow capabilities are documented under Infoveave Automation v8.

What a data workflow is

A data workflow is a series of steps that define how data is collected, processed, transformed, and analysed. Automating those steps keeps pipelines organised, saves time, and improves accuracy — the same framing as Introducing Workflows in the help centre.

Workflow automation

Infoveave’s Automation module orchestrates connected Activities — extract, transform, validate, report generation, alerts — in a visual, no-code/low-code designer. Workflows can run on a schedule, on events, or manually, and integrate spreadsheets, databases, cloud storage, and third-party applications.

A sibling capability, Web Automation, simulates operational user actions in web apps (login, navigation, data entry, extraction) with validation and transformation against business rules. Workflow automation focuses on data-centric orchestration; web automation covers human-like UI interactions. Both sit on the same platform.

Activity ecosystem

Activities are the building blocks of automation (Introducing activities):

  • ETL — SQL, file operations, database ops, transformations, REST/API calls
  • Applications — connectors for Salesforce, Braintree, Dynamics, HubSpot, Shopify, Genesys, Mailgun, and more
  • Data transformation — aggregation, formatting, geospatial transforms, filtering
  • Email — Gmail, Outlook, send/move/download flows
  • Flow control, RPA, Infoveave-native ops — packages, file transfer, Google services

What I build

At Noesys I work across the full lifecycle of data movement and transformation:

  • Connectors — Salesforce, Braintree, Dynamics, GraphQL, OData; incremental ingestion, schema-aware extraction, resilient retries
  • Transformation engine — mapping rules, schema normalization, JSON flattening, deduplication, validation
  • Performance — DuckDB + Apache Arrow for columnar batch/micro-batch processing; warehouse integrations (BigQuery, Athena, Databricks, Snowflake)
  • Orchestration UI — React + TypeScript workflow designer; CodeMirror, AG Grid, React Flow; ECharts for analytics views
  • Platform core — .NET (gRPC/REST), SQLite for transient state, structured logging, metrics, CI/CD
  • AI & extensibility — AI Agents and GenAI-driven automation paths; dynamic workflow execution (reflection/DI) so tasks can ship via assemblies without core changes; ML.NET pipelines for ML-ready preparation
  • Observability — execution metadata, error routing, full data lineage; production safeguards on connectors

Resume highlights from this work: ~60% build-time reduction and ~90% project size reduction on deployment paths; ~40% memory footprint reduction using Apache Arrow on large workloads; led integration of DuckDB transformations into the workflow engine.

## related articles

// EOF fusedata.mdx
main
fusedata.mdx
UTF-8
LF
Markdown
Ln 1, Col 1