qGrid — Aniket Agarwal

The qGrid dashboard: a data reliability score of 76%, the top five incidents, active incidents by severity and business rule status

Problem

Existing tools buried teams in alerts, split the picture across dashboards, and hid every check behind SQL only engineers could read.

Approach

One loop, designed for four roles: define what "correct" means, detect when it isn't, trace the cause, and route it to the right team.

Outcome

In beta, qGrid onboarded the enterprise databases of two Fortune 500 companies.

The challenge

Pipelines don't fail loudly. They fail quietly, and someone finds out later

Enterprise data runs through lakes, warehouses and dozens of jobs owned by different teams. When one of them breaks, a column goes null, a load arrives half-empty or a schema changes, and nothing errors. The numbers just drift, and the dashboards built on them go on looking fine.

That's data downtime: hours or days when the business is making decisions on data that's wrong. The design problem was giving technical and business teams one shared view of data health, and a way to act on it, without either side having to learn the other's tools.

The Update Executive Reports pipeline: start time and duration charts over a week, with one failed run on July 5 among successful ones — Fig 01 A pipeline that feeds the executive reports. Most runs succeed; one on July 5 fails and its duration spikes. Nothing downstream errors, so without this view the reports would simply be wrong the next morning.

Strategy

More monitoring wasn't the answer. Less friction was

The tools teams already used produced the same three complaints: alert fatigue, dashboards split across products, and checks written as SQL that only their author could follow. Adding more alerts would only make the problem louder.

So qGrid is built as one loop instead of four separate tools: define what correct data looks like, detect when it drifts, trace where it broke, and route it to whoever can fix it. Every screen had to work for four different people.

Data engineersNeed to know which job broke and where, fast.
Analytics engineersNeed checks on the models they build, without hand-writing each one.
Data stewards & analystsKnow the business rules, but not the SQL to enforce them.
Enterprise data leadersNeed one honest answer to "can we trust this data?"

Define

Write the rule in the language you already know it in

Stewards and analysts know the rules the data has to follow: a zip code has five digits, every order has a customer. What stopped them was the SQL. Quality Studio starts them with recommendations qGrid has already matched to their tables, grouped by category, to accept or hide in one click.

For anything custom, the Rule Builder takes a plain-language request, writes the check, explains what it does, and validates it against real data before it's saved.

Quality Studio's Recommendations tab: suggested rules such as email format, US state abbreviations and zip code checks, each mapped to an asset, column and category — Fig 02 Recommended rules waiting for each domain, and the Rule Builder, where a plain-language request becomes a validated SQL check. Nobody starts from an empty editor.

The Rule Builder panel: a user asks qGrid AI in plain language for a null-value check, and the AI returns SQL to apply — Fig 02 Recommended rules waiting for each domain, and the Rule Builder, where a plain-language request becomes a validated SQL check. Nobody starts from an empty editor.

Detect

Rules cover what you know. The models watch for everything else

ML models learn each table's normal volume, freshness, schema and distribution, and open an incident when something drifts. The incident list is ranked by severity and says in one line what happened and how many tables it hit, so a team works the worst problem first, not the loudest.

From here the story follows one incident on the Warehouse.SQL.Prod table. Opening it leads with a qInsight summary: the AI's plain-English account of what was detected and how many tables, reports and people it affects, before any chart.

The Incidents list: open and investigating counts, average resolution time, and volume, freshness and distribution anomalies with status, severity and owner — Fig 03 The incident list, then one incident opened. The qInsight summary puts the impact (12 tables, 5 reports, 109 users) above the chart, so a data leader gets the answer without reading the graph.

An incident on Warehouse.SQL.Prod: a qInsight summary of impact above a time series of the anomaly, with affected pipelines listed alongside — Fig 03 The incident list, then one incident opened. The qInsight summary puts the impact (12 tables, 5 reports, 109 users) above the chart, so a data leader gets the answer without reading the graph.

Trace

Every incident opens on the path it travelled

Knowing a table is wrong is only half the job; the other half is finding out why. The RCA tab lays out the pipeline as a lineage graph, with the failing jobs and tables marked, so the break and everything downstream that inherited it are visible at once.

One level deeper, each pipeline opens into its own steps. Here the chain stops at Copy Data, and the step's runtime result gives the reason: access denied.

Root cause analysis: a lineage graph of pipelines and tables, with failing nodes in orange and healthy ones in blue — Fig 04 From the lineage graph down to the single failed step. The same incident, traced from which tables are affected to what went wrong, without leaving the page.

Pipeline orchestration for the incident: Execute Pipeline and Transformation succeed, Copy Data fails, and a side panel shows the runtime error — Fig 04 From the lineage graph down to the single failed step. The same incident, traced from which tables are affected to what went wrong, without leaving the page.

Route

An alert should arrive where the team already works

Once the cause is known, the incident needs an owner. It can be sent to Jira in one step, and qGrid keeps the two in sync: when the Jira ticket closes, the incident resolves. Notification groups decide who hears about what, by asset, pipeline or storage, and on which channel: email, Slack or Microsoft Teams.

Fewer alerts, each one carrying its own explanation, beat more alerts every time.

The Warehouse.SQL.Prod incident with a confirmation to send it to Jira, above its timeline of status and severity changes — Fig 05 The incident sent to Jira, and the connection behind it. Status mapping means a closed ticket resolves the incident in qGrid too, so nobody updates two tools.

Jira connection settings: server, credentials, default project and a mapping from Jira statuses to qGrid incident statuses — Fig 05 The incident sent to Jira, and the connection behind it. Status mapping means a closed ticket resolves the incident in qGrid too, so nobody updates two tools.

Notification groups: team groups with their members, the assets they watch, and status — Fig 06 Notification groups route each incident to the team that owns the asset, on the channel they already use. That's how fewer alerts reach the right people.

Creating a notification group: choose channels such as email, Slack and Microsoft Teams, then the data assets, pipelines or storage the group hears about — Fig 06 Notification groups route each incident to the team that owns the asset, on the channel they already use. That's how fewer alerts reach the right people.

Outcome

Proven at enterprise scale, in beta

During the beta, qGrid onboarded the enterprise databases of two major Fortune 500 companies: real production data at the scale the platform was designed for.

Fortune 500 companies onboarded during beta

Takeaway

Trust in data is a design problem, not just an engineering one

The models could already find anomalies. What people were missing was a way to understand them: which ones mattered, where they came from, and who needed to know. Following one incident from detection to a closed ticket is what shaped the product.

What I'd carry forward: design for the least technical person in the room, and the experts get a faster tool too.

Next project Amp'd — Collaborative app for sharing playlists, reviews and listening activity with friends

Catching broken data before the business does