Loading
Data engineering · Warehousing · Power BI
The hard part is almost never the chart. It is agreeing what a number means, getting it from four systems that disagree, and refreshing it before the meeting where somebody has to decide something.
The problem
At twenty people, one person exports two systems into a spreadsheet on Monday and everyone trusts the result, because everyone knows who made it and how. At eighty people that same file has been copied five times, three departments maintain their own version, and two of them define revenue differently. Nobody is wrong. There is simply no agreed definition and no place where one could live.
The symptom leadership notices is that meetings start with arguing about the numbers instead of deciding what to do. The symptom the team notices is that the report takes a day and a half to build and is out of date by the time it is circulated.
Underneath it are three separate problems that get treated as one. Definitions: what counts as an order, when it counts, and how a cancellation is handled. Plumbing: getting data out of the ERP, the CRM, the shop-floor system and the spreadsheets on a schedule and in a consistent shape. Presentation: a dashboard someone opens without being reminded.
Most failed BI projects skipped the first two and bought a tool for the third. A dashboard on top of undefined, unreconciled data is a faster way to distribute a disagreement.
What is included
7 areas of work. Most engagements start with one or two and widen once the first release is in use.
An inventory of where the numbers actually come from: ERP tables, CRM exports, Tally, spreadsheets, machine logs and the file somebody maintains privately. Ownership, refresh frequency, entry quality and the specific fields that disagree between systems.
Scheduled extraction, incremental loads, change tracking, retries and alerting when a source is late or a row count looks wrong. Written as code in version control so a transformation can be reviewed and explained rather than living inside a tool nobody can read.
Dimensional modelling for the reporting layer: conformed dimensions, slowly changing attributes handled deliberately, and grain decided before anyone builds a measure. Sized honestly, which for most mid-sized businesses means PostgreSQL or a modest cloud warehouse rather than a platform priced for a bank.
Semantic models with a defined measure layer, row-level security so a branch manager sees their branch, incremental refresh, and reports designed around a question rather than around every available field. Fast enough to open on a phone before a meeting.
A certified model business users can build their own views on, with the shared measures locked so nobody quietly redefines margin. Workspace structure, publishing rules and a clear line between the trusted layer and someone's exploratory sheet.
Tests on the pipeline: uniqueness, referential integrity, freshness, range checks and row-count anomalies, with a failure that alerts a person rather than silently publishing yesterday's numbers as today's. Lineage so any figure can be traced back to its source rows.
Demand and cash forecasting, lead and churn scoring, and reorder point modelling once the history is clean enough to support it. Starting with a baseline a spreadsheet could produce, so any model has to prove it is better than the obvious approach.
How we work
It starts with the decisions, not the data. Which recurring decisions are being made badly or slowly, who makes them, and what they would need to see. That produces a short list of metrics with written definitions, agreed by the people who will argue about them later. This step is unglamorous and it is the one that determines whether anything gets used.
Then we map the sources: where each field lives, who owns it, how reliably it is entered, and what it looks like when a person is in a hurry. Data quality problems get surfaced here rather than being discovered as a strange dip in a chart three months on. Sometimes the finding is that a field is not captured at all, and a small change to an operational form is worth more than any amount of modelling.
Delivery is incremental. One subject area, one pipeline, one dashboard in front of real users, then the next. A warehouse built for eleven subject areas before anyone has used a report is a large investment with no feedback in it. We would rather have one dashboard opened daily than a complete model opened at the steering meeting.
In practice
It starts with the decisions, not the data. Which recurring decisions are being made badly or slowly, who makes them, and what they would need to see. That produces a short list of metrics with written definitions, agreed by the people who will argue about them later. This step is unglamorous and it is the one that determines whether anything gets used.
Where it comes up
04 situations
If none of these is quite your situation, describe it in your own words — that is a better starting point than picking a service off a list.
Stack
We pick from a stack we actually run in production. Where your team already has a preference and it is a reasonable fit, we work in yours instead of arguing for ours.
Sector context
Domain knowledge shortens discovery. These are the sectors where we already understand the vocabulary and the failure modes, so the first version of the build is closer to right.
Shift-based production data, downtime causes, quality records and the daily gap between what the ERP holds and what the floor actually did.
Lot and parcel movement, job-work with karigars, grading records and the reconciliation that follows a stone through five pairs of hands.
Appointments, records, billing and reporting, where patient data handling and clinical workflow set the constraints before anything is designed.
We have not published a case study for this service yet. When we do, it will name the client with written permission and quote numbers the client can defend. Nothing else is worth reading.
Until then, the useful version of proof is a conversation about your problem. Ask us how we would approach it, what we would refuse to do, and where the estimate is soft.
Ask about comparable workQuestions
If you have one source system, light transformation and a handful of reports, connecting directly is the right answer and a warehouse is overhead. You need the separate layer once reports combine two or more systems, once history has to be preserved beyond what the source keeps, or once reporting queries start affecting operational performance. That third one is the trigger people notice first, usually as an ERP that slows down when the month-end report runs.
Yes, and the audit phase exists precisely to find out how bad it is before commitments are made. Some inconsistency can be handled in transformation with documented rules. Some cannot, because the information was never captured, and in that case the honest fix is a change to the operational form or process rather than a cleverer pipeline. We report which category each problem falls into rather than quietly averaging over it.
A first subject area, meaning one pipeline and one dashboard against real data, is typically four to six weeks including the definition work. Additional areas are faster because the plumbing exists. We deliberately avoid the pattern where nothing is visible for a quarter, both because feedback improves the result and because a project with no visible output is one that gets cancelled.
Less than you would expect, and less than a spreadsheet process does. Pipelines run on a schedule with alerting, and most changes are new reports rather than new plumbing. Someone does need to own the metric definitions, because that is a business decision rather than a technical one. Where nobody has capacity, we cover it under a small monthly arrangement and hand it over when there is an owner.
Usually. The route depends on the system: an ODBC connection, a supported export, direct database reads on a replica, or a scheduled file drop. Older systems sometimes have no interface worth the name, and then we work from scheduled exports, which is less elegant and works reliably. We check this during the audit rather than assuming an API exists.
Works well with
Send a paragraph about what is not working today. You will get a written reply from someone who would be on the build, covering how we would approach it, roughly what a first phase looks like, and whether data & bi is even the right place to spend the budget.
What happens next
