Why you don’t always need real-time data

I have seen the same conversation play out repeatedly in BI projects.

A business stakeholder asks for a dashboard. Someone asks how often the data can refresh. The answer is "every 30 minutes", and the next question is almost immediately: "Can we make it real-time?"

Sometimes the answer should be yes.

But sometimes the better answer is: "Why?"

That is not being obstructive. It is architecture.

The difference matters because every reduction in latency has a cost. The closer we get to real-time, the more we generally need to think about continuous ingestion, event handling, monitoring, failure recovery, data consistency, capacity and operational support.

If nobody is making a decision during those missing 30 minutes, we may simply be paying for speed that nobody uses.

Real-time is a business requirement, not a technology preference

Real-time data

Data that is made available with sufficiently low latency that a system or person can respond to events as they happen. There is no universal number of seconds that defines "real-time". The required latency depends on the business decision and the consequences of being late.

The important distinction is not really between "old data" and "new data".

It is between data that is sufficiently fresh for the decision and data that is not.

Imagine a sales manager looking at yesterday's revenue at 9am. If the manager is reviewing performance and deciding what the team should focus on today, yesterday's completed data may be perfectly appropriate.

Now imagine a manager responsible for a live contact centre. If the number of customers waiting changes every few seconds and staffing decisions need to be made immediately, yesterday's data is useless.

The data could come from the same organisation, use the same platform and appear in the same Power BI environment. The required architecture is different because the decision is different.

That is the point I would want architects and BI teams to start with.

A clean architectural illustration showing a horizontal spectrum from "Once a day" through "Hourly" and "Every 15 minutes" to "Seconds". Above each section, show representative business decisions. Once a day should show board reporting and financial analysis. Hourly should show operational management. Every 15 minutes should show team performance monitoring. Seconds should show fraud detection, machine telemetry and safety alerts. At the bottom, show the principle: "Choose latency based on the decision, not the technology."

Consider the recruitment company

Recruitment provides a useful example because it contains almost every type of reporting requirement in one business.

A recruitment company might track vacancies, candidates, CV submissions, interviews, placements, timesheets, invoicing, margins and consultant activity.

It would be easy to say that all of this should be real-time.

I would challenge that.

Take CV sends.

A manager monitoring a team's activity might benefit from seeing CV sends as they happen. If a consultant has a target of sending 20 CVs by midday and the manager is actively intervening throughout the day, live or near real-time information could be useful.

But does the consultant themselves need their CV send count updated within five seconds?

Probably not.

If the dashboard refreshes every 30 minutes, or even a few times during the working day, the business outcome may be exactly the same.

Now consider jobs added that day.

If a new vacancy is added at 10:04 and a consultant sees it at 10:20, has the business failed?

Probably not.

But consider a vacancy with a very short response window, where the first consultant to identify and respond to it has a materially higher chance of filling it. Suddenly, latency becomes much more important.

The metric has not changed.

The business decision has.

That is why requirements such as "we need real-time data" are incomplete requirements.

The architect needs to ask what the user will do differently because the data arrived sooner.

There is another important point here.

Not every user needs the same version of the truth at the same speed.

A recruitment consultant may need a simple activity view. A team leader may need live operational information. Finance may need reconciled numbers at the start of each day. The executive team may need weekly trends.

Trying to satisfy all of these requirements through a single real-time architecture can be unnecessarily complicated.

A better architecture can provide different data products for different decisions.

The same underlying event can feed a live operational view while also being incorporated into a governed analytical model during the next scheduled processing cycle.

This is often where the hybrid approach becomes powerful.

The real cost of real-time

When people talk about the cost of real-time data, they often think about infrastructure cost first.

That matters, but it is only part of the picture.

Streaming architectures introduce a different operating model. Instead of waiting for a pipeline to run at 2am, the platform may be continuously receiving, processing and delivering events. That creates different requirements for monitoring, alerting, resilience, replay, error handling and capacity management.

Microsoft's Real-Time Intelligence capabilities, for example, are explicitly designed around ingesting, processing, visualising and acting on data in motion. That is extremely useful for the scenarios that need it.

But the existence of the capability does not mean every dataset should use it.

Real-time architecture should be treated as a deliberate design choice.

• Continuous ingestion from source systems
• Event or message infrastructure
• Streaming transformations
• Handling late or duplicate events
• Monitoring data pipelines continuously
• Alerting when processing falls behind
• Managing failures without silently losing events
• Supporting replay or reprocessing
• Managing schema changes in event streams
• Handling spikes in data volume
• Ensuring downstream systems can consume data at the required speed
• Establishing clear ownership for operational incidents
• Testing what happens when the source becomes unavailable
• Testing what happens when downstream processing fails
• Maintaining appropriate security and governance
• Managing the additional platform and operational costs

When real-time genuinely matters

None of this is an argument against real-time data.

Real-time data is brilliant when the business actually needs it.

There are situations where waiting five minutes, an hour or a day is unacceptable.

Fraud detection is an obvious example. If a transaction appears suspicious, knowing about it tomorrow is of limited value.

Operational technology provides another. If a machine begins operating outside safe parameters, the useful response is to intervene while the problem is occurring.

Customer-facing systems can also need immediate information. If a customer places an order and the available stock does not update quickly enough, another customer could be promised something that is no longer available.

The common factor is not that the data is interesting.

It is that there is an action to take now.

Microsoft describes Real-Time Intelligence as particularly suitable for scenarios where organisations need to respond to events as they happen, process continuously flowing data or analyse logs. Its current capabilities include event ingestion, transformation, visualisation and actions triggered by changing conditions.

That is the right way to think about real-time architecture.

It is not simply "a faster dashboard".

It is an architecture designed around events, changing conditions and actions.

That distinction is important.

The answer is often a hybrid architecture

One of the most useful architectural patterns is to stop asking whether an organisation should be real-time or batch.

It can be both.

A business can have a real-time path for operational events and a scheduled analytical path for governed reporting.

For example, a recruitment platform might capture a new CV submission immediately. That event could feed a live operational dashboard used by a team leader.

The same event could then be persisted into the analytical data platform and incorporated into the trusted semantic model during the next scheduled refresh.

The live view answers:

"What is happening now?"

The analytical model answers:

"What has happened, and what does it mean?"

Those are different questions.

They do not necessarily need the same architecture.

Modern platforms explicitly support these patterns. Microsoft Fabric, for example, brings together scheduled data integration and Real-Time Intelligence within the same broader analytics platform, while Real-Time Intelligence supports streaming data and event-driven actions.

This is not a compromise.

It is often better architecture.

A practical test for deciding whether you need real-time data

• What event causes the user to look at the data?
• What action will they take after seeing it?
• Does that action need to happen immediately?
• What happens if the data is 30 minutes old?
• What happens if it is two hours old?
• What happens if it is one day old?
• Is there a measurable business consequence to the delay?
• Is the user actively monitoring the information?
• Is an automated response required?
• Could an alert be more useful than a dashboard?
• Does the source system support the required latency reliably?
• Can the organisation support the operational requirements of a streaming solution?
• Is accuracy more important than immediacy?
• Does the metric depend on reconciliation or validation?
• Would a scheduled refresh provide the same business outcome?

One of the most important questions in this assessment is what happens when data arrives quickly but is wrong.

A five-second-old number that is incomplete, duplicated or unreconciled can be less useful than a two-hour-old number that has been validated.

This is particularly important in financial reporting.

Imagine a finance dashboard showing revenue immediately after every transaction. That sounds impressive. But if transactions can subsequently be amended, reversed, credited or reconciled against other systems, the "live" number may not represent the figure finance actually wants to report.

The requirement may not really be "show me the latest transaction".

It may be "show me the latest reliable financial position".

Those are very different requirements.

This is why data freshness and data correctness should be treated as separate dimensions.

Latency should follow the human workflow

There is also a human factor that is easy to overlook.

People do not make decisions continuously simply because data is available continuously.

A consultant might review their KPI dashboard at 10am, 12pm and 3pm. Refreshing the dashboard every five seconds does not create five-second decision making.

A sales director might review performance once every morning. A board might review a dashboard once a week.

If the user only looks at the information periodically, the value of continuous refresh may be limited.

There are exceptions. A manager actively running a live operation may genuinely need to watch a metric continuously. In that case, real-time data can become part of the operational workflow itself.

The architecture should therefore reflect the behaviour of the user.

Not the theoretical maximum speed of the platform.

Real-time BI is not the same as a real-time business

There is a subtle distinction between a dashboard being capable of real-time updates and the business actually operating in real time.

A dashboard can refresh continuously while the underlying process remains slow.

A recruitment system might show a vacancy seconds after it is created, but if recruiters only review new vacancies during their morning allocation meeting, the real-time dashboard has not changed the workflow.

Equally, a live dashboard can be incredibly valuable when the business has people or automated processes ready to respond immediately.

Microsoft's Real-Time Dashboard capability is designed for live monitoring and continuously updating visualisations, particularly where users need to spot changes and act quickly.

The important word is act.

If there is no action associated with the changing data, ask why the data needs to be changing so quickly.

Data freshness

How recently the information represented in a dataset was updated.

Data latency

The elapsed time between an event occurring and that event becoming available for its intended consumer.

Real-time

A low-latency operating model where information is processed and made available quickly enough to support decisions or actions as events occur. The required latency is business-specific.

Near real-time

Data that becomes available shortly after an event, typically through frequent refreshes or micro-batching, without requiring continuous streaming.

Start with the cheapest architecture that meets the decision

There is a principle I find useful when designing data platforms:

Do not buy lower latency than the business can use.

If an hourly refresh supports the decision, start there.

If 15 minutes is genuinely valuable, design for 15 minutes.

If the business needs events within seconds because a decision must happen immediately, then build the streaming architecture.

This is not about choosing the cheapest technology.

It is about choosing the simplest architecture that reliably meets the business requirement.

Sometimes that will be a scheduled pipeline.

Sometimes it will be a change data capture process.

Sometimes it will be a streaming architecture.

Sometimes it will be a combination.

The mature answer is not "real-time is expensive, so avoid it".

The mature answer is "real-time has a purpose, so use it where that purpose exists".

Cloud platforms make streaming increasingly accessible, but accessibility should not be confused with necessity. AWS, Google Cloud and Microsoft all provide mature capabilities for streaming and real-time analytics, while also supporting batch processing patterns.