Skip to content
Abdimalik
All notes

Before the chart, check what one row means

A practical data-analysis workflow for grain, joins, missing values and totals you can explain.

Data Analysis· 2 min read

Draft for review · General engineering guidance; not a project case study.

Conceptual workflow · From records to meaning

Define the unit of observation

Suppose a fictional retailer provides order and customer tables. Before calculating revenue, establish whether one order-table row represents an order, an item or a payment. Summing an order total repeated on every item row would answer a different question from summing one total per order.

State the join you expect

A customer lookup should not unexpectedly multiply order rows. Pandas supports merge validation, which can check expected key relationships. In this example, many orders may belong to one customer; the customer table is expected to contain a single record for each key.

Python · hypothetical DataFrames
enriched = orders.merge(
    customers,
    on="customer_id",
    how="left",
    validate="many_to_one",
    indicator=True,
)

Inspect what did not match

The merge indicator distinguishes records matched across the two inputs from records found on only one side. For the example analysis, inspect unmatched customer keys before deciding whether to exclude them. Missing information deserves an explicit rule; silently dropping rows can change the story the chart tells.

Reconcile the result before presenting it

Write a small reconciliation table for the hypothetical dataset: input rows, output rows, unmatched keys and totals before and after enrichment. These are checks to calculate from real data, not values to guess. If a total changes, explain why before formatting a dashboard.

Give the chart a definition

The final visual should make its scope clear:

  • What does one row represent?
  • Which date field and time range are used?
  • Are cancelled or refunded orders included?
  • How were duplicates and missing values handled?
  • Can another analyst reproduce the transformation?

Reference

Pandas · Merge, join, concatenate and compare

Abdimalik Osman Hassan

Full-stack · AI engineering · Data analysis