Missing values and duplicate records
Learn the reasoning. Try the task. Compare your answer.
Prepared for DataCareerHub · Curriculum version 2026-10-15.1
By the end of this lesson
- Apply missing values and duplicate records to the fictional case.
- Record the rule, evidence and validation result.
- Explain a limitation and a defensible next step.
Fictional learning case · Local practice files · No personal data.
Understand the method
Missing values and duplicate records
Missingness has causes: uncollected, not applicable, failed parsing or absent export. Decide whether the field is required for the intended use before choosing a response. Replacing every blank with zero mixes absence with a real measurement. Duplicate detection also needs a rule: an exact repeated record, repeated business key and legitimate repeated parent key are different situations. Preserve the raw row number so every decision can be traced back.
Work through the case
L004 has blank quantity, L007 has an ambiguous date and L010 has an unknown SKU. L001 appears twice identically. L005 occurs twice with conflicting quantities. The reference workflow quarantines both L005 versions because there is no authoritative update timestamp or version flag. For the exact duplicate L001, it retains the first occurrence and records the later occurrence as excluded. Reconciliation counts excluded duplicates among quarantined rows; that label includes records set aside for several reasons.
A method you can reproduce
Profile required fields first. Group raw records by stripped line_id and compare their complete values. Identify conflicting keys before processing any row so the first conflicting version cannot slip through. Track a set of exact raw fingerprints to identify repeated records. Assign explicit reason codes; one row can have several reasons, so the number of reasons need not equal the number of rejected records. Keep raw_record as evidence and never sum rejected values into accepted totals.
Check your reasoning
Keeping the first version of a conflicting key is arbitrary without evidence of version order. Conversely, deleting every repeated order_id removes legitimate lines. The disposition depends on the business entity and available authority.
Further reading: pandas merge validation, pandas melt, Python CSV parsing. These official references support the methods; the case and exercises are authored for DataCareerHub.
Practical exercise
List the missing quantity, exact duplicate and conflicting-key rows. Define a disposition for each. Predict the number of accepted and quarantined records before checking expected_results.json.
Write your answer in learner_worksheet.txt and retain the evidence file. Attempt this before opening the solution.
Open the worked solution
The workflow accepts 8 of 15 records and quarantines 7. The two L005 records are both quarantined. The later L001 copy is excluded as exact_duplicate. L004, L007, L010 and L013 fail required validity or lookup rules. Count records, not summed reason codes: accepted + quarantined = 15.
Independent extension: change one relevant input in a separate copy. Predict the result first, rerun the checks and explain any difference. Keep the supplied source unchanged.
One more question.
How these learning materials are prepared and maintained · Report a correction: support@datacareerhub.io
Read and practice without an account. Sign in with an existing account to save lesson completion.
Next lesson →This is a fictional learning case. Results do not establish real-world performance or causal effects. Portfolio practice is not a claim of client work or an accredited qualification. Explore practical lessons, worked examples and projects. Follow a published path, practise the method and build work you can explain.