Data Quality and Cleaning in Large Archives
The central idea is transformation rather than deletion: an original address remains in the archive, while processing can treat it as equivalent to a chosen canonical address.
Why Identity Fragments
A large email archive may contain several addresses for the same person. Email addresses and domain names change as people move between organizations. If each address is treated as a separate identity, the archive can represent one contributor as several unrelated contributors. Mapping rules provide a way to consolidate those variants into canonical forms during processing.
The central idea is transformation rather than deletion: an original address remains in the archive, while processing can treat it as equivalent to a chosen canonical address.
Consolidating Individual Addresses
The Mapping table in mapping.sqlite stores individual email mappings in source-to-destination form. The source is the address that appears in the archive, and the destination is the canonical address that the processing system should use instead. When gmodel.py encounters an address, it checks the Mapping table. If a mapping exists, it uses the destination address. If no mapping exists, it uses the address as-is.
Building a Career History
Steve Githens and Three Addresses
Consolidate Steve Githens' three email addresses into the most recent address, swgithen@mtu.edu.
- 1Choose the destination
Use swgithen@mtu.edu as the canonical destination because it is Steve Githens' most recent address.
- 2Add the first source mapping
Create a Mapping entry from the Northwestern address to swgithen@mtu.edu.
- 3Add the second source mapping
Create a Mapping entry from the Cambridge address to swgithen@mtu.edu.
- 4Leave the canonical address unmapped
Do not add a third entry from swgithen@mtu.edu to itself. It is already the final form.
Two Mapping entries consolidate the three addresses. Whenever gmodel.py encounters either the Northwestern address or the Cambridge address, it uses swgithen@mtu.edu instead. The canonical address is used as-is.
For three known addresses, two entries are sufficient when one address is selected as canonical. The mapping count is based on the noncanonical addresses, not on the total number of addresses.
Normalizing Domains First
The DNSMapping table works at the domain level rather than the individual address level. For an address such as someone@iupui.edu, gmodel.py first checks whether iupui.edu has a DNS mapping. If the table maps iupui.edu to indiana.edu, the address becomes someone@indiana.edu. Individual email address mappings are checked after this domain transformation.
Following One Archive Record
When gmodel.py processes the archive, it reads each message and extracts sender and recipient addresses. It checks DNSMapping first, then checks the individual-address Mapping table. If a relevant rule exists at either stage, the destination is used. If no rule exists, the current address continues unchanged. This process is repeated for every email in the archive, so one mapping entry can affect thousands of messages when the associated person was a prolific contributor.
Protecting the Original Archive
Email and DNS mappings are applied during data processing, not during data collection or storage. They act as a transformation layer, so the original email archive remains unchanged. If a mapping later proves incorrect, it can be modified or removed without changing the archived source data. This makes it possible to test different mapping strategies while preserving the original records.
Treat mappings as adjustable rules for analysis, not as replacements for the underlying archive. Preserve the original data and revise the transformation rules when your understanding of an identity or domain changes.
Mistakes in Archive Cleaning
Common mistake Adding a self-mapping for the canonical address
What it looks likeAdding a third entry from swgithen@mtu.edu to swgithen@mtu.edu
Why it failsThe canonical address is already in its final form and does not need a mapping entry.
Do this insteadMap only the two noncanonical addresses to swgithen@mtu.edu.
Common mistake Using DNSMapping for an individual address change
What it looks likeTrying to use a domain-level rule to represent the change from one complete email address to another
Why it failsDNSMapping operates on domain names, while the Mapping table stores individual email address mappings.
Do this insteadUse the Mapping table for individual address variants and DNSMapping for domain variants.
Common mistake Checking individual address mappings before DNS mappings
What it looks likeApplying the Mapping table before transforming iupui.edu to indiana.edu
Why it failsThe documented processing order places the DNSMapping check before the individual email address check.
Do this insteadProcess the domain-level mapping first, then apply the individual address mapping.
Common mistake Editing the original archive to normalize addresses
What it looks likeReplacing archived addresses directly instead of adding processing rules
Why it failsMappings are designed as non-destructive transformation rules and are applied during processing.
Do this insteadKeep the original archive unchanged and adjust the Mapping or DNSMapping rules.
Common mistake Assuming that an unmapped address is discarded
What it looks likeTreating an address with no mapping entry as missing from the processed data
Why it failsIf no mapping exists,
gmodel.pyuses the address as-is.Do this insteadDistinguish between an address that has no mapping and an address that has been mapped to a canonical destination.
Practice the Mapping Decisions
A contributor appears in an archive under three addresses, and you choose the most recent address as the canonical destination. Decide how many Mapping entries are needed, which addresses should be sources, and whether the canonical address needs an entry. Then explain whether a domain change should be handled in Mapping or DNSMapping.
Hints
- Count the noncanonical addresses rather than counting all addresses.
- A Mapping entry points from a source address to a destination address.
- Use DNSMapping when the transformation concerns a domain name rather than one complete email address.
What do you think happens?
If gmodel.py encounters an address with no entry in the Mapping table, what happens to that address?
Reveal answer
It is used as-is.
The processing rule checks for a mapping entry. When no mapping exists, the address remains as encountered.
Unified Archive View
- 1The Mapping table consolidates individual email address variants by sending source addresses to a canonical destination.
- 2When three addresses belong to one person and one is canonical, two Mapping entries are sufficient; the canonical address does not map to itself.
- 3DNSMapping consolidates domain names and is applied before individual email address mappings.
- 4Mappings are non-destructive processing rules: the original archive remains unchanged and the rules can be revised or removed.
- 5Email and DNS mappings work together to create a unified view of contributors across organizational and career changes.
Key Takeaways
- Use the Mapping table for individual email address changes and DNSMapping for domain changes.
- Choose one canonical destination and map each noncanonical source address to it.
- Apply DNS mappings before individual email mappings when processing archive records.
- Keep the original archive unchanged because mappings form a reversible transformation layer.
- A small number of carefully chosen mappings can normalize many archive messages.