Normalize and Deduplicate Leads in n8n

Managing incoming lead data often involves dealing with messy formatting, duplicate entries, and invalid email addresses. The Normalize and Deduplicate Leads template provides a lightweight, self-contained solution to clean your lead batches directly within n8n before passing them to your CRM or marketing tools.

This workflow is part of the broader library of n8n workflows designed to streamline data transformation without relying on external APIs or paid third-party services.

Use Case

When importing leads from webhooks, CSV uploads, or external forms, the data is frequently inconsistent. This workflow automates several cleanup tasks in a single run:

• Whitespace Trimming: Removes leading, trailing, and consecutive spaces from names and company fields.
• Email Normalization: Converts email addresses to lowercase and trims whitespace to ensure consistent formatting.
• Deduplication: Identifies and removes duplicate records based on a case-insensitive email check within the current batch.
• Validation: Filters out records with missing emails or invalid email formats, returning them in a separate list of rejected items with reasons for failure.

Input and Output Example

The workflow expects a single input item containing a leads array. Each lead object can contain name, email, and company fields.

Example Input

{
  "leads": [
    {
      "name": " Ada Example ",
      "email": " ADA@example.com ",
      "company": " Example Co "
    },
    {
      "name": "Ada Duplicate",
      "email": "ada@example.com"
    },
    {
      "name": "Ben Example",
      "email": "ben@example.org"
    },
    {
      "email": "not-an-email"
    }
  ]
}

Example Output

Processing the synthetic data above yields a structured output containing accepted leads, a summary of the execution, and a list of rejected records:

[
  {
    "leads": [
      {
        "name": "Ada Example",
        "email": "ada@example.com",
        "company": "Example Co"
      },
      {
        "name": "Ben Example",
        "email": "ben@example.org",
        "company": ""
      }
    ],
    "summary": {
      "received": 4,
      "accepted": 2,
      "duplicatesRemoved": 1,
      "rejected": 1
    },
    "rejected": [
      {
        "index": 3,
        "reason": "Missing or invalid email format"
      }
    ]
  }
]

Implementation Decisions

• In-Memory Processing: The workflow processes data entirely inside n8n using a built-in Code node. This avoids external API calls, keeping execution fast and secure.
• First-Record-Wins Logic: For duplicate emails, the workflow retains the first record it encounters. Subsequent duplicates are dropped and counted, but they do not overwrite or merge data into the original record.
• Strict Schema Enforcement: Only name, email, and company are preserved. Any extra fields present in the input are intentionally dropped to maintain a clean, predictable output structure.
• Zero External Dependencies: The workflow runs entirely on built-in n8n nodes (Manual Trigger, Edit Fields, and Code) and does not require any credentials or external integrations.

Limits and Troubleshooting

• Basic Format Validation: The email validation uses a basic regular expression check (/^[^\s@]+@[^\s@]+\.[^\s@]+$/). It ensures the string looks like an email address, but it does not verify domain deliverability or mailbox existence.
• Batch-Only Deduplication: Deduplication is performed only within the current input batch. It does not check against historical data in an external database or CRM.
• Memory Constraints: Because processing occurs in-memory, you should use manageable batch sizes to prevent performance degradation on your n8n instance.
• Error: "Expected a leads array": This error occurs if the input to the Normalize and Deduplicate node does not contain a valid leads array. If your preceding node outputs multiple separate n8n items instead of a single item with an array, you must aggregate them into a single array first before passing them to this workflow.