🧹 Removing Duplicates

Python provides several methods to remove duplicate elements from lists while preserving order. Understanding these methods is essential for data cleaning and processing.

# Basic duplicate removal
numbers = [1, 2, 2, 3, 3, 3, 4, 5, 5]
unique_numbers = list(dict.fromkeys(numbers))
print(unique_numbers)  # [1, 2, 3, 4, 5]

🎯 Duplicate Removal Methods

Python offers multiple approaches to remove duplicates.

Basic Examples

# Different ways to remove duplicates
numbers = [1, 2, 2, 3, 3, 3, 4, 5, 5]

# Using dict.fromkeys (preserves order)
unique1 = list(dict.fromkeys(numbers))
print(f"Using dict.fromkeys: {unique1}")

# Using set (doesn't preserve order)
unique2 = list(set(numbers))
print(f"Using set: {unique2}")

# Using list comprehension
unique3 = []
[unique3.append(x) for x in numbers if x not in unique3]
print(f"Using list comprehension: {unique3}")

🔍 Advanced Duplicate Removal

Python allows custom duplicate removal based on specific criteria.

# Advanced duplicate removal examples
# Remove duplicates based on a key
items = [{"id": 1, "name": "A"}, {"id": 1, "name": "B"}, {"id": 2, "name": "C"}]
unique_by_id = list({item["id"]: item for item in items}.values())
print(f"Unique by ID: {unique_by_id}")

# Remove duplicates while preserving order of first occurrence
from collections import OrderedDict
numbers = [1, 2, 2, 3, 3, 3, 4, 5, 5]
unique_ordered = list(OrderedDict.fromkeys(numbers))
print(f"Ordered unique: {unique_ordered}")

# Remove duplicates from list of tuples
pairs = [(1, 2), (2, 1), (1, 2), (3, 4)]
unique_pairs = list(dict.fromkeys(pairs))
print(f"Unique pairs: {unique_pairs}")

🎯 Key Takeaways

Was this helpful?

😔Poor
🙁Fair
😊Good
😄Great
🤩Excellent