🧹 Removing Duplicates
Python provides several methods to remove duplicate elements from lists while preserving order. Understanding these methods is essential for data cleaning and processing.
# Basic duplicate removal
numbers = [1, 2, 2, 3, 3, 3, 4, 5, 5]
unique_numbers = list(dict.fromkeys(numbers))
print(unique_numbers) # [1, 2, 3, 4, 5]
🎯 Duplicate Removal Methods
Python offers multiple approaches to remove duplicates.
Basic Examples
# Different ways to remove duplicates
numbers = [1, 2, 2, 3, 3, 3, 4, 5, 5]
# Using dict.fromkeys (preserves order)
unique1 = list(dict.fromkeys(numbers))
print(f"Using dict.fromkeys: {unique1}")
# Using set (doesn't preserve order)
unique2 = list(set(numbers))
print(f"Using set: {unique2}")
# Using list comprehension
unique3 = []
[unique3.append(x) for x in numbers if x not in unique3]
print(f"Using list comprehension: {unique3}")
🔍 Advanced Duplicate Removal
Python allows custom duplicate removal based on specific criteria.
# Advanced duplicate removal examples
# Remove duplicates based on a key
items = [{"id": 1, "name": "A"}, {"id": 1, "name": "B"}, {"id": 2, "name": "C"}]
unique_by_id = list({item["id"]: item for item in items}.values())
print(f"Unique by ID: {unique_by_id}")
# Remove duplicates while preserving order of first occurrence
from collections import OrderedDict
numbers = [1, 2, 2, 3, 3, 3, 4, 5, 5]
unique_ordered = list(OrderedDict.fromkeys(numbers))
print(f"Ordered unique: {unique_ordered}")
# Remove duplicates from list of tuples
pairs = [(1, 2), (2, 1), (1, 2), (3, 4)]
unique_pairs = list(dict.fromkeys(pairs))
print(f"Unique pairs: {unique_pairs}")
🎯 Key Takeaways
Was this helpful?
Track Your Learning Progress
Sign in to bookmark tutorials and keep track of your learning journey.
Your progress is saved automatically as you read.