Python: How to Remove Duplicates from a List (with Examples)

Removing duplicates from a list in Python means creating a new list that contains only the unique elements of the original list. This can be useful for eliminating redundant data or for performing operations that require distinct elements.

This succinct and straight-to-the-point article will show you a couple of different ways to discard duplicate elements from a given list in Python. We’ll also discuss the performance of each approach to figure out how fast it is.

Using a set (not preserving the order of elements)

This is the easiest and fastest way to remove duplicates from a list in Python. You just need to use the built-in set() function to create a set from the original list. A set is an unordered collection of distinct objects that does not allow duplicates. Then, you can convert the set back to a list using the built-in list() function.

Example:

# Creating an original list
original_list = ['sling', 'academy', 'Python', 'sling', 'Python', 'academy']

# Removing duplicates from the list using a set
unique_list = list(set(original_list))

# Printing both lists
print("Original List:", original_list)
print("Unique List:", unique_list)

Output:

Original List: ['sling', 'academy', 'Python', 'sling', 'Python', 'academy']
Unique List: ['academy', 'sling', 'Python']

One important thing you should be aware of is that this method does not preserve the order of the original list since sets are unordered collections. Therefore, if you care about the order of the elements, see the next section of this article.

Performance: This method has a time complexity of O(n), where n is the number of elements in the list. It also has a space complexity of O(n), since it creates a new set and a new list objects.

Using a loop (preserve the order of elements)

The core idea of this approach is to use a loop to iterate over the elements of the original list and check if they are already in a new list. If not, you can append them to the new list. This creates a new list object that contains only the unique elements of the original list.

This method is slower (I will explain this later) and longer than the previous one, but it maintains the order of the original list since it appends the elements in the same order as they appear in the original list.

Example:

# Creating an original list
original_list = [1, 2, 2, 3, 4, 3, 5, 6, 7, 7]

# Removing duplicates from the list using a loop
unique_list = []
for element in original_list:
    if element not in unique_list:
        unique_list.append(element)

# Printing both lists
print("Original List:", original_list)
print("Unique List:", unique_list)

Output:

Original List: [1, 2, 2, 3, 4, 3, 5, 6, 7, 7]
Unique List: [1, 2, 3, 4, 5, 6, 7]

Performance: This method has a time complexity of O(n^2) . This is because, for each element of the original list, it has to check if it is already in the new list, which takes O(n) time. It also has a space complexity of O(n), since it creates a new list object.