Skip to content
Hacking with RustHacking with Rust
Select theme

How to deduplicate a Vec

Short answer: Vec::dedup removes only consecutive duplicates — the same value sitting directly next to itself. On an unsorted Vec it removes almost nothing. When any order will do, sort first and then dedup; when the order matters, keep the first occurrence of each value with a HashSet and retain.

dedup compares neighbours

Section titled “dedup compares neighbours”

From a fresh project (cargo new dedup-demo), put this in src/main.rs and run cargo run:

fn main() {
    let mut scores = vec![1, 1, 2, 3, 3, 3, 2];
    scores.dedup();
    println!("{:?}", scores);
    println!("len {} capacity {}", scores.len(), scores.capacity());
}
Run this in the Rust Playground
[1, 2, 3, 2]
len 4 capacity 7

One pass, no allocation: each element is dropped when it equals its predecessor, and the elements past the new length stay allocated — that’s why len falls to 4 while capacity is still 7. Only runs of equal neighbours shrink, so the trailing 2 survives its distant cousin.

Now the trap. The same call on an unsorted Vec does nothing at all:

fn main() {
    let mut unsorted = vec![1, 2, 1];
    unsorted.dedup();
    println!("{:?}", unsorted);
}
Run this in the Rust Playground
[1, 2, 1]

The two 1s never sit next to each other, so neither is “a duplicate of its neighbour”. Nothing sorts for you; dedup only looks sideways.

Sort first

Section titled “Sort first”

If any order will do, sort and then dedup. Replace the contents of src/main.rs with:

fn main() {
    let mut nums = vec![3, 1, 2, 1, 3];
    nums.sort();
    nums.dedup();
    println!("{:?}", nums);
}
Run this in the Rust Playground
[1, 2, 3]

Sorting brings the equal values together, which turns “remove the repeats” into “remove the neighbours”. sort_unstable works too — the duplicates here are identical values, so which one survives doesn’t matter. The order that comes back is sorted order, not input order.

Keep the first occurrence

Section titled “Keep the first occurrence”

When the order matters, drop the later duplicates instead:

use std::collections::HashSet;


fn main() {
    let mut words = vec![
        "apple".to_string(),
        "pear".to_string(),
        "apple".to_string(),
        "fig".to_string(),
        "pear".to_string(),
    ];
    let mut seen = HashSet::new();
    words.retain(|w| seen.insert(w.clone()));
    println!("{:?}", words);
}
Run this in the Rust Playground
["apple", "pear", "fig"]

HashSet::insert returns true when the value is new, so retain keeps each value the first time and drops it after. This needs T: Hash + Eq rather than PartialEq, the values live in a second collection of their own, and every element is cloned once on the way through. For a Vec of values that are expensive to hash, sorting in place can be the cheaper trade.

Two things that surprise people

Section titled “Two things that surprise people”

dedup compiles for f64, but PartialEq for floats says NaN is not equal to itself:

fn main() {
    let mut values = vec![1.0, f64::NAN, f64::NAN, 2.0];
    values.dedup();
    println!("{:?}", values);
}
Run this in the Rust Playground
[1.0, NaN, NaN, 2.0]

Consecutive NaNs survive, because “equal to the previous element” is false for NaN. That same contract is why f64 can’t be a HashMap key — a later article in this chapter works through it.

And dedup needs T: PartialEq, so a struct that doesn’t implement it doesn’t compile at all:

struct Person {
    name: String,
}


fn main() {
    let mut people = vec![Person { name: "Ada".to_string() }, Person { name: "Ada".to_string() }];
    people.dedup();
    println!("{}", people.len());
}
Run this in the Rust Playground
error[E0599]: the method `dedup` exists for struct `Vec<Person>`, but its trait bounds were not satisfied
 --> src/main.rs:7:12
  |
1 | struct Person {
  | ------------- doesn't satisfy `Person: PartialEq`
...
7 |     people.dedup();
  |            ^^^^^
  |
  = note: the following trait bounds were not satisfied:
          `Person: PartialEq`
help: consider annotating `Person` with `#[derive(PartialEq)]`
  |
1 + #[derive(PartialEq)]
2 | struct Person {
  |


For more information about this error, try `rustc --explain E0599`.

The compiler names the missing trait and its own fix. Or skip the derive and compare a field: people.dedup_by_key(|p| p.id) drops adjacent entries with the same id.

Section titled “Related”

Next in this chapter: how to find an element in a Vec.