Q5 - MongoDB Fleet Error Report (Ranking Incidents Across Servers)

The situation: you are the on-call engineer. A rough night across a MongoDB Atlas fleet is over, and now you have to stand up in the incident review and say which errors actually mattered. What you have is the raw log haul: for every node in the fleet, the list of error codes it emitted, in the order it emitted them, repeats and all.

Each node is named atlas-<replicaSet>-node-<n>, so the fleet looks like this:

You will answer two different questions about the same log haul, and they do not have the same answer:

Part 1   "which error reached the most of the fleet?"
         -> count the distinct NODES that logged it                    (its spread)

Part 2   "which error came closest to taking a replica set down?"
         -> count the nodes it hit INSIDE ONE replica set              (its blast radius)

FLEET (6 nodes, 3 replica sets)

  atlas-rs0-node-1   ->  WT_CACHE_FULL, REPL_LAG_EXCEEDED, WT_CACHE_FULL
  atlas-rs0-node-2   ->  WT_CACHE_FULL
  atlas-rs0-node-3   ->  WT_CACHE_FULL, INDEX_BUILD_ABORTED
  atlas-rs1-node-1   ->  REPL_LAG_EXCEEDED
  atlas-rs1-node-2   ->  REPL_LAG_EXCEEDED, INDEX_BUILD_ABORTED
  atlas-rs2-node-1   ->  REPL_LAG_EXCEEDED, CURSOR_NOT_FOUND x50

The input. fleet is a list of nodes. Each entry pairs the node's name with the list of error codes that node logged, in order, repeats included:

fleet = [
  ("atlas-rs0-node-1", ["WT_CACHE_FULL", "REPL_LAG_EXCEEDED", "WT_CACHE_FULL"]),
  ("atlas-rs0-node-2", ["WT_CACHE_FULL"]),
  ("atlas-rs0-node-3", ["WT_CACHE_FULL", "INDEX_BUILD_ABORTED"]),
  ...
]

Node names are unique, and a node that logged nothing carries an empty list. In the typed languages below the same entry is a small ServerLog record holding a name and an errors list.

CURSOR_NOT_FOUND x50 is shorthand for fifty separate log lines. Real log hauls look exactly like that: one unhappy node repeating itself into the thousands while the genuinely dangerous error shows up once or twice per node and is easy to miss.

Part 1: The Error Board (~15 minutes)

Write top_errors(fleet, k), returning the k error codes that belong at the top of the incident review, each with the numbers that justify its place.

Rank by how many distinct nodes logged the error, which we will call its spread. Break ties with the rules below, and return (code, spread, total_occurrences) for each:

1. spread descending, the number of distinct nodes that logged it at least once,
2. then total occurrences descending,
3. then error code ascending, so the board is always in the same order for the same input.

If k is larger than the number of distinct error codes, return all of them.

INPUT:  the 6-node fleet, k = 3

OUTPUT:
  ("REPL_LAG_EXCEEDED",   4, 4)
  ("WT_CACHE_FULL",       3, 4)
  ("INDEX_BUILD_ABORTED", 2, 2)

WHY:
  REPL_LAG_EXCEEDED    seen on rs0-node-1, rs1-node-1, rs1-node-2, rs2-node-1   -> spread 4
  WT_CACHE_FULL        seen on rs0-node-1, rs0-node-2, rs0-node-3               -> spread 3
                       (4 occurrences, because rs0-node-1 logged it twice)
  INDEX_BUILD_ABORTED  seen on rs0-node-3, rs1-node-2                           -> spread 2
  CURSOR_NOT_FOUND     50 occurrences but only ONE node                         -> spread 1, last

  Counting lines would have put CURSOR_NOT_FOUND first. Counting nodes puts it dead last.

INPUT:
  atlas-rs0-node-1   ->  DISK_SPACE_LOW, ELECTION_TIMEOUT, DISK_SPACE_LOW
  atlas-rs1-node-1   ->  DISK_SPACE_LOW, ELECTION_TIMEOUT, AUTH_FAILURE
  atlas-rs2-node-1   ->  AUTH_FAILURE
  k = 3

OUTPUT:
  ("DISK_SPACE_LOW",   2, 3)
  ("AUTH_FAILURE",     2, 2)
  ("ELECTION_TIMEOUT", 2, 2)

WHY:
  every code here has spread 2, so the first tie-break decides:
    DISK_SPACE_LOW logged 3 times, the other two twice each  ->  it takes the top spot

  AUTH_FAILURE and ELECTION_TIMEOUT are still level on spread AND occurrences,
  so the last tie-break settles it:  "AUTH_FAILURE" < "ELECTION_TIMEOUT"

  Without that final alphabetical key the order of those two would depend on
  hash iteration order, and the same input could print two different boards.

Solution - Part 1

Two tallies in one pass, then one sort. The only decision that matters is what each tally holds.

Step 1: One pass, two tallies, and only one of them is a counter

Walk every node, and for each error code it logged, record two things: which node saw it, and one more occurrence. The first has to be a set of node names, not a counter. A node that logged CURSOR_NOT_FOUND fifty times must still move the spread by exactly one, and a set is what makes that true without any special-casing.

After a single pass over the six nodes, the two tallies look like this:

error code             nodes that logged it                                spread   occurrences
--------------------------------------------------------------------------------------------
WT_CACHE_FULL          {rs0-node-1, rs0-node-2, rs0-node-3}                     3             4
REPL_LAG_EXCEEDED      {rs0-node-1, rs1-node-1, rs1-node-2, rs2-node-1}         4             4
INDEX_BUILD_ABORTED    {rs0-node-3, rs1-node-2}                                 2             2
CURSOR_NOT_FOUND       {rs2-node-1}                                             1            50

note WT_CACHE_FULL: spread 3 but 4 occurrences, because rs0-node-1 logged it twice
note CURSOR_NOT_FOUND: 50 occurrences collapse into a single node in the set

for node_name, error_codes in fleet:
    for code in error_codes:
        nodes_hit[code].add(node_name)   # a set: 50 repeats from one node still count once
        occurrences[code] += 1           # a counter: every line counts

Step 2: Sort on all three keys at once

Now rank the distinct codes. Spread leads, occurrences break the first tie, and the code itself breaks the second. Sorting on a tuple gives you all three at once, and negating the two numeric keys flips them to descending while the string stays ascending.

code                   key = (-spread, -occurrences, code)        position
-------------------------------------------------------------------------
REPL_LAG_EXCEEDED      (-4,  -4, "REPL_LAG_EXCEEDED")                    1
WT_CACHE_FULL          (-3,  -4, "WT_CACHE_FULL")                        2
INDEX_BUILD_ABORTED    (-2,  -2, "INDEX_BUILD_ABORTED")                  3
CURSOR_NOT_FOUND       (-1, -50, "CURSOR_NOT_FOUND")                     4   <- 50 lines, still last

The last row is the whole lesson in one line. CURSOR_NOT_FOUND has more occurrences than every other error put together, and it still finishes last, because the first key it is compared on is the one that reflects reality.

from collections import Counter, defaultdict


def top_errors(fleet, k):
    """fleet: list of (node_name, [error codes]). Returns (code, spread, occurrences)."""
    nodes_hit = defaultdict(set)     # error code -> distinct nodes that logged it
    occurrences = Counter()          # error code -> how many log lines it produced

    for node_name, error_codes in fleet:
        for code in error_codes:
            nodes_hit[code].add(node_name)   # a set, so repeats from one node count once
            occurrences[code] += 1

    ranked = sorted(
        nodes_hit,
        key=lambda code: (-len(nodes_hit[code]), -occurrences[code], code),
    )
    return [(code, len(nodes_hit[code]), occurrences[code]) for code in ranked[:k]]

Complexity Analysis

• Time: O(L + d log d), where L is the total number of log lines across the fleet and d is the number of distinct error codes. The pass is O(L) and the sort only touches the d distinct codes, never the raw lines.
• Space: O(d + S), where S is the total size of the node sets. In the worst case every code was seen on every node, so S is d times the fleet size, but on a real haul the sets stay small.
• Worth saying out loud: the sort is over distinct codes, not log lines. A fleet can produce millions of lines and still sort a list of a few dozen error codes.

Part 2: Blast Radius (~15 minutes)

Your board is up, and someone senior asks the sharper question: which of these actually took writes down?

Spread cannot answer that, because it treats every node as interchangeable and they are not. A replica set keeps accepting writes only while a majority of its members are healthy, so what actually hurts is concentration. Three nodes failing inside rs0 walks that one replica set straight past its majority and writes stop, while the same three failures landing one-each in rs0, rs1 and rs2 leave all three sets with a working majority. Same three nodes, completely different night.

So rank by blast radius: the largest number of distinct nodes an error hit inside any single replica set. The replica set is the middle segment of the node name, so atlas-rs0-node-1 belongs to rs0.

Write top_errors_by_blast_radius(fleet, k), taking the same fleet as Part 1 and returning (code, worst_replica_set, nodes_hit_in_it) for the top k, ranked by:

1. blast radius descending,
2. then spread descending (the Part 1 number, as a tie-break),
3. then error code ascending.

If an error ties itself across two replica sets, report the one whose name sorts first, so the board stays deterministic.

INPUT:  the same 6-node fleet from Example 1, k = 3

OUTPUT:
  ("WT_CACHE_FULL",       "rs0", 3)
  ("REPL_LAG_EXCEEDED",   "rs1", 2)
  ("INDEX_BUILD_ABORTED", "rs0", 1)

WHY:
  WT_CACHE_FULL       hit rs0-node-1, rs0-node-2 AND rs0-node-3, every rs0 node in
                      the haul, so that one replica set took the full force of it
  REPL_LAG_EXCEEDED   hit 4 nodes, more than anything else, but never more than
                      2 of them inside the same set, so the damage stayed spread out
  INDEX_BUILD_ABORTED never hit more than one node in any single set

  Part 1 put REPL_LAG_EXCEEDED first. Part 2 puts WT_CACHE_FULL first.
  The most widespread error and the most damaging error are not the same error.

Walk the same fleet twice, once looking at how far each error reached and once at how hard it hit a single replica set:

1 / 4

Solution - Part 2

Same single pass, one extra level of grouping.

Step 1: The replica set is already in the node name

No extra input is needed. Split on - and take index 1.

"atlas-rs0-node-1"  ->  ["atlas", "rs0", "node", "1"]  ->  "rs0"
"atlas-rs2-node-1"  ->  ["atlas", "rs2", "node", "1"]  ->  "rs2"

Step 2: Tally nodes per error PER replica set

Part 1 kept one set of nodes per error. Part 2 keeps one set of nodes per error per replica set, so a map of maps of sets. Everything else about the pass is unchanged.

error code             rs0                rs1               rs2            worst set
-----------------------------------------------------------------------------------
WT_CACHE_FULL          3 nodes            -                 -              rs0 -> 3
REPL_LAG_EXCEEDED      1 node             2 nodes           1 node         rs1 -> 2
INDEX_BUILD_ABORTED    1 node             1 node            -              rs0 -> 1
CURSOR_NOT_FOUND       -                  -                 1 node         rs2 -> 1

CURSOR_NOT_FOUND still has its 50 log lines and they still buy it nothing:
fifty lines on one node is one node, in one replica set, that kept its majority.

Step 3: Keep each error's worst set, then rank

For every error, collapse its per-set tallies down to the single worst one, then apply the three sort keys.

code                   key = (-blast, -spread, code)        position
--------------------------------------------------------------------
WT_CACHE_FULL          (-3, -3, "WT_CACHE_FULL")                  1
REPL_LAG_EXCEEDED      (-2, -4, "REPL_LAG_EXCEEDED")              2
INDEX_BUILD_ABORTED    (-1, -2, "INDEX_BUILD_ABORTED")            3
CURSOR_NOT_FOUND       (-1, -1, "CURSOR_NOT_FOUND")               4

rows 3 and 4 tie on blast radius, so spread separates them: 2 beats 1

worst_set = min(by_set, key=lambda rs: (-len(by_set[rs]), rs))

That one line carries the "ties go to the alphabetically first replica set" rule: sorting on (-size, name) and taking the minimum picks the biggest set, and among equals the earliest name.

from collections import defaultdict


def replica_set_of(node_name):
    return node_name.split("-")[1]          # "atlas-rs0-node-1" -> "rs0"


def top_errors_by_blast_radius(fleet, k):
    """Returns (code, worst_replica_set, nodes_hit_in_it) for the top k."""
    nodes_in_set = defaultdict(lambda: defaultdict(set))  # code -> replica set -> nodes
    nodes_hit = defaultdict(set)                          # code -> all distinct nodes

    for node_name, error_codes in fleet:
        replica_set = replica_set_of(node_name)
        for code in error_codes:
            nodes_in_set[code][replica_set].add(node_name)
            nodes_hit[code].add(node_name)

    stats = []
    for code, by_set in nodes_in_set.items():
        worst_set = min(by_set, key=lambda rs: (-len(by_set[rs]), rs))
        stats.append((code, worst_set, len(by_set[worst_set]), len(nodes_hit[code])))

    stats.sort(key=lambda row: (-row[2], -row[3], row[0]))
    return [(code, worst_set, blast) for code, worst_set, blast, _ in stats[:k]]

Complexity Analysis

• Time: O(L + d log d) again, where L is total log lines and d is distinct error codes. The extra grouping level adds a constant factor to the pass, and picking each error's worst set is linear in the number of replica sets it touched, which is bounded by the fleet.
• Space: O(d * r) in the worst case, for r replica sets, since an error can appear in every set.
• The shape of the work has not changed at all between the two parts: one pass to tally, one sort to rank. Only the key changed, which is exactly the point.

Final Notes

1. Count nodes, not lines. A single node stuck in a retry loop can log one error ten thousand times, and flattening the fleet hands that node the top of your board. The set of distinct nodes is what makes a repeat from one machine worth exactly one, and it is the reason Part 1 exists at all.
2. A set and a counter, side by side. They look redundant until you notice they answer different questions: the set says how far an error reached, the counter says how loud it was. Keeping both costs one pass and gives you the tie-break for free.
3. Sort on a tuple, never in stages. Three keys, one comparison, no re-sorting and no fragile "sort by this, then stable-sort by that" chains. Negate the numeric keys for descending and leave the string ascending.
4. The alphabetical key is not decoration. Without it, two errors level on both numbers come out in whatever order the hash map felt like, and the same log haul prints a different board on the next run. Interviewers ask about determinism here.
5. Widespread and dangerous are different words. Part 2's whole content is that a replica set survives on a majority, so three failures inside one set is an outage while three failures across three sets is a slow Tuesday. Knowing that distinction is what separates an engineer who has carried a pager from one who has not.

The reason MongoDB reaches for a problem shaped like this is that the code is never the hard part. Reading a fleet correctly is, and the engineers who do it well are the ones who ask what a number is going to be used for before they write the loop that produces it. 🚀