What physically removing Salt Typhoon actually takes — and what stops the next one.

The constraint that reshapes every operational decision: simultaneous eviction

Before sequencing any of the steps below, the load-bearing operational fact from CISA AA25-239A:

Three implications follow from that sentence, and they govern everything else on this page:

(1) Discovery comes before remediation. Inventory first — every device in the affected estate, every iBGP peer, every TACACS+ client, every device that ever held a session with a compromised neighbor. Then plan the cut.

(2) Out-of-band coordination is required. Salt Typhoon was observed compromising mail servers and admin accounts.1 Eviction comms cannot route over the same admin email or chat systems the actor may be watching. Use a separate signal-flare channel (clean PKI-backed bridge call, paper SOP at each POP, or a dedicated incident-response collaboration tool not linked to compromised AAA).

(3) The fleet is cut as one operation. Per-device sequential cleaning lets a foothold on device B watch the IR team clean device A and entrench further before B’s turn. For a tier-1 carrier with thousands of POPs, that means a coordinated multi-team blackout window — engineering, ops, vendor TAC, federal IR support, and (where lawful intercept is in scope) FBI / FCC oversight.

Per-device eviction sequence

Once discovery is complete and the simultaneous cut-window is scheduled, each compromised device walks the following sequence. Steps are ordered so that forensic value is preserved before any state-destroying action.

1. Quarantine the production data plane; keep the management plane reachable.

   Drop the device’s production-side BGP / OSPF / ISIS peering at the next hop (upstream router-level ACL or BGP route-policy) so the GRE tunnel and any traffic-mirror exfil stop. Do not power-cycle — volatile state (running processes, in-memory implant artefacts, GRE tunnel counters) is forensic evidence. Keep mgmt-VRF reachable via the OOB console server for the next steps.

2. Capture volatile state and running configuration to forensic storage.

   From the OOB session, pull the full snapshot. Treat the output as evidence: ship it to forensic storage with a cryptographic hash and a chain-of-custody log before continuing.

   ! On the device, into an OOB-reachable scp/tftp destination.
   show tech-support
   show running-config
   show startup-config
   show ip interface brief
   show ip route vrf *
   show users | include 5772[0-9]     ! sshd_operns port range
   show tcp brief | include 5772[0-9]
   show ip socket
   show aaa servers
   show snmp user
   show running-config | include username
   show running-config | include tunnel mode gre
   show guestshell
   show platform integrity sign nonce 1      ! image integrity report

3. Audit for the Talos persistence-kit indicators.

   Compare the captured config against the carrier’s golden template. The list is exhaustive — every one of these is documented in the Talos IR writeup2:

   • GRE / IPsec tunnels — any interface Tunnel with tunnel mode gre ip not in the golden config.
   • Alternate-port sshd — Linux-side sshd_operns on TCP 57722 or any non-standard SSH listener.
   • AAA / TACACS+ server IP changes — tacacs-server host or aaa group server tacacs+ not matching the carrier’s declared AAA endpoint.
   • SNMP community-string changes — snmp-server community lines beyond the carrier’s rotation list.
   • ACL modifications — new permit entries upstream of deny any any log.
   • Created local users — any username not in the golden user list.
   • Guest Shell enabled — guestshell enable in running-config, plus Linux-side /etc/passwd and authorized_keys audit.
   • Loopback interface changes — new loopbacks or IP changes on existing loopbacks.
   • Log scrubbing artefacts — missing entries in .bash_history, auth.log, lastlog, wtmp, btmp compared with the carrier’s centralized syslog mirror.
4. Verify image integrity against Cisco’s published hash.

   Run verify /sha512 flash:<running-image> on the device and compare against the SHA-512 hash Cisco publishes for the running release at software.cisco.com. A mismatch is the trigger to escalate to Cisco TAC for ROMMON-level inspection3 — defenders cannot independently validate ROMMON integrity per MITRE T1542.004.

5. Escalate to Cisco TAC for ROMMON validation (if hash mismatch or any persistence indicator from step 3).

   ROMMON / golden-ROMMON image verification is a vendor-side operation on Cisco IOS XE. If TAC confirms ROMMON modification, the device is in MITRE T1542.004 territory and the only remaining options are platform-supported secure factory reset (next step) or physical replacement (see “When the network can’t reach the device” below).

6. Reflash to a known-good vendor-signed image — chosen by platform support level.

   Three cases:

   Case A — IOS XE 17.18.1a or later

   factory-reset all secure clears SPI NOR FLASH, the config-register, and ROMMON variables.5 Confirm BIOS-protection is enabled so golden ROMMON is write-protected for the reflash. Reload with the vendor-signed image and verify the post-reset SHA-512 hash against Cisco’s published value.

   Case B — IOS XE prior to 17.18.1a, or IOS Classic / NX-OS

   Network-only reflash is insufficient for the persistence layer. Standard factory-reset clears running and startup configuration but leaves ROMMON variables on platforms without the secure mode. Console-cable access from an OOB console server (or, if console isn’t reachable, a physical site visit) is required to boot into ROMMON, TFTP a vendor-signed image, and force-write it. This pulls the device out of the “network reachable” bucket and into the next section.

   Case C — EOL hardware (legacy Catalyst 6500 line cards, IOS 12.x ISR families, EOL NX-OS chassis)

   Physical replacement is the only fully-trusted option. Vendor signed images may no longer be available for the platform, ROMMON-level validation is not supported, and the carrier carries depreciated-asset risk on top of the security risk. The compromised device goes to a forensic lab; a replacement device is racked under the prevention-controls baseline (see below) before the cut-window closes.

7. Rebuild from the golden configuration template — not from the device’s saved startup-config.

   Any backup or startup-config produced during the residency window is presumed contaminated. Push the carrier’s declarative golden template via the configuration-management system (Ansible / NSO / Salt) with per-device overlays from a source-of-truth IPAM. The golden template carries every prevention control from the section below.

8. Rotate every credential that ever touched the device.

   Local secrets: enable secret, every local username … secret (Type 8 / PBKDF2-SHA-2566), line passwords, SNMP communities (or migrate to SNMPv3), BGP / OSPF / EIGRP authentication keys, IKEv2 / IPsec pre-shared keys. Cryptographic material: crypto key zeroize rsa followed by crypto key generate rsa modulus 4096 for a fresh SSH host keypair, plus a fresh device identity certificate. Server-side: every TACACS+ / RADIUS shared secret used for this device rotates, and every service account credential used by mgmt tooling rotates as well.

9. Lateral audit — every device with a trust relationship to the cleaned device gets the same flow.

   iBGP peers in the same AS. OSPF area neighbors. ISIS L2 adjacencies. MPLS LDP sessions. TACACS+/RADIUS clients sharing the same AAA server. SNMPv3 EngineID trust. Each is assumed contaminated until proved otherwise, and the proof is the same nine-step sequence.

10. Telemetry burn-in (30/60/90 days) with the carrier SIEM watching for re-pivot.

    NetFlow / IPFIX on every edge interface exported to behavioural analytics, syslog to append-only SIEM (so on-device log clearing doesn’t lose history), config-drift alerts on every line of running-config. Triggers: GRE tunnel creation anywhere in the affected estate, guestshell enable, any new high-port SSH listener, any new entry in show running-config | include username, AAA-server-IP change.

11. Validation: carrier PSIRT sign-off corroborated by CISA hardening-guidance lead.

    This is the documented work product Milestone (3) of the open-letter contract is structured to pay on. The reason it has to be a carrier-and-government joint sign-off, not a vendor self-declaration: per the eviction-scope page, US officials publicly disputed every prior carrier “purged” claim through 2024–2025.7

When the network can’t reach the device: physical access and replacement

Three real-world cases force a transition from the network-reachable runbook to physical access. For each, the operational shape is different.

Console-server reflash from the OOB management network

If the device is reachable on the OOB mgmt network but the production data plane is fully compromised (running-image SHA mismatches, ROMMON-modification suspected, or a Cisco TAC engagement is required to mediate), the next step short of a truck roll is OOB console access. Tier-1 carriers typically run a console-server deployment (OpenGear, Lantronix, Avocent, Cisco TS) reachable through a separate routed mgmt fabric with its own AAA and its own physical fibre or T1-back-haul path. The runbook from this point:

1. Open a console session to the device’s physical console port through the OOB server.

   Production-side connectivity remains quarantined; only the serial-console path is in play. AAA on the console server itself rotates as part of step 8 above — the console-server credentials are an attractive lateral-pivot target.

2. Reload the device, break into ROMMON during the boot countdown.

   Ctrl-Break or Ctrl-C per platform documentation. From the ROMMON prompt: tftpdnld a vendor-signed image from a TFTP server reachable on the OOB fabric. Set BOOT variable to the new image, write to ROMMON, reload, verify.

3. From the freshly booted clean image, push the golden config (step 7 above) and continue with credential rotation (step 8 above).

   The remainder of the per-device sequence resumes normally once the device is back on a known-good image.

Truck roll — physical site visit to the POP, central office, or remote installation

Physical site visit is required when (a) the OOB console fabric is itself compromised and can’t be trusted for the reflash, (b) the device’s serial console port is not on the OOB fabric (legacy POPs), or (c) Cisco TAC’s ROMMON-validation engagement returns “cannot remediate in software — RMA.” The site visit itself:

A carrier ops engineer travels to the physical location with: a sealed replacement device pre-staged with the golden config (or a USB-stored vendor-signed image for in-place re-flash from console), a tamper-evident transport bag for the compromised device, a portable console-cable kit (USB serial console + a known-good laptop, not the engineer’s daily-driver), and the chain-of-custody paperwork for the seized hardware. The compromised device’s exit path is to a forensic lab — Cisco TAC’s hardware analysis unit, and (for federally-implicated breaches) CISA’s Cyber Hunt and Incident Response Teams or NSA’s Cybersecurity Collaboration Center.

For tier-1 U.S. carriers, the operational scale of physical access is its own constraint: Verizon, AT&T, T-Mobile each operate thousands of central offices, wire centers, headends, and aggregation POPs distributed across every U.S. state and territory. A nationwide simultaneous cut-window requires standing field-engineering capacity that no single carrier maintains as a baseline. The federal brokering Milestone (3) calls for is precisely this: coordinating mutual-aid between carriers, the National Guard signal units, and Cisco TAC field engineers so a cut-window of acceptable scope can actually be staffed.

Hardware replacement — when nothing short of RMA suffices

End-of-life platforms (IOS Classic 12.x ISR families, Catalyst 6500 with legacy supervisor engines, EOL NX-OS chassis on older silicon) are not safely recoverable in the field. Vendor doesn’t support secure factory reset, ROMMON validation isn’t available, and signed images may have stopped shipping. The replacement path:

1. Procure replacement on the supported platform baseline.

   The carrier’s declared baseline is Cisco IOS XE 17.18.1a or later (or the equivalent NX-OS / IOS XR supported release), so the replacement device has factory-reset all secure semantics and BIOS protection for golden ROMMON write protection5 from day zero.

2. Stage on a known-clean network with the golden config applied, prevention controls baseline, image and ROMMON hashes verified, full credential set rotated.

   The staging network must not have ever held a session with the compromised device. Use a physically separate staging lab fabric, not a logical VLAN on the production fabric.

3. Field swap during the cut-window.

   Cable migration, BGP / IGP peering brought up with the new authentication keys, validation of routing convergence and traffic restoration. The compromised device transports to forensic lab in a tamper-evident bag.

The signaling and lawful-intercept layer: what no edge-device patch can touch

Salt Typhoon’s observed objectives reach beyond IP-layer edge routers. The most consequentially-cited compromise category is lawful-intercept (LI) / CALEA infrastructure and the SS7 / Diameter signaling planes that ride beneath the data-plane stack. Eviction at these layers does not look like “reflash the device” — it looks like “rebuild the system under federal supervision.”

SS7 and Diameter signaling

SS7 (legacy 2G/3G) and Diameter (4G/5G control plane) are the protocols carriers use for call setup, SMS delivery, subscriber location, and roaming. They lack origin authentication by design; any peer connected to the SS7 international transit network can issue queries that the receiving switch will honor. Salt-Typhoon-class abuse at this layer involves inserting tap points or fraudulent signaling sources at HLR / HSS, MSC, MME, or PGW boundaries. Defenses are architectural, not patches:

• SS7 firewalls / STP screening — e.g., Mavenir, Adaptive Mobile Security, Cellusys deployments at the SS7 interconnect boundary, scoring incoming Category 1 / 2 / 3 messages against GSMA FS.11 / FS.19 filtering rules.
• Diameter Edge Agent (DEA) screening — Diameter signaling firewall between the carrier’s Diameter Routing Agent (DRA) and the IPX / roaming hub, validating origin and applying GSMA FS.19 controls.
• GSMA roaming-hub authentication controls — partner-validation policies enforced at the IPX boundary.
• Replace legacy — per public reporting, the SS7 / Diameter layer “will take years (and billions of dollars) to replace.”7 5G Standalone (5G SA) with Service-Based Architecture and HTTP/2 transport between Network Functions is the structural fix, but the migration is multi-year.

Lawful intercept (LI) / CALEA systems

CALEA-mandated LI infrastructure is the highest-sensitivity carrier asset: by statute it routes copies of subscriber communications to law-enforcement agency (LEA) collection systems. A Salt-Typhoon-class compromise of an LI mediation device gives the actor real-time access to the same content the FBI sees. Remediation is correspondingly heavyweight: full rebuild and recertification under FBI / FCC supervision, with new HSM-backed keys for the LEA delivery encryption, new mTLS certs for the LEA-to-carrier session, and a complete audit of every active intercept order from the residency window onward.

Forward prevention: the controls that stop the next Salt Typhoon

The eviction sequence above documents what is required when prevention has already failed. Forward prevention is cheaper by orders of magnitude. The controls below are split into upstream (where Bulwark plugs in — the research workflow that shortens the variant-to-patch loop), device-level (the running-config baseline every edge device should ship with), and architectural (the carrier network’s overall posture).

Upstream — the research workflow Bulwark proposes to run

Device-level — the running-config baseline every edge device ships with

Architectural — the carrier network’s overall posture

How Fulcrum and Bulwark fit into the runbook

The runbook above is mostly carrier ops, vendor TAC, and federal IR. Fulcrum and Bulwark are not on the per-device eviction path. Their contribution is upstream:

Bulwark’s continuous-research loop is the input that makes the prevention stack actually load-bearing. A carrier can have the cleanest config in the world — no vstack, OOB mgmt, FIDO2 admin auth, Type 8 secrets — and still be compromised the day after a new vendor-disclosed bug class drops, if no one is patch-diffing for the variants and disclosing them ahead of weaponization. The 14-month gap between CVE-2023-20198’s disclosure and Salt Typhoon’s weaponization of it is exactly the gap continuous variant-research is designed to close.

Fulcrum is the same loop, operator-driven, for the per-engagement procurement-side variant of this work — the cases where a known target advisory needs deep workup with a human in the seat, with the same coordinated-disclosure terminal.

Neither product runs on a victim network. Neither rotates carrier credentials. Neither reflashes ROMMON or signs a TAC ticket. The runbook above is the work product Milestone (3) of the open letter would fund — the carrier-and-federal operation that turns “variants no longer slip through the inflow side” into “a tier-1 carrier signs off that Salt Typhoon is no longer resident,” corroborated by CISA’s hardening-guidance lead. That sign-off is what the eviction reward exists to make happen, and what no purely-technical vendor can credibly promise alone.