Hydro Grid Operator Discovers Multiple Unpatched ICS Vulnerabilities During Third-Party Pen Testing

The Challenge

In early spring, Northern Hydro, a mid-sized electricity transmission operator serving several rural regions in Canada, commissioned its annual third-party penetration test. The engagement was part of the organization’s operational resilience assurance program, required under provincial energy oversight and federal critical infrastructure protection expectations. The expectation was routine validation of controls within the organization’s industrial control systems (ICS).

The findings were anything but routine.

Within days, the external testing firm uncovered several severe, unpatched vulnerabilities embedded in Northern Hydro’s legacy ICS environment. These systems, many in continuous operation for more than fifteen years, controlled automated substations and switching gear. The issues included outdated firmware, insecure remote-access configurations, and multiple instances of default administrative credentials on supervisory control terminals.

The most alarming discovery was an obsolete human–machine interface (HMI) that remained connected to both the corporate IT network and the operational technology (OT) environment. The pen testers demonstrated that an attacker could pivot from a compromised office workstation directly into the ICS control layer with minimal effort.

The internal security team scrambled to validate the findings. Many affected assets had no clear ownership records due to the absence of a centralized asset inventory. Some vulnerabilities traced back to third-party maintenance vendors who had installed remote diagnostic connections without complying with the operator’s security policy.

Senior management and the board were immediately concerned, given increasing regulatory attention on energy-sector cyber resilience. The compliance officer noted that, although the test was voluntary, the results could trigger reportability to the provincial energy regulator under critical infrastructure guidelines.

Operational risk was no longer theoretical. Simulated scenarios showed that a coordinated intrusion could disrupt power distribution to more than 200,000 residents, creating the potential for cascading outages. Although no breach occurred, the presence of these vulnerabilities highlighted years of deferred maintenance and unclear accountability across IT, OT, and third-party contractors.

Subsequent discussions revealed systemic challenges: outdated patch management, legacy vendor dependencies, and insufficient segregation between business and control networks. The organization realized that its corporate cybersecurity program had advanced, while OT systems had lagged.

The test served as a wake-up call, exposing critical blind spots in technical controls and governance oversight. In Canada’s energy sector, resilience depends on managing legacy systems as much as defending against new threats.

Our Solution

We led an ICS Risk Containment and Remediation Program aligned with Canadian privacy and cybersecurity obligations (including PIPEDA where personal data and logs were implicated) and with reliability standards adopted by provincial regulators, including NERC CIP-aligned controls where applicable. Core service components were:

– OT incident containment and segmentation: Removal of dual-homed links, establishment of zones and conduits, deployment of one-way gateways where required, and hardening of jump hosts with multi-factor authentication and session brokering.
– Authoritative OT asset inventory and vulnerability baseline: Rapid discovery of PLCs, RTUs, HMIs, relays, and historian servers; mapping firmware, support status, and business ownership; integration with change management.
– Patch and compensating control strategy for legacy devices: Vendor-approved firmware updates where feasible and, for end-of-support assets, allow-listing, strict access control lists, and monitored enclaves.
– Third-party access governance: Updated vendor agreements to enforce security controls, session recording, time-bound access, notification service levels, and evidence of compliance.
– Compliance and regulator readiness: Documented remediation plan, risk register, and evidence pack suitable for provincial regulator review, with decision criteria for privacy reporting.
– Operational assurance: Tabletop exercises for loss-of-visibility and loss-of-control scenarios, and deployment of passive OT network monitoring to baseline normal operations.

The Value

• Risk reduction: A reduction of 85 to 95 percent in identified critical and high OT vulnerabilities within 60 days, moving from 27 critical and 41 high findings to two residual highs with compensating controls.
  – Exposure containment: Elimination of IT-to-OT flat paths across all substations in scope. Lateral-movement exposure decreased by more than 80 percent based on attack-path analysis.
  – Credential hygiene: Removal of all default and shared administrative accounts in OT. Privileged access is now brokered and enforced with multi-factor authentication for remote maintenance.
  – Operational continuity: No unplanned outages during remediation. The organization avoided an estimated 6 to 10 hours of potential interruption risk per substation based on historical detection and recovery benchmarks.
  – Regulatory confidence: Auditable artifacts, including the asset inventory and access-control evidence, closed 12 prior audit observations and improved readiness for regulator inquiries.

Implementation Roadmap

Phase 0: Stabilize (Weeks 0–2)
– Quarantine obsolete HMIs and engineering workstations.
– Disable default credentials and revoke unmanaged vendor tunnels.
– Establish an emergency change window and deploy read-only OT monitoring taps.

Phase 1: Baseline (Weeks 2–4)
– Build the authoritative OT asset inventory with owners, firmware versions, and criticality.
– Conduct configuration reviews, map zones and conduits, and document privacy touchpoints for PIPEDA compliance.

Phase 2: Contain and Segment (Weeks 4–8)
– Implement network segmentation, jump hosts, and one-way data flows where needed.
– Roll out privileged access management and multi-factor authentication for all contractor sessions, with full session logging and retention.

Phase 3: Remediate and Harden (Weeks 6–12)
– Apply vendor-approved patches and firmware. For unsupported assets, enforce allow-listing and strict access control lists, and schedule replacements in the capital plan.
– Deploy configuration baselines and golden images for HMIs and engineering laptops.

Phase 4: Govern Third Parties (Weeks 8–12)
– Update contracts with security requirements, evidence provisions, notification service levels, and rights to audit.
– Certify remote diagnostic tools against internal standards and remove non-compliant methods.

Phase 5: Assure and Report (Weeks 10–14)
– Run tabletop exercises for loss-of-control scenarios and tune alerts in passive OT monitoring.
– Compile the regulator-ready evidence pack, including remediation progress, test results, and logs. Finalize risk acceptance for any residual items.

Phase 6: Sustain (Quarterly)
– Conduct quarterly OT penetration tests on a rotating sample and review vulnerabilities against vendor advisories.
– Perform annual policy and standards reviews to maintain alignment with Canadian privacy and critical infrastructure requirements, and refresh training for operators and contractors.