Emerson’s Adam Boeckmann presented the Power and Water Cybersecurity Suite at the 2017 Ovation Users Group conference. He opened describing the team that formed over the last several years that now supports more than 200 sites in the U.S. alone.

He shared the story of the recent Wannacry/WannaCrypt ransomware cyber-attack. In March of 2017, Microsoft released a patch. A month later, the National Security Agency (NSA) toolkit was leaked. Microsoft released a patch for Windows XP and Windows Server 2003 in May. This ransomware encrypted the files on a PC and demanded payment in Bitcoin to unencrypt.

A Ukraine cyber-attack, crash override, was designed once into a system, established a backdoor, and download a program for activities to happen on the next day. It would modify the control code and ultimately crash the system and render the PCs unable to reboot—basically an inhibit, modify and crash malware program. A solid backup plan with periodic testing to verify that the backup works in one way to recover from these types of attacks.

With NERC CIP, standards exist for low, medium and high risks. Standards organizations help to drive regulations and best practices and processes to help maintain defenses from these types of attacks. The Power and Water Cybersecurity Suite provides technologies and programs to meet the standards and help to apply best practices. Every plant has different requirements based on their expertise and regulatory requirements in the markets in which they operate. The Cybersecurity Suite is modular to be able to provide the modules required.

The suite includes scheduled security services, security assessments, compliance services, network services, incident response services, on-demand consulting and unplanned on-site security services. Technologies in the suite include antivirus protection, patch management, application control, device control, security incident & event management (SIEM), system backup and recovery, vulnerability assessment, network intrusion detection, rogue system detection, and change management.

Adam contrasted blacklisting from whitelisting. Blacklisting is a list that excludes known malware from running. It must be known to be added to the list. On the other hand, whitelisting is putting all the known good applications in a list and preventing anything not in the list from running.

He described rogue detection intrusion where it sniffs out and records all the connected devices and builds and asset inventory of network connections. It looks for changes outside of normal communications to alert users or service providers to users of this software. Tripwire manages changes in configuration files from the integrity of these files, notifies users of file changes and runs autonomously.

Here’s a link to find out more about ICS cybersecurity with the Power and Water Cybersecurity Suite.

The post Suite of Cybersecurity Technologies and Practices appeared first on the Emerson Automation Experts blog.

Emerson’s Ben Skal presented on reducing the Ovation distributed control system hardware footprint at the 2017 Ovation Users Group conference. Virtualization technology is at the heart of this hardware footprint reduction and having less workstation and server hardware to maintain. Ben’s presentation focused on:

• What is virtualization & how it works
• Appropriate uses of virtual machines (VMs)
• Benefits of VM environments in increased redundancy & reliability, reduced footprint and recovery from equipment failures
• Requirements to move from traditional workstations & server to VM

Ben opened describing virtualization which separates the operating system from the physical hardware. Multiple virtual machines can run on physical services. Server host clusters have a management console that oversees the cluster of servers on which the VMs run.

Traditionally separate PCs run database servers, historians, engineering stations, operator stations, etc. In a virtual environment, these physical workstations run as VMs in a server cluster with thin clients running remote desktops into the VMs. This configuration is recommended for non-critical applications since the loss of a server cluster would cause a loss of all the VMs running in it.

To avoid this situation, high availability virtualization architecture increases reliability has the data stored in storage area network (SAN) storage devices. If a virtual host is lost, the other virtual hosts automatically reboot on the other virtual hosts.

All networks and interfaces are fully redundant including the Ovation highway, management network, remote desktop network and each Ovation VM is configured with two virtual network interfaces. The hardware is based on Dell server technology with RAID 6 hard drive redundancy. The virtual environments for the VMs and thin clients are based on VMware. Since the thin clients do not contain local storage, they do not require patch management.

With high availability virtualization lifecycle maintenance is simplified with hardware decoupled from software, faster recovery from workstation failures and extended system life by avoiding hardware obsolescence. Deployment and maintenance are more flexible since applications such as database server can be accessed from multiple locations on the network. Cybersecurity protection is also simplified since the VMs are located together and can be patched and updated together.

The post Reducing Control System Hardware Footprint appeared first on the Emerson Automation Experts blog.

In a Water Online article, Point Versus Continuous Level Measuring Technologies, Emerson’s Lydia Miller provides guidance on the technologies and which ones are best suited for particular applications.

Lydia opens noting that level measurements are not just for determining the height of the liquid in vessels, but also are used to calculate volume, flow rate, and as inputs for pump control.

She describes the range of common level measurement technologies:

Point measurement technologies include float switches, vibrating forks, capacitive, and others. Continuous level technologies include radar, ultrasonic, magnetostrictive, capacitive, float-and-tape, differential pressure, and others.

Point-level measurements are often used because:

They’re simple, comparatively cheap and have been used for years and continuous readings aren’t always necessary. Point level device outputs are simple on/off, so it keeps everything as straightforward as possible.

Technology has advanced where some types of point-level switches, such as vibrating fork switches can have diagnostics as proving capabilities when used in safety instrumented system applications:

Some of the newest vibrating forks have the ability to do remote proof testing and continuous health monitoring. There are switches with HART communication that can use frequency monitoring to know if the switch is in oil, water or alcohol. They can detect settled sediment within a liquid and can even detect the presence of foam.

Lydia highlighted advantages continuous level measurement devices had over point level measurement devices. Levels can be set in the control system instead of manually at the device, use of a single device instead of two for high and low level indication/control/alarming/etc. points, changing level setpoint to prevent scum line buildup, use in real-time volume calculations, and more.

Technologies such radar (guided wave & non-contacting) and ultrasonic:

…stay above the liquid and are not wetted at all… Radar and ultrasonic instruments read from the top down, with ultrasonics being non-contact and radars having both a contacting and non-contacting version. Ultrasonic and pulse radar technologies send sound or microwave energy from a transducer toward the liquid, and calculate distance by timing how long it takes for the pulse to be reflected back.

Read the article for more on the advantages of the level measurement technologies and data they can provide in improving operational performance through early problem detection, process flexibility, and increased efficiency.

You can also connect and interact with other level measurement and water & wastewater industry experts in the Level and Water and Wastewater groups in the Emerson Exchange 365 community.

The post Point and Continuous Level Measurement in Water and Wastewater Applications appeared first on the Emerson Automation Experts blog.

During the week of July 29 through August 3, I’ll be attending the Ovation Users’ Group Conference in Pittsburgh. This users’ group is composed of people working in the power, water and wastewater industries. If you’re in one of these industries and an Ovation distributed control system user, I look forward to seeing you there.

I’ll highlight a few things that Ben presented at last year’s Ovation Users’ Group conference regarding alarm management to give some flavor for what you might see—and in case you can’t join us. His presentation covered the key objectives of alarm management, the alarm management process, alarm terminology, to IEC 62682 and ISA 18.2 (which were advances from the precursor standard EEMUA 191) alarm system benchmarks, alarm design, and the Ovation Alarm Health Advisor.

There are many key objectives in effective alarm management. Some include building operator confidence and trust, reducing and eliminating non-value alarms, organizing/classifying/prioritizing alarms, eliminating chattering and frequent alarms, and managing alarm floods and root-cause data.

Alarm management is a continuous improvement process that begins with an overall philosophy of how the plant will be operated. It then goes through the steps of identification, rationalization, design and implementation. From there, the alarm strategy is put into operation and ongoing maintenance. Periodic monitoring and assessment of how these alarms are performing for effective and prompt operator actions helps to identify changes that need to be managed and made.

Like so many other areas of process automation, alarm management has its own set of terminology. Some of the frequent terms and phrases include alarm rationalization, alarms & alerts, alarm classification, grouping by plant areas, regulatory notifications, alarm prioritization, alarm limits, deadband, off-delay, on-delay, standing/stale alarm, alarm flood, and first-out.

Many of these are defined with associated metrics by the alarm management standards described above. The Ovation Alarm Health Advisor examines alarm data, collected by the Ovation Process Historian, and provides the user with information necessary to minimize unnecessary alarms. This, in turn, enables plant operators to more effectively address real, high-priority issues and focus on actionable items.

If you’re not able to join us in Pittsburgh, make sure to follow @OvationUsers on Twitter to get the latest news and trends on alarm management, ICS cyber security, power plant simulation, microgrid controllers, gas/steam/wind turbine controls and more. You can also connect and interact with other experts on the Ovation Users’ Group site or in the Ovation group in the Emerson Exchange 365 community.

Update: Updated post to reflect IEC 62682 and ISA 18.2 standards.

The post Improving Control System Alarm Effectiveness for Operators appeared first on the Emerson Automation Experts blog.

Chlorine is an important disinfectant whether used in swimming pools or the water that flows into our homes and businesses. For water treatment facilities, water treatment monitoring is one of the most important facility operations. Meeting regulatory requirements for residual chlorine is critical for public safety as well as for facility personnel.

This 3-minute YouTube video, Enhance Chlorine Analysis to Reduce Water Treatment Costs, shows how the Rosemount FCL Free Chlorine Measuring System measures free chlorine in fresh water.

This system does not use expensive sample conditioning systems or reagents to control pH. Instead, the analyzer automatically compensates for changes in the pH of the sample. Amperometric chlorine sensors combined with pH sensors provide simple and accurate readings of free chlorine levels that meet the U.S. Environmental Protection Agency (EPA)’s 334.0: Determination of Residual Chlorine in Drinking Water Using an On-line Chlorine Analyzer.

The system contains a Rosemount 3900 pH sensor and a Rosemount 499ACL Free Chlorine sensor. Together they work to accurately adjust the measurement based on process pH fluctuations. These sensors combined with the Rosemount 56 dual-channel transmitter provide a local display with data logging, diagnostics, and digital communications (HART or Profibus DP) back to the control system.

You can connect and interact with other water monitoring and analyzer experts in the Analytical and Water & Wastewater groups in the Emerson Exchange 365 community. Also, meet them face-to-face at the October 1-5 Emerson Exchange conference in San Antonio, Texas.

The post Accurate and Efficient Water Treatment Chlorine Analysis appeared first on the Emerson Automation Experts blog.

Emerson’s Mircea Lupu shared his thoughts on Ovation system algorithms for handling processes with long lag times—a control challenge that vexes many.

Mircea opened noting that control algorithms are at the heart of any control system to provide stable and accurate response and robustness against disturbances and process changes. Tuning these algorithms to achieve the desired performance can be laborious.

Proportional-Integral-Derivative (PID) control is the workhorse algorithm but has difficulty with processes with time lags. There are solutions to address these lags but setting and maintaining tuning can be tedious. From a loop stability standpoint, the integral gain in a PID loop works against the time delay.

The derivative gain in a PID helps if tuned in a proper range. Too little or too much causes oscillation in the loop. In summary, proportional gain should be decreased, integral reset time should be increased with time delay and derivative gain may increase with time delay, but too much may destabilize the loop.

A Smith-Predictor (PREDICTOR) algorithm uses an internal model of the process to remove time delay from the control loop. The control algorithm will act on the predicted process variable rather than the actual (delayed) process variable. If the prediction is good, larger proportional gains and smaller reset times can be used. This increases the stability and robustness and improves transient performance. The performance of the loop is only as good as the prediction.

The Ovation Advanced Process Control (APC) Toolkit is an extension of the Ovation system that allows advanced algorithms to be added to an advanced controller.

Predictive algorithms solve an optimization problem with constraints to compute the manipulated variable (MV). This algorithm is ideal for supervisory control or setpoint generators for fast (low-level) feedback loops. This control block does multivariable process control—multiple input and multiple output (MIMO) applications.

This predictive algorithm is based on a linear quadratic optimization function with optimization constraints.

Mircea also discussed control with a cascaded loop configuration. In this arrangement the primary (upstream) controller drives the setpoint of the secondary (downstream) controller. This type of control strategy is recommended for processes with slow dynamics in which a relatively fast process has to be manipulated to control the slow process. This isolates a slow control loop (outer loop) from non-linearities by using an inner (fast) loop.

The disadvantages of cascaded control loops are the added measurement (PV) for the inner loop and having to tune the additional control algorithm. When tuning these loops, the inner loop has to be faster than the outer loop at least by 3X. Start by tuning the inner loop first and then the outer loop.

Mircea summarized his presentation reiterating the control challenge of processes with long lag times, such as those found in water & wastewater treatment facilities. You must first understand the dynamics of the process to design a stable and robust control strategy. Alternatives to PID control include PREDICTOR algorithms and model predictive control. The Ovation DPATUNE tool helps for model estimation. Finally, proper tuning techniques are important, especially in cascaded control loops.

The post Handling Processes with Long Lag Times appeared first on the Emerson Automation Experts blog.

Deadly hydrogen sulfide (H₂S) can be present in industries such as oil & gas production and wastewater treatment. The Wikipedia Hydrogen Sulfide page highlights the danger:

Short-term, high-level exposure can induce immediate collapse, with loss of breathing and a high probability of death. If death does not occur, high exposure to hydrogen sulfide can lead to cortical pseudolaminar necrosis, degeneration of the basal ganglia and cerebral edema.^[31] Although respiratory paralysis may be immediate, it can also be delayed up to 72 hours.^[38]

In a Water Online article, Avoiding The Deadly Legacy Of Hydrogen Sulfide, Emerson’s Sean McLeskey is quoted:

Sean McLeskey

“H₂S gas is naturally present and produced in wastewater treatment… It is found in raw sewage and throughout sludge processing stages, so it’s critical to monitor for the gas in sewers, sludge de-watering systems, anaerobic digesters, and wet/dry wells.”

As Sean indicates, continuous monitoring for H₂S is critical for personnel safety. Solutions:

…like Emerson’s Millennium II Series of universal fixed gas detection transmitters, combined with its range of toxic and combustible gas sensors, can be applied to combat the threat of leaking H₂S.

Advancements in technology and communications standards have enabled the development of wireless gas monitoring sensors. One example is:

…the Rosemount 928 Wireless Gas Monitor [hyperlink added]. In some plants, it can be cost prohibitive to install and operate conventional gas detection systems at these sites due to geography and infrastructure. The installation, wiring, and commissioning costs for each additional wired device can add tens of thousands of dollars to the instrument’s total installed cost.

Read the article for more on these technologies improve personnel safety and how these technologies work. Visit the Wireless Gas Detectors and Sensors section of Emerson.com for more on the technologies and applications.

You can also connect and interact with other gas monitoring experts in the Measurement Instrumentation group in the Emerson Exchange 365 community.

The post Monitoring Hydrogen Sulfide for Personnel Safety appeared first on the Emerson Automation Experts blog.

Peter Gabor

Water & wastewater utilities can improve efficiency using Industrial Internet of Things (IIoT) technologies. In a WaterWorld article, Leveraging IIoT for Energy Management, Emerson’s Peter Gabor participated in a question & answer session to share his thoughts on how these utilities are incorporating IIoT.

Peter noted that energy costs are a big part of total operating costs and driving down energy consumption is an opportunity where IIoT technologies can be applied.

When looking for opportunities to improve energy usage, he explained:

For energy usage we often recommend customers start at the incoming power source and work toward the individual assets. Monitoring switchgear is often the easiest as most equipment has this information readily available, but it provides no granularity of the downstream devices being powered. Monitoring individual equipment provides the visibility on usage and efficiency degradation over time.

IIoT sensors lower the installation economic barriers to enable more equipment to be monitored. But, it’s important that transmission of this additional data be focused.

The data needs to be presented to the individuals who have the responsibility to act on it. Right now, we often see them pulling in the data just to monitor and there is no context to it.

There are benefits to data consolidation with cybersecurity as a key focus.

Bringing the data into one central location paves the way for information sharing, data analytics, performance monitoring, and optimization. Cybersecurity should be at the forefront to ensure the data feed is reliable. Once the connection is protected, then the data quality and accurate time stamping of the data received should be considered, which includes verifying the reliability and accuracy of the instrumentation installed.

Read the article for more on considerations for an IIoT architecture and how IIoT technologies enable more comprehensive and effective energy management solutions.

Visit the Water & Wastewater sections on Emerson.com for more on solutions to improve energy and operational efficiency. You can also connect and interact with other IIoT and water & wastewater industry experts in the IIoT & Digital Transformation and Water & Wastewater groups in the Emerson Exchange 365 community.

The post Energy Management with IIoT for Water and Wastewater Utilities appeared first on the Emerson Automation Experts blog.

Emerson’s Mike Ames and Nick Koonce teamed up to discuss Ovation Lifecycle Planning and the need to think beyond periodic upgrades at the 2019 Ovation Users’ Group conference. Nick opened by defining lifecycle planning for the Ovation system:

Ovation Lifecycle Planning provides a broader or more holistic approach to best align Ovation products and services to customers’ business drivers (KPI’s) and problem-solving needs resulting in optimum system/asset performance & reliability and reduced total cost of ownership.

To better facilitate this lifecycle planning process, tools are available, such as lifecycle planning & guides aligned to customers business objectives, multi-year, multi-site implementation planning, comprehensive budget and project planning, and awareness presentations.

Total cost of ownership includes lifecycle planning, spares, repairs, service & engineering, proposal accuracy, budgeting, justification, po issuance, outage coordination, inventory, project management, startup, schedule change, change notices, performance requirements, optimization, security, service; response and quality.

Risk and cost assessments on the control system include system or component failures, threat of pending downtime or incidents, support unavailable or high cost (spares, repairs), incompatibility with new beneficial technologies, incompatibility for implementing new regulations, and lack of capacity or expansion capabilities.

From a benefits and optimization perspective, analysis should include increased reliability, new technology benefits, regulatory compliance, extend benefits fleetwide, improvements in KPI’s, faster profitability, increased profit, superior manageability, procurement leverage, and performance optimization.

Nick described the Ovation Advantage program which justifies the replacement of PLCs with Ovation Compact Controlllers. This reduces complexity and maintenance costs by eliminating PLCs and using the power and capacity of the existing Ovation control system.

One of the most important things for lifecycle planning is to engage the executive and decision makers. It’s important that they understand that industrial control systems are based on commercial off the shelf technologies that must be updated over time compared with proprietary control systems from the prior era. Component lifecycles range from 20 years for I/O systems and subsystems to less than 5 years for security software and solutions.

With a 10-year Microsoft support policy for each version of their operating systems, Ovation software needs to fit within this window and include development, testing and distribution. Other planning fundamentals include defining key parameters and timing and considering age and lifespan of existing components.

Working with Emerson, a customized lifecycle plan includes a strategic 5-10+ year plan, OEM recommendations, detail BOM, pricing & schedule, lowest cost alternatives, CAPEX/OPEX alternatives, SureService contract alignment, and justification support. The planning process to arrive at this plan includes a phase 1 kickoff to identify scope, goals and objectives. Phase 2 is to do a site assessment and data collection by performing interviews with key personnel, KPI alignment and system health assessment. Phase 3 is performing analysis and documentation and phase 4 is a final plan submittal by Emerson Lifecycle Services team and review with the end user.

With the plan in place, it is budgeted and executed over time to keep the system delivering the performance required to meet the business objectives and key performance indicators. Visit the Ovation Lifecycle Services section on Emerson.com for more on developing and executing a comprehensive lifecycle plan.

The post Ovation Lifecycle Planning appeared first on the Emerson Automation Experts blog.

There are many paths to pursue in improving cyber defenses for industrial control systems (ICS). At the 2017 Ovation Users Group conference, Emerson’s Tom Kizer presented on applying threat intelligence for system defense.

Tom opened by acknowledging that identifying effective threat intelligence is difficult. Threat intelligence is data collected, organized, analyzed and refined about potential or current attacks such as indicators of compromise, zero-day threats, advanced persistent threats (APTs) and exploits. Threat intelligence helps users understand the risks of the most common and severe external threats.

With control system software, the amount of software is limited which helps with the task. External sources such as McAfee has rule sets and policies that are one source of gathering threat intelligence. Tom noted that the SureService program for Ovation users puts out notices on relevant threats which might impact the software loaded on these systems. Other external sources include SANS, Threatstop.com, AlienVault and Anomali STAXX.

Indicators to monitor include accounts: lockouts by asset and user, activity in accounts of former staff, activity on the same asset with different user names in a short time frame, outside of hours’ logins, privilege account changes, repeated unsuccessful logins, and the creation and deletion of assets.

More indicators include configuration changes at the system and application level that no one can explain, external activity on commonly hacked network ports, login and access logs, intrusion detection system (IDS) events and traffic between test and development or live environments.

The tools Tom and team normally use are security information and event management (SIEM) and Intrusion Detection systems.

Once gathered, the learnings from the threat intelligence should be applied to firewalls, IDS, vulnerability management, SIEM, host security systems, application security systems, identity and access management and analytics platforms other than SIEM. These tools provide data but not necessarily the intelligence to make decisions. Intelligence requires analysis by people, perhaps augmented with advanced analytics applied by software.

ICS-CERT is a good source of cyber threat data related to control systems. Tom recommended you aggregate this data from all possible sources and correlate this aggregated data. Developed policies and procedures need to address regular updating of supplier feeds, reviewing and analyzing the data on a regular basis, regular tuning of monitoring rules and incidence response. There is an internal team that regularly monitoring the ICS-CERT, EnergySec and many others and evaluate what should be sent on to Ovation users as an alert.

Tom wrapped his presentation with a discussion of SHODAN, a search engine for the Internet of Things. This search engine targets specific ports—HTTP, HTTPS, SSH, FTP, Telnet, SNMP and RTSP. Performing regular searches for your organization is good practice.

Threat intelligence is about collecting, organizing and analyzing the data and refining your technology and work practices to contend with potential and current cyber threats.

The post Applying Threat Intelligence for Industrial Control System Defense appeared first on the Emerson Automation Experts blog.