System Monitor

Configuration

System Monitor uses a pair of INI files for configuration: Systems.ini and TaskList.ini

The first file, Systems.ini, contains some operating parameters for System Monitor, a list of the managed subsystems, by name, and a section for each subsystem that specifies subsystem criticality and the list of managed PICS applications that will be run on that subsystem.

NOTE:

In PICS, we use the term subsystem to describe a primary/backup pair of computers that will be performing the same function. Each computer within a subsystem is referred to as a node. Everything together makes up a Plant Integrated Computer System (PICS).

The second file, TaskList.ini, contains definitions of every managed task in PICS. These task definitions include the name and location of the program file on the node, task criticality and a list of other tasks that provide a service necessary before the task being defined may be started. The task dependencies may cross subsystem boundaries (for example, the static database system is designed like a tree, and as we all know, trees require roots before they may have any branches).

Operation

When configured to operate in SOLO mode, any subsequent instances of System Monitor will be instructed to terminate. When configured to operate in primary/backup mode, System Monitor will first search for a primary. If two searching System Monitors find each other, they arbitrate selection of a primary and the other becomes the backup. The INI files on the primary are considered correct (assuming they were readable and correct, otherwise the node will be restarted). The backup will check the CRC32 of each file against the values computed by the primary to validate the backup INI files. If the backup INI files are different, the node will be rebooted.

Once the primary System Monitor is functional, it starts listening for Task Monitors that are searching for the primary System Monitor. Each Task Monitor is informed the name and address of the current primary System Monitor, at which point the Task Monitor will request an operation to perform. System Monitor will send task definition blocks to the Task Monitor when the task's requirements have all been met. The actual sequence that tasks are started is determined by System Monitor when it processes the INI files, based on the dependencies for each task, and the order they are listed in the Systems.ini file (when the dependencies are the same). When all of the managed tasks scheduled for a managed node are reported as running, System Monitor sends a "load complete" message to Task Monitor, indicating that the node is now consider to be in a running status. (Until then, the node was marked as starting and any task failure would result in a reboot -- even non-critical tasks.)

Once all of the critical subsystems have at least a primary node running, the non-critical and backup subsystems will start to load. When there is at least a primary node for all defined subsystems, System Monitor declares PICS to be running.