Hotfix/restart build

Created by: TerrenceMcGuinness-NOAA

Description

Improved CI robustness for reverting back to CI-Ready from any given state See a full review of the latest updated PR user's messaging from the CI Scripts (here)

New Features:

• Improved scancel routine (refactored into bash "subroutine")

• Improved messaging (see below) when ever a user changes state

• Any and all previous build scripts and running experiments are killed as a result of reset to Ready

  Resolves #2060 Resolves #2067

• Bug fix (Resetting CI tests had lingering builds simultaneously running)

• New feature (Robust Resetting with messages)

• Maintenance (code refactor - use of BASH subroutines in main drivers)

Example of output when user updates state and request a restart:

CI Update on Hera at 11/28/23 09:33:11 PM
=================================================
PR:201 Reset to Hera-Ready by user and is now restarting CI tests
Driver PID: 302744 on hfe12 is no longer running this test
Driver PID: has restarted as 163669 on hfe12
No current experiments to cancel in PR: 201 on Hera

Change characteristics

• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? NO

How has this been tested?

Reset at all steps in dev Cron CI space with appropriate kills taking place on Hera and Orion

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes generate no new warnings
• New and existing tests pass with my changes
• I have made corresponding changes to the documentation if necessary