Need better way to distinguish intended host state from current state

[rmustacc]

I was debugging some cases earlier today where some sleds appear to have been rebooting. I appear to have caught a case where the sled was briefly in A2. I got this via pilot sp status, which is using faux-mgs to understand the current power state. However, the thing that I didn't know was that this was probably in the process of rebooting due to a host boot / panic loop. We need some way to better distinguish what is the current power state from what is happening or transient. In general when I see us in A2, I assume we have come to rest in A2 not that we're in the process of making a transition. Perhaps there's some additional set of APIs or way we can expose this. I'm not sure what makes the most sense or how we want to denote intent versus reality.

[hawkw]

Last year, I merged #1953, which added a "reason" enum to the Sequncer.set_power_state IPCs that records why the power state changed. However, right now this only makes it into the sequencer's ringbuf and isn't exposed anywhere that's accessible without Humility. We could wire that value through into the MGS status messages, perhaps?

[hawkw]

This would probably look like either the sequencer tasks hanging onto the last state change reason and exposing an IPC to read it that MGS would call, or eagerly posting it to MGS when the state changes. Without actually having thought too hard about it, the first seems nicer.

I'd want to put the state change reason in the same MGS message as the power state itself, rather than in a separate request to read the state change reason. Otherwise, there could be a race where MGS sees a state, and the power state changes again before MGS asks why we ended up in that state. At present, though, this would be a compute-sled-only field, so it wouldn't be included by sidecars/PSC. We might consider adding a similar thing to Sidecar's sequencer, but some of the state change reasons wouldn't apply there (there's no "host OS" to panic...).

[rmustacc]

We might consider adding a similar thing to Sidecar's sequencer, but some of the state change reasons wouldn't apply there (there's no "host OS" to panic...).

A big challenge I've had is where we apply things to only of many things that something applies to, e.g. just gimlet when also cosmo, compute sleds, when no it also applies to switches, etc.. I would really want to make sure that we do have this for all the sequencers and state. Even the PSC where there is no transition going on.