ENT-5666 Extract shutdown events when retrying a flow (#6684)

* ENT-5666 Extract shutdown events when retrying a flow

When a flow is retrying, only a select set of events are transferred
over to the new fiber. Shutdown events were not included in this set.

This meant that if a flow retries when an `Event.SoftShutdown` is in its
queue, it will never process it. This causes the node to hang, as the
node awaits `liveFibers`, which never reaches 0 because the shutdown
event is never processed.

To resolve this, `Event.SoftShutdown` is added to the set of events to
extract and reschedule.

* ENT-5666 Don't schedule extra shutdown event

When a flow is stopped by an `Event.SoftShutdown` it will eventually
reschedule another shutdown even when it reaches `SMM.removeFlow`. It
won't actually be processed because the flow returns an abort
continuation. But, it does look odd.

Therefore, it now does nothing instead since that is what it was
implicitly doing.
This commit is contained in:
Dan Newton 2020-09-04 15:44:14 +01:00 committed by GitHub
parent e562c5828b
commit 64dc94bd18
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

View File

@ -415,10 +415,10 @@ internal class SingleThreadedStateMachineManager(
if (flow != null) {
decrementLiveFibers()
totalFinishedFlows.inc()
return when (removalReason) {
when (removalReason) {
is FlowRemovalReason.OrderlyFinish -> removeFlowOrderly(flow, removalReason, lastState)
is FlowRemovalReason.ErrorFinish -> removeFlowError(flow, removalReason, lastState)
FlowRemovalReason.SoftShutdown -> flow.fiber.scheduleEvent(Event.SoftShutdown)
FlowRemovalReason.SoftShutdown -> { /* No further tidy up is required */ }
}
} else {
logger.warn("Flow $flowId re-finished")
@ -601,7 +601,9 @@ internal class SingleThreadedStateMachineManager(
val events = mutableListOf<Event>()
do {
val event = oldEventQueue.tryReceive()
if (event is Event.Pause || event is Event.GeneratedByExternalEvent) events.add(event)
if (event is Event.Pause || event is Event.SoftShutdown || event is Event.GeneratedByExternalEvent) {
events.add(event)
}
} while (event != null)
// Only redeliver events if they were not persisted to the database