Interface StandbyUpdateListener


@Public public interface StandbyUpdateListener
Class for listening to the progress of the standby and warm-up tasks of a Kafka Streams client.

When calling KafkaStreams.setStandbyUpdateListener(StandbyUpdateListener) the passed instance is expected to be stateless since the StandbyUpdateListener is shared across all state updater threads, of which there is one per stream thread (num.stream.threads).

Users desiring stateful operations will need to provide synchronization internally in the StandbyUpdateListener implementation.

Note that this listener is only registered at the per-client level and users can use the storeName parameter to define specific monitoring for different StateStores.

Warm-up tasks are reported through this interface as well as standby tasks. A warm-up task is one that the group hands out so that this client can build a task's state up before taking the task over, and the runtime runs it as a standby task, so the two are indistinguishable to this listener. Relabelling a task between the two roles does not restart the task, and does not trigger a callback; a task might be converted from a warm-up task to a standby task between two calls of onBatchLoaded(TopicPartition, String, TaskId, long, long, long) transparently.

Also note that, unlike the restoration of an active task, the update of a standby task never finishes, since it keeps applying the changelog records written by the active task. There is therefore no callback corresponding to StateRestoreListener.onRestoreEnd(TopicPartition, String, long); a standby task is reported only once it stops being updated, through onUpdateSuspended(TopicPartition, String, long, long, SuspendReason).

Even if warm-up tasks are expected to "complete" restoration, the same pattern as for standby tasks applies. Once a warm-up task is considered hot (based on acceptable.recovery.lag), its restoration will be suspended when it is promoted to an active task, and the active task will complete the restoration (if registered, the corresponding StateRestoreListener callback is invoked on the active task for this case).

Incremental updates are exposed so users can estimate how much progress has been made.

See Also:
  • Method Details

    • onUpdateStart

      void onUpdateStart(TopicPartition topicPartition, String storeName, long startingOffset)
      Method called at the very beginning of StateStore updating.
      Parameters:
      topicPartition - the changelog TopicPartition for this task
      storeName - the name of the store being loaded
      startingOffset - the offset from which the task begins consuming from the changelog
    • onBatchLoaded

      void onBatchLoaded(TopicPartition topicPartition, String storeName, TaskId taskId, long batchEndOffset, long batchSize, long currentEndOffset)
      Method called after loading a batch of records. In this case the maximum size of the batch is whatever the value of max.poll.records is set to.

      This method is called after loading each batch and it is advised to keep processing to a minimum. Any heavy processing will block the state updater thread and slow down the rate of standby task loading. Therefore, if you need to do any extended processing or connect to an external service, consider doing so asynchronously.

      Parameters:
      topicPartition - the changelog TopicPartition for this task
      storeName - the name of the store being loaded
      taskId - the TaskId of the task that owns the store being loaded
      batchEndOffset - the changelog end offset (inclusive) of the batch that was just loaded
      batchSize - the total number of records in the batch that was just loaded
      currentEndOffset - the current end offset of the changelog topic partition
    • onUpdateSuspended

      void onUpdateSuspended(TopicPartition topicPartition, String storeName, long storeOffset, long currentEndOffset, StandbyUpdateListener.SuspendReason reason)
      Method called when the corresponding standby or warm-up task stops being updated, for the provided reason.

      If the reason is StandbyUpdateListener.SuspendReason.MIGRATED, this callback will be invoked after this state store (and the task itself) are closed, in which case the data will be cleaned up after state.cleanup.delay.ms. If the task was StandbyUpdateListener.SuspendReason.PROMOTED to an active task, the state store will not be closed, and the callback will be invoked after unregistering it as a standby task but before re-registering it as an active task and beginning restoration. In other words, this will always be called before the corresponding StateRestoreListener.onRestoreStart(TopicPartition, String, long, long) call is made.

      Parameters:
      topicPartition - the changelog TopicPartition for this task
      storeName - the name of the store being loaded
      storeOffset - the offset of the last changelog record that was read and put into the store at the time of suspension
      currentEndOffset - the current end offset of the changelog topic partition
      reason - the reason why the task stopped being updated