Skip to content

Conversation

@elliVM
Copy link
Contributor

@elliVM elliVM commented Apr 2, 2025

Changes

Goals is to replace poll(timeout) method completely since it was causing stalls when topic had no records or was unauthorized. Old way subscribed using a pattern and then used poll() to get offsets from the meta data.

Now we use the listTopics() method to list all topics the user has access to and then filter them using a regex pattern from the query. After that we can use the resulting topics to get offsets for all the partitions.

Class responsibilities

  • CombinedTopicPartitions combined partitions of a list of topics
  • KafkaQueryImpl implements KafkaQuery to get beginning and end offsets of all topics partitions for a query
  • KafkaSubscriptionPatternFromQuery parses a regex pattern from incoming query XML
  • RegexMatchingKafkaTopics filters topics using a regex pattern
  • SetOnce allows for delayed setting of a immutable state, used to store a snapshot of Kafka topics fetched from server
  • TopicPartitions provides partitions of a topic
  • TopicsFromKafka lists all topics in Kafka available for user

@elliVM elliVM self-assigned this Apr 2, 2025
@elliVM elliVM linked an issue Apr 7, 2025 that may be closed by this pull request
@elliVM elliVM added the bug Something isn't working label Apr 9, 2025
@elliVM elliVM marked this pull request as ready for review April 9, 2025 11:24
@elliVM elliVM requested a review from Tiihott April 9, 2025 11:37
@elliVM elliVM requested review from eemhu and removed request for Tiihott April 11, 2025 11:01
@elliVM elliVM removed the request for review from eemhu April 28, 2025 08:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

kafka may get stuck if no results, permission denied etc

2 participants