in Education by
I am using the Kafka Streams Processor API to construct a Kafka Streams application to retrieve messages from a Kafka topic. I have two consumer applications with the same Kafka Streams configuration. The difference is only in the message size. The 1st one has messages with 2000 characters (3KB) while 2nd one has messages with 34000 characters (60KB). Now in my second consumer application I am getting too much lag which increases gradually with the traffic while my first application is able to process the messages at the same time without any lag. My Stream configuration parameters are as below, application.id=Application1 default.key.serde=org.apache.kafka.common.serialization.Serdes$StringSerde default.value.serde=org.apache.kafka.common.serialization.Serdes$StringSerde num.stream.threads=1 commit.interval.ms=10 topology.optimization=all Thanks JavaScript questions and answers, JavaScript questions pdf, JavaScript question bank, JavaScript questions and answers pdf, mcq on JavaScript pdf, JavaScript questions and solutions, JavaScript mcq Test , Interview JavaScript questions, JavaScript Questions for Interview, JavaScript MCQ (Multiple Choice Questions)

1 Answer

0 votes
by
In order to consume messages faster, you need to increase the number of partitions (if it's not yet done, depending on the current value), and do one of the following two options: 1) increase the value for the config num.stream.threads within your application or 2) start several applications with the same consumer group (the same application.id). as for me, increasing num.stream.threads is preferable (until you reach the number of CPUs of the machine your app runs on). Try gradually increasing this value, e.g go from 4 over 6 to 8, and monitor the consumer lag of your application. By increasing num.stream.threads your app will be able to consume messages in parallel, assuming you have enough partitions.

Related questions

0 votes
    I am using the Kafka Streams Processor API to construct a Kafka Streams application to retrieve messages from ... for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked May 7, 2022 in Education by JackTerrance
0 votes
    I have an elasticsearch kafka-connect connector consuming some topics. With the following configuration: { connection ... for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Apr 23, 2022 in Education by JackTerrance
0 votes
    I have an elasticsearch kafka-connect connector consuming some topics. With the following configuration: { connection ... for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Apr 21, 2022 in Education by JackTerrance
0 votes
    Explain the maximum size of a message that can be received by the Kafka?...
asked Nov 1, 2020 in Technology by JackTerrance
0 votes
    Define consumer lag and ways to monitor it in Kafka...
asked Nov 5, 2020 in Technology by JackTerrance
0 votes
    Aryan has been given an assignment to write three paragraphs on the topic keep you city clean help him to format ... to fromat these Select the correct answer from above options...
asked Dec 25, 2021 in Education by JackTerrance
0 votes
    It means all points of the message should connect to each other and should be relevant to the main topic. Select the correct answer from above options...
asked Dec 31, 2021 in Education by JackTerrance
0 votes
    It means all points of the message should connect to each other and should be relevant to the main topic Select the correct answer from above options...
asked Dec 31, 2021 in Education by JackTerrance
0 votes
    Each message published to a topic is delivered to _______ within each subscribing consumer group. (1)all consumer ... (3)zero consumer instances (4)multiple consumer instances...
asked Jun 17, 2021 in Technology by JackTerrance
0 votes
    Which processor sends any received records from its upstream processors to a specified Kafka topic ? (1)Sink (2)Intermediate (3)All the options (4)Source...
asked Jun 17, 2021 in Technology by JackTerrance
0 votes
    What is the default retention period for a Kafka topic ? (1)1 day (2)7 days (3)7 hours (4)1 hour...
asked Jun 16, 2021 in Technology by JackTerrance
0 votes
    ____________ function can be used to select the random sample of size n' from a huge dataset. (a) ... Debugging of R Programming Select the correct answer from above options...
asked Feb 15, 2022 in Education by JackTerrance
0 votes
    What are the types of traditional method of message transfer in Apache-Kafka?...
asked Nov 1, 2020 in Technology by JackTerrance
0 votes
    The strategy used to ensure that the message is a multiple of the cipher block size without compromising the size ... 1)Ciphertext stealing (2)Padding (3)Refactoring (4)Diffusion...
asked Mar 19, 2021 in Technology by JackTerrance
0 votes
    There is a practice in coding to keep some code blocks in comment symbols than delete it when debugging. How this affects when debugging?...
asked Nov 8, 2020 in Technology by JackTerrance
...