Posts

Prompt

Yes, actually I used this approach to avoid blocking the Kafka polling thread for a long time. Earlier, I was using CallerRunsPolicy. When all 16 threads were busy and the queue of 50 was full, the Kafka listener thread could start processing the message itself. The problem was that if the SFMC call took a long time due to retries, it could block the Kafka polling thread and cause consumer rebalancing. So I changed it to a bounded waiting approach. Now, when the queue is full, it waits for up to 10 seconds for an available slot. If a slot becomes available, the task is accepted for processing. If the queue is still full after 10 seconds, it throws a RejectedExecutionException instead of processing the SFMC call on the polling thread. For failed message processing, we log the error and do not explicitly acknowledge the failed record. But this 10-second waiting is not a two-attempt retry policy. I will verify the existing Kafka retry and error-handler behavior to confirm what happens to ...

Kafka Batching

Image
1. हिंदी में समझिए — Kafka Streams migration में अलग से batching क्यों implement करनी पड़ी? भाई, आपका reason technically सही है। लेकिन manager को explain करते समय एक important distinction रखना है: आपने batching कोई नई business functionality जोड़ने के लिए implement नहीं की थी। आपने existing @KafkaListener का batch-processing behavior Kafka Streams migration के बाद भी बनाए रखने के लिए implement की थी। पहले @KafkaListener में क्या हो रहा था? Spring Kafka में batching का support पहले से available था। हमारे existing implementation में: Kafka consumer एक poll में multiple records fetch कर सकता था। Spring Kafka का batch listener उन records को List<ConsumerRecord<String, String>> के रूप में method को देता था। max.poll.records और batch listener configuration से यह behavior control होता था। हमारा existing Data Service consumer पूरी list को एक साथ process करता था। इसलिए हमें manually records collect करने की जरूरत नहीं थी। Kafka Streams में migrate करने के बाद क्या बदला? Kafka Stre...

My Challanging Work

Image
  Challenging work i have done in my previous project ✅ 1️⃣ One-minute interview explanation (you can speak this) “We were reading millions of records from one system and inserting them into another database. Initially, we faced performance issues and OutOfMemoryError because the entire data was loaded into memory. To solve this, I implemented batch processing with multithreading. I processed records in chunks of 1000, performed transformation, inserted them into the database, cleared memory, and then processed the next batch. This made the system memory-efficient and scalable public void processLargeData(List<String> records) {         for (int i = 0; i < records.size(); i += BATCH_SIZE) {             int end = Math.min(i + BATCH_SIZE, records.size());             List<String> batch = records.subList(i, end);...

JunitTest

import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; import org.mockito.Mockito; import org.springframework.batch.core.Step; import org.springframework.batch.core.job.builder.StepBuilder; import org.springframework.batch.core.repository.JobRepository; import org.springframework.batch.core.step.builder.StepBuilderHelper; import org.springframework.batch.repeat.RepeatStatus; import org.springframework.transaction.PlatformTransactionManager; import static org.junit.jupiter.api.Assertions.*; import static org.mockito.Mockito.*; public class MyJobConfigurationTest {     private MyJobConfiguration myJobConfiguration;     private FileBackUpTasklet fileBackUpTasklet;     private JobCompletionListener jobCompletionListener;     private JobRepository jobRepository;     private PlatformTransactionManager transactionManager;     @BeforeEach     public void setUp() {         fileBackUpTas...

Duplicate Implementation

How to Resolve Git Author Identify Unknown and please tell me who you are

Image
  git config user.name " irfankhansoft" git config user.email "irfankhansoftwareengineer@gmail.com" git config --list Create Repository inside your Github and copy that url Now add the remote repo using this url git remote add origin https://github.com/irfankhansoft/git-localconfig-repo git push origin master while push it might ask to authorised by browser just login with your credential to login

Thread Termination and Demon Thread

Image
 Thread Termination: Why : Because Thread are Consuming Resources : Like Memory, Kernel Recourses, cpu Cycle and cache memory When: 1. if thread finished its work, so we need to clean up the thread's resources 2. if thread misbehaving  3. By default application will not stope the thread as long as at least one thread is still running  Therad.intrrupt(): 1 if the thread is executing a method that throws an intrrupedException 2. if the thread's code is handling the interrupt signal explicitly  3.  If any one of the thread is in sleeping or waiting state (i.e. sleep() or wait() is invoked), calling the Thread.interrupt() method on the thread, breaks out the sleeping or waiting state throwing InterruptedException. If the thread is not in the sleeping or waiting state, calling the interrupt() method performs normal behavior and doesn't interrupt the thread but sets the interrupt flag to true   Daemon Thread: Daemon threads are thread that do not prevent the...