Kafka with KafkaJS

Setup

npm install kafkajs

Producer

const kafka = new Kafka({ clientId: 'app', brokers: ['localhost:9092'] });
const producer = kafka.producer();
await producer.connect();
await producer.send({ topic: 'events', messages: [{ key: 'user-1', value: JSON.stringify(event) }] });

Consumer

const consumer = kafka.consumer({ groupId: 'my-service' });
await consumer.subscribe({ topic: 'events' });
await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
        const event = JSON.parse(message.value!.toString());
        await processEvent(event);
    },
});

Admin

const admin = kafka.admin();
await admin.createTopics({ topics: [{ topic: 'events', numPartitions: 3 }] });

Key Patterns

• Message keys for partition ordering
• Consumer groups for parallel processing
• Idempotent consumers (handle duplicates)
• Transactional producer for exactly-once