Skip to content
Slingshot

Overview

Human-owned documentation. This page is the package-level contract for Kafka transport behavior, operator expectations, and production guidance.

Purpose

Section titled “Purpose”

@lastshotlabs/slingshot-kafka provides two Kafka integration surfaces for Slingshot:

  • createKafkaAdapter(...) for using Kafka as the internal event bus
  • createKafkaConnectors(...) for bridging between the internal event bus and external Kafka topics

This package owns Kafka transport concerns only. It does not provision brokers, topics outside its auto-create paths, ACLs, or cluster infrastructure.

Supported Connection Modes

Section titled “Supported Connection Modes”
  • SASL mechanisms:
    • plain
    • scram-sha-256
    • scram-sha-512
  • TLS modes:
    • ssl: true for platform trust store / ambient CA trust
    • ssl: { ca } for explicit broker trust
    • ssl: { ca, cert, key } for mTLS / client-certificate auth

These modes are live-verified in Docker against Redpanda listeners, including negative paths for:

  • bad SASL credentials
  • wrong CA bundle
  • mTLS listener access without a client certificate

Package Boundaries

Section titled “Package Boundaries”
  • Kafka serialization, topic naming, durable-consumer wiring, retries, and DLQ handoff belong here.
  • Event schema validation contracts come from @lastshotlabs/slingshot-core.
  • App config assembly, secrets resolution, and framework startup belong in @lastshotlabs/slingshot.
  • Broker lifecycle, ACL policy, topic retention, partition count strategy, and cluster sizing belong to the broker provider or platform team.

Production Guidance

Section titled “Production Guidance”

For transactional outbox delivery, the framework calls the adapter’s awaited publishEnvelope() boundary and marks a row delivered only after Kafka acknowledges the produce request. Retries reuse the original envelope and event ID. Configure transactional inbox consumers with stable names because Kafka redelivery and the outbox accept/finalize crash window can produce duplicates.

  • Prefer TLS for every non-local environment.
  • Treat ssl.rejectUnauthorized: false as local-development-only. The runtime warns because it disables broker certificate verification.
  • Prefer provisioned topics over autoCreateTopics / autoCreateTopic in production.
  • If you must auto-create topics, do not leave replication factor at 1 outside disposable environments. The runtime warns on that path.
  • If using SASL, pair it with TLS unless the broker is on a fully trusted isolated network segment and that exception is deliberate.
  • For mTLS, provide PEM strings through ssl objects on createKafkaAdapter / createKafkaConnectors. The built-in secret bundle only supports the coarse KAFKA_SSL=true switch, not PEM material injection.

Config And Secrets Behavior

Section titled “Config And Secrets Behavior”
  • Pass eventBus: createKafkaAdapter({ ... }) directly in app.config.ts to use the Kafka adapter as the framework’s event bus.
  • Pass kafkaConnectors: createKafkaConnectors({ ... }) to mount inbound/outbound topic bridges.
  • Secret-driven TLS bootstrap currently supports KAFKA_SSL=true.
  • Advanced TLS and mTLS require explicit ssl config:
    • ssl.ca
    • ssl.cert
    • ssl.key
    • ssl.rejectUnauthorized

Broker / Provider Requirements

Section titled “Broker / Provider Requirements”

The broker or managed Kafka provider must supply:

  • reachable broker endpoints
  • the auth mode the client is configured for
  • the correct CA chain when TLS is enabled
  • a client certificate policy, if mTLS is required
  • ACLs for the topics, groups, and create/read/write operations the app needs

This package assumes the provider owns:

  • retention and compaction policy
  • replication and ISR policy
  • cluster quotas and throughput limits
  • topic pre-provisioning, unless you intentionally enable auto-create
  • certificate issuance and rotation outside local test fixtures

Operational Runbook

Section titled “Operational Runbook”

Startup checklist

Section titled “Startup checklist”
  • Confirm brokers resolve from the running process.
  • Confirm the configured SASL mode matches the broker listener.
  • Confirm the CA bundle is the one that signed the broker certificate.
  • For mTLS, confirm the client cert and key are both present and signed by a CA the broker trusts.
  • Confirm ACLs allow topic metadata, produce, consume, and group membership for the configured topics/groups.

Failure triage

Section titled “Failure triage”
  • SASL authentication failed
    • Check username, password, and mechanism.
    • Confirm the broker listener actually enables that SASL mechanism.
  • certificate / TLS verification failures
    • Check ssl.ca, certificate chain, hostname/SAN, and whether rejectUnauthorized is being forced.
  • auth succeeds but produce/consume fails
    • Check topic and group ACLs.
    • Check whether the broker allows topic auto-creation for that principal.
  • durable consumer never reaches connected state
    • Check group ACLs, topic existence, listener reachability, and partition assignment.
  • repeated buffering / drain failures
    • Check broker availability, ACLs, and whether the producer can reconnect after transient disconnects.

Observability hooks

Section titled “Observability hooks”
  • createKafkaAdapter(...).health() exposes producer/admin/consumer connectivity and pending buffer size.
  • createKafkaConnectors(...).health() exposes inbound/outbound runtime state and pending buffer size.
  • createKafkaConnectors(...).hooks can be used to wire package-level metrics or structured logs for inbound success/error, outbound success/error, suppression, and DLQ writes.

Gotchas

Section titled “Gotchas”
  • SASL authentication and authorization are different. A principal can authenticate successfully and still fail topic or group operations without ACLs.
  • ssl: true depends on the process trust store. For private CAs, prefer ssl: { ca }.
  • The framework secret bundle does not currently ingest PEM blobs for Kafka TLS. Use explicit ssl objects on createKafkaAdapter / createKafkaConnectors for advanced TLS and mTLS.
  • This workspace applies a temporary local KafkaJS patch during install to avoid TimeoutNegativeWarning under Bun. Remove that patch when upstream KafkaJS ships the fix we are pinned waiting for.

Key Files

Section titled “Key Files”
  • packages/slingshot-kafka/src/kafkaAdapter.ts
  • packages/slingshot-kafka/src/kafkaConnectors.ts
  • packages/slingshot-kafka/tests/integration/
  • tests/docker/kafka-tls.test.ts