Each service module added via the add service module GitHub workflow is created in its own directory a.k.a. module, with its descriptor created in the services module.

The workflow creates a bare Kafka Streams based microservice that utilises the Creek Kafka service and test extensions to simplify development and testing.

Each service module has the following structure:

Entry Point

The entry point to the service is defined in the ServiceMain class. The entry point initialised Creek, builds a Kafka Streams topology and executes it.

   public static void main(final String... args) {
       final CreekContext ctx =
               CreekServices.builder(new ExampleServiceDescriptor())
                       .with(
                               KafkaStreamsExtensionOptions.builder()
                                       // Customize the Kafka streams extension here...
                                       .build())
                       .build();
       final KafkaStreamsExtension ext = ctx.extension(KafkaStreamsExtension.class);
       final Topology topology = new TopologyBuilder(ext).build();
       ext.execute(topology);
   }

ProTip: The application will terminate if the Kafka Streams application fails.

Kafka Streams topology

Topology Builder

The Kafka Streams topology is built by the TopologyBuilder class. A bare implementation is provided.

See the Basic Kafka Streams Tutorial to learn how to create a simple topology, and the Kafka Streams documentation for more info.

ProTip: Break more complex topologies up into sub topology builders, utilising the Name instance to reduce the change of node-name clashes. See Name Javadoc to learn more.

Testing the topology

The Service’s Kafka Streams topology can be unit tested via the TopologyBuilderTest class. A bare implementation is provided.

The test class includes an existing test called shouldNotChangeTheTopologyUnintentionally. The intent of this test is to detect any unintentional changes to the Kafka Streams topology.

Warning: There are certain categories of topology changes that are not backwards compatible with earlier versions of a deployed service, e.g. those that change topic names. Creek recommends always naming operators in the Kafka Streams DSL. (See the Kafka Streams docs for more information).

The test compares the topology with the last know topology and fails if they differ. If the change is intentional, then the reverse-service/src/test/resources/kafka/streams/expected_topology.txt file can be updated to reflect the latest topology.

If you find this test more of a hindrance than a help… delete it! :smile:

See the Basic Kafka Streams Tutorial to learn how test a simple topology, and the Kafka Streams test util documentation for more info.

Docker

A Dockerfile is included in the root of each service’s directory and the build.gradle.kts file defines several Docker related tasks. Combined, these controls how the Docker image for the service is built and published.

Include directory

The include directory defines the run.sh script referenced by the services Dockerfile, and other configuration, such as a log4j2.xml file to configure service logging.

These files are copied to the service’s Docker image.

Backwards compatability

Because each service is encapsulated within a Docker image, there is no backwards compatibility concerns with changing the API of classes within the module.

Care should be taken when changing Kafka Stream topologies to ensure they remain backwards compatible with any existing deployment.

Warning: There are certain categories of topology changes that are not backwards compatible with earlier versions of a deployed service, e.g. those that change topic names. Creek recommends always naming operators in the Kafka Streams DSL. (See the Kafka Streams docs for more information).

Updated: