Event Streams - Client Integration Guide
Platform
The Event Streams service in built using the Kafka distributed streaming platform with support from Confluent open-source tools. Data that is delivered via the Event Streams service is defined and serialized using Avro schema.
Terminology
Bootstrap Server Listing
| Term |
Definition |
| Broker |
A single server that stands alone, or works as part of a cluster, to host the Kafka platform.
|
Producer
|
Application that publishes a stream of records to one or more Kafka topics.
|
| Consumer |
Application that consumes a processes a stream of records from the topics to which is is subscribed.
|
| Connector |
Reusable producers and consumers that bridge the Kafka platform together with common application and/or database technologies.
|
| Topic |
A category of event stream data (e.g. PersonEvent) that serves as the hand-off between publishers and subscribers.
|
| Partition |
An ordered, immutable sequence of records that organizes the data on a given topic. A topic typically is made up of multiple partitions, and thus a consumer will need to consume from all partitions to consume the entire topic.
|
| Offset |
Immutable sequence that facilitates the ordering of data within a topic.
|
Streams Processor
|
An application that serves as both consumer and producer, to carryout business rule processing (joining, filtering, aggregating streams data) before delivering a finished product to client-facing topics.
|
Environments
AITS will maintain a production, and at least one non-prod Event Streams environment. The anticipated environment settings are displayed below, but are subject to change with the maturity of the Event Streams service, and according to shifts in testing needs.
NOTE: The Development environment is subject to increased downtime and volatility due to ongoing development activities. Event Streams clients are encouraged to utilize the Test environment for all testing, unless a specific need is identified otherwise. The Test environment is expected to be more stable, given that only those code changes that have passed unit and system testing will be promoted to the environment before reaching production.
Bootstrap Server Listing
| Environment |
Integration Database Source |
Upstream Authoritative Source(s) |
| Development |
STRMINTD
|
BANDEV
|
| Test |
STRMINTD
|
BANQA
|
| Production |
STRMINTP |
BANPROD
|
Integration Strategy
Application developers interested in
consuming events from the Event Streams service will have
the option to either build a native Kafka client, or to integrate via a
RESTful API hosted in the AITS Web Services proxy. The remainder of this guide is pertinent to the building of a native Kafka client. For more information about the RESTful API integration option, please review the Streams API service definition.
Configuration
Security
The Event Streams service implements the SASL/PLAIN authentication mechanism. All interactions with the Event Streams platform are protected with TLS.
# Sample Security Properties
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
username="alice" \
password="alice-secret";
Bootstrap Servers
Every Event Streams client configuration must include a setting for bootstrap servers. These are the nodes that provide initial connectivity with the platform. Actual bootstrap server details will be provided during new client on-boarding. Please contact the AITS Integration Competency Center at anytime to begin a conversation.
# Sample Bootstrap Properties
bootstrap.servers=https://<server>:<node>,https://<server>:<node>
Schema Registry Servers
Event Stream data is defined and serialized using Avro schema, with support from the Confluent Schema Registry. The properties displayed below will need to accompany all native client configurations. ICC will provide server details during initial client on-boarding.
# Sample Schema Registry Properties
schema.registry.url=https://<server>:<node>/<resource>
key.deserializer=io.confluent.kafka.serializers.KafkaAvroDeserializer
value.deserializer=io.confluent.kafka.serializers.KafkaAvroDeserializer