Streams API

Service Information:

Version: 1.0

Availability: Development

Release Date: (planned) August 2018

Framework: AITS RESTful Data API

Description: Provides a RESTful interface to the Event Streams service. For additional information about the Event Stream service, please see the Event Streams service guide.

Source System: Various

Data Format: JSON

Operations:

poll

GET /{resource}/stream/poll/{senderAppId}/{topic}

Retrieves all events that occurred since the last committed offset (last call to the service) for a given senderAppId and topic. This service transparently manages offsets for, and consumes from all partitions that exist for the topic being polled.

NOTE: The Kafka platform will persist a consumer's offset values for up to seven days of inactivity since the last "poll" execution. After that time, the consumer will automatically seek to the end of the topic with the next "poll" execution. The Kafka platform also has configured a max.poll.records (max number of records retrieved with each "poll" execution) of 500. This can be exceeded occasionally as the platform attempts to budget the max over topics having numerous partitions.

Sample Request

GET /personWS/stream/poll/mySenderAppId/TestEvent HTTP/1.1
Host: webservices-dev.admin.uillinois.edu                         

Sample Response

[
    {
        "object": "TestEvent",
        "topic": "avro-testEvent",
        "partition": 1,
        "offset": 0,
        "record": {
            ...
        }
    },
    {
        "object": "TestEvent",
        "topic": "avro-testEvent",
        "partition": 1,
        "offset": 1,
        "record": {
            ...
        }
    }

Object Detail

read

GET /{resource}/stream/read/{senderAppId}/{topic}/{partition}/{offset}

Provides an array of events as are avaiable for a specified topic and partition, starting from the specified offset.

Sample Request

GET /personWS/stream/read/mySenderAppId/TestEvent/1/20 HTTP/1.1
Host: webservices-dev.admin.uillinois.edu                         

Sample Response

[
    {
        "object": "TestEvent",
        "topic": "avro-testEvent",
        "partition": 1,
        "offset": 0,
        "record": {
            ...
        }
    },
    {
        "object": "TestEvent",
        "topic": "avro-testEvent",
        "partition": 1,
        "offset": 1,
        "record": {
            ...
        }
    }

Object Detail

stats

GET /{resource}/stream/stats/{senderAppId}/{topic}

Provides a count of messages currently on a topic, along with a listing of partitions and the current start/end offset for each.

GET /personWS/stream/stats/mySenderAppId/testEvent HTTP/1.1
Host: webservices-dev.admin.uillinois.edu                         

{
    "object": "testEvent",
    "topic": "testTopic",
    "total": 362,
    "countData": [
        {
            "partition": 0,
            "beginOffset": 0,
            "endOffset": 112,
            "total": 112
        },
        {
            "partition": 1,
            "beginOffset": 0,
            "endOffset": 134,
            "total": 134
        },
        {
            "partition": 2,
            "beginOffset": 0,
            "endOffset": 116,
            "total": 116
        }
    ]
}
                            

lag

GET /{resource}/stream/lag/{senderAppId}/{topic}

Provides "stats" data along with lag data detailing a consumer group's place in a topic.

GET /personWS/stream/lag/mySenderAppId/testEvent HTTP/1.1
Host: webservices-dev.admin.uillinois.edu                         

{
    "object": "Object",
    "topic": "Topic",
    "groupName": "ConsumerGroupName",
    "total": 671,
    "lag": 0,
    "lagPercent": 0.0,
    "countData": [
        {
            "partition": 0,
            "beginOffset": 0,
            "endOffset": 640,
            "total": 640
        },
        {
            "partition": 1,
            "beginOffset": 0,
            "endOffset": 6,
            "total": 6
        },
        {
            "partition": 2,
            "beginOffset": 0,
            "endOffset": 25,
            "total": 25
        }
    ],
    "lagData": [
        {
            "partition": 0,
            "currentOffset": 640,
            "endOffset": 640,
            "lag": 0
        },
        {
            "partition": 1,
            "currentOffset": 6,
            "endOffset": 6,
            "lag": 0
        },
        {
            "partition": 2,
            "currentOffset": 25,
            "endOffset": 25,
            "lag": 0
        }
    ]
}