Siddhi HTTP Transport 101
Siddhi is capable of receiving messages from an HTTP service and publish messages to the HTTP services. This functionality is powered by siddhi-io-http extension. In this article, I am going to discuss how to use HTTP transport features with Siddhi, and bit of its internals.
Before getting started, let’s have a look at how Siddhi operates.
Siddhi is capable of receiving HTTP services and process them and publishes the results in a relevant format to another HTTP service. In Siddhi, inputs are received by sources and output is published using sinks. So, for the HTTP transport also, there are sources and sinks.
There are different ways we can configure and use Siddhi HTTP IO functionalities.
NOTE:
The names of the HTTP source and sink types have been changed in the siddhi-io-http v2.1.0 and upwards to reflect their purposes properly.The names I have used in this article are for the versions below v2.1.0. Please find the changes below and use the correct name for the types of the HTTP source/sink, according to the siddhi-io-http version you are using.
HTTP-request sink -> HTTP-call sink
HTTP-response source -> HTTP-call-response source
Receiving HTTP requests…
HTTP source enables Siddhi to receive messages from HTTP clients. This enables Siddhi to act as an HTTP server so that clients can publish data to it using HTTP POST requests. Then Siddhi will transform those messages accordingly, convert them to Siddhi events, process, and act upon them.
Let’s see how we can define an HTTP source in a Siddhi app.
When running the Siddhi application with above source, it will start an HTTP server through Siddhi with the endpoint https://localhost:8280/Analytics/InputStream .
Then the clients can connect to that server and publish HTTP requests to Siddhi. Since the whole point of this HTTP source is receiving messages through HTTP to Siddhi, it only supports POST requests. If we need to use other HTTP methods, we can use HTTP-request source which will be discussed later.
In above example, we have used Siddhi JSON mapper expecting a JSON payload from the request. Once it receives such payload, Siddhi will extract its attribute values and map them to the attributes of the defined source stream(InputStream). Then the message will be converted to a Siddhi event. Afterthat, Siddhi processes that event according to the Siddhi app we have written.
Following is a sample JSON message which is compatible with above example source configuration.
{
“id”:”sample_1",
“price”:123.45,
“volume”:1000.500
}
There are multiple properties you can configure in this HTTP source. For example, ‘’worker.count” property in above sample source defines the thread pool size which consumes the messages coming to the HTTP source. You can find the full list of configuration properties of Siddhi HTTP source in its documentation. [1]
HTTP Source is only supposed to receive the messages to Siddhi. Therefore, we cannot provide custom responses through Siddhi for those requests. For such scenarios, we can use HTTP-request source and HTTP-response sink which will be discussed later in this post.
Sending HTTP requests out..
While HTTP source facilitates Siddhi to consume HTTP requests from clients, HTTP sink provides the facility to send HTTP requests to outside HTTP services through Siddhi.
Let’s see how we can configure an HTTP sink in a Siddhi app.
HTTP Sink acts as an HTTP client and publishes HTTP requests to the end point which is specified as ‘publisher.url’. We can specify which HTTP method to be used and other properties of the requests such as headers.
Let’s have a look at the properties we have used in above example.
- Type: This defines the type of the siddhi sink . Since this is supposed to send HTTP requests out, type is HTTP.
- Publisher.url: This property defines the end point to which the request should be sent.
- Method: Defines which HTTP method should be used to send the request. In this case, it has been defined as a dynamic property and its value will be populated by the value comes to OutputStream. However this can also be defined as a static value such as ‘POST’, ‘GET’ etc. In that case, the sink will support only one HTTP method type.
- Headers: HTTP headers that need to be included in the request. This can either be dynamically populated or a static value.
In above example, the XML mapper has been used so that the payload will be converted to a XML message in following format.
<events>
<event>
<id>WSO2</id>
<price>55.75</price>
<volume>100.00</volume>
</event>
</events>
You can refer to the Siddhi HTTP sink documentation[2] to learn about all the configuration properties.
However, this HTTP sink only sends the request out. It does not handle the response it’s receiving. If we want to send the request and process the response again using Siddhi, we have to use HTTP-request sink and HTTP-response source together.
Sending HTTP Requests out, receiving the responses and processing them..
In the use cases where we need to send requests and process the responses through Siddhi, we can use HTTP-request sink and HTTP-response source features. These two features are linked and working together. Therefore, if we use an HTTP-request sink, we need to define an HTTP-response source also for that.
Let’s have a look at how to configure a request sink and response sink.
In this scenario, simply the HTTP-request sink sends an HTTP request out and corresponding response source receives the response for that request. Then it converts that response to a siddhi event and passes to Siddhi for further processing.
The “sink.id” property is used to link a request sink to its corresponding response source. When we define a request sink, we have to define a response source with the same sink.id. Once a request is sent out from a request sink, the response will be caught by the response source with the same sink.id.
All the properties available in HTTP sink can be used with HTTP-request sink as well. However, there are some additional properties which are unique to the HTTP-request sink. You can find all the configuration properties in the HTTP-request sink documentation [3].
Receiving HTTP requests, generating and sending responses..
If we want to implement a complete flow of a web server using Siddhi, we can use HTTP Request source and HTTP Response sink.
As the name suggests, request source receives the HTTP requests from outside clients and feed them to Siddhi. Then Siddhi can process the request and generate the response for that request according to the Siddhi app we have written.
When a request comes to the request source, Siddhi assigns a UUID for that request and pass it through the rest of the flow of the Siddhi application.
We can generate the response for that particular request through Siddhi and send it back to the corresponding client by co-relating response and the request using above mentioned UUID. This UUID is exposed to the user as ‘messageId’.
Before going further, let’s have a look at sample Siddhi app.
All the configuration properties of HTTP source are supported by HTTP-Request source also [4]. However, there is an additional property called ‘source.id’ which is used to bind a request source with a response source.
Once a request is received by the source, it will be assigned a UUID which is named as ‘messageId’ and queued. Then Siddhi will process the payload and generate a response for that particular request. After generating the response, Siddhi seeks for the original request in the relevant source using source.id and the messageId and then sends the response out.
MessageID is used to keep track of multiple requests coming to a single source while source.id is used to link request sources and response sinks since there can be multiple sources and sinks in the same Siddhi app.
However, this messageId is not available as an attribute of the payload. It is passed down from the Siddhi source level to Siddhi mapper level as a transport property. I will be discussing about transport properties at the end of this article. Until then, just think it as a way to pass down the meta data of the request/message we receive, such as HTTP headers, from Siddhi sources to Siddhi mappers.
At the source configuration, we can extract this messageId from transport properties and we have to pass it along the way down to the response sink in order to send the response back to the client.
Let’s quickly have a look at what are the transport properties and how to use them.
Siddhi transport properties
As mentioned above, usually Siddhi mappers are only capable of extract values from the payload messages received by Siddhi sources. See below example.
As per the above source configuration, the expected payload would be a JSON message in following format.
{
“username”: “thanos”,
“loginIp”: “123.456.78.901”,
“loginTime”: “1565364934”
}
Siddhi JSON mapper (siddhi-map-json) converts the above JSON message to a Siddhi event with the attributes username, loginIp and loginTime.
@map(type=’json’, @attributes(messageId=’trp:messageId’, username=’$.username’, loginTime=’$.loginTime’, loginIp = ‘$.loginIp’)))
However, as we can see, this payload message does not have an attribute called messageId. But we need that attribute in order to create the Siddhi event.(messageId=’trp:messageId’).
Siddhi is able to pass some other properties/meta data which are related to the input message such as HTTP headers to the mapper level as transport properties. Although these properties are not available in the payload, we can access them at the mapper level by calling trp:<property name>.
In this case, messageId is also passed as a transport property. In addition to that, if we need to get HTTP headers of the original request, we can extract them also from transport properties.
I hope this article gave you some insight on Siddhi HTTP transport capabilities and the ways we can use them. Please let me know if there’s anything to be improved or added.
References:
[1] https://siddhi-io.github.io/siddhi-io-http/api/latest/#http-source
[2] https://siddhi-io.github.io/siddhi-io-http/api/latest/#http-sink
[3] https://siddhi-io.github.io/siddhi-io-http/api/latest/#http-request-sink
[4] https://siddhi-io.github.io/siddhi-io-http/api/latest/#http-response-source
