Inform Streaming API
The Validic Streaming API gives developers access to all of their data through a continuous stream of data in near-realtime over an HTTP(S) connection using SSE (Server Sent Events). A streaming client will be pushed data, without any of the overhead associated with polling a REST endpoint.
Warning
This guide is intended as an overview and quick start guide and does not include all documentation regarding Streams. Please review the Streaming API Documentation for more information.
Stream
The Stream is intended to be used by customers to consume and process all of their user's data. A customer will create a Stream resource and then connect one or more clients to the created resource's /connect endpoint. A stream resource will load balance the event stream across the connected clients, and the event stream will be partitioned across the connected clients based upon user ID. The event stream will be rebalanced as clients connect or disconnect from the stream resource. A customer horizontally scales their ability to process the stream by connecting additional clients.
The stream resource will internally keep track of which events were streamed to a client. If all of a customer's clients are disconnected, upon reconnecting to the stream resource, the re-connected clients will start receiving events since the last connected client's disconnect.
Step 1. Create a Stream
Before connecting to the Stream endpoint, a customer will need to create a stream. You will create the stream by submitting a name and start date. You may also optionally limit the Stream to specific resources. For instance, if you're only interested in Sleep data, you can limit this Stream to the "Sleep" resource.
To do this, you'll place a POST request
To create a stream, you'll need to POST the following request: https://streams.v2.validic.com/streams?token=:yourAPIKey
POST https://streams.v2.validic.com/streams?token=6f46db3SAMPLEb82d91698fcd0896
{
"name": "customer_meaningful_name",
"start_date": "2017-01-01",
"resource_filter": ["summary", "measurement"]
}
In response, Validic will respond with an "id" which will be used to open a stream. Here is the sample response:
{
"id": "59778dSAMPLE70001565a2d",
"name": "customer_meaningful_name",
"resource_filter": [
"summary",
"measurement"
],
"start_date": "2017-07-25",
"group": "stream_59778d430b11e70001565a2d",
"members": 0,
"created_at": "2017-07-25T18:26:11Z",
"updated_at": "2017-07-25T18:26:11Z"
}
Validic allows a customer to create up to 5 streams (note that this number is the number of streams, not the number of client connections to the stream) and load balance the responses across those additional connections. Attempting to create more than the allowed number will result in an error.
Tip
Optional filters need to be assigned when creating a stream. This allows Validic to apply the filters for each client that connects to the created stream’s connect endpoint, ensuring all clients receive a consistent view of the stream.
Step 2. Opening the Stream
Once you've created the Stream the next step is to open the stream. This is done by placing a GET request to
https://streams.v2.validic.com/streams/:ID/connect?token=:yourAPIKey
For example, to open the Stream that was created above, you would make this call:
https://streams.v2.validic.com/streams/59778dSAMPLE70001565a2d/connect?token=6f46db36dSAMPLEb82d91698fcd0896
Once this stream is open, you'll begin to receive data from all users within your organization. A Poke Event will indicate the Stream is open and ready to send data. Think of it as a heartbeat.
event: poke
data: {"ts":"2017-09-26T16:53:21.687Z","stream":"59c51e117f3b920001aa2674","members":{"count":2,"max":20}}
event: poke
data: {"ts":"2017-09-26T16:53:26.692Z","stream":"59c51e117f3b920001aa2674","members":{"count":1,"max":20}}
event: poke
data: {"ts":"2017-09-26T16:53:31.697Z","stream":"59c51e117f3b920001aa2674","members":{"count":1,"max":20}}
event: poke
data: {"ts":"2017-09-26T16:53:36.703Z","stream":"59c51e117f3b920001aa2674","members":{"count":1,"max":20}}
event: data
data: {"category":"daily","checksum":"32b58b21c9f5ca4675146fe9e731ca23","created_at":"2017-09-22T11:03:21.850Z","deleted_at":null,"end_time":"2017-09-23T03:59:59Z","id":"3f21635a99b9946abb857f85920a980d","log_id":"2017-09-22","metrics":[{"type":"active_energy_burned","origin":"device","unit":"kcal","value":17.0},{"type":"basal_energy_burned","origin":"device","unit":"kcal","value":573.0},{"type":"distance","origin":"device","unit":"m","value":40.0},{"type":"energy_burned","origin":"device","unit":"kcal","value":588.0},{"type":"steps","origin":"device","unit":"count","value":64},{"type":"heart_rate_zone_very_low","origin":"device","unit":"s","value":0.0},{"type":"heart_rate_zone_low","origin":"device","unit":"s","value":0.0},{"type":"heart_rate_zone_medium","origin":"device","unit":"s","value":0.0},{"type":"heart_rate_zone_high","origin":"device","unit":"s","value":25320.0}],"offset_origin":"profile","segments":[],"source":{"type":"fitbit","device":null},"start_time":"2017-09-22T04:00:00Z","type":"summary","user":{"organization_id":"59b02bbdf758800001e13e75","user_id":"59b02bdef758800001e13e8c","uid":"vpt12345sp"},"user_notes":[],"utc_offset":-14400,"version":"1.0"}
event: poke
data: {"ts":"2017-09-26T16:53:41.708Z","stream":"59c51e117f3b920001aa2674","members":{"count":1,"max":20}}
Step 3: Processing the Stream
Once connected to a stream, all data received by Validic will be transmitted in a data event.
Messages streamed by this API are JSON encoded.
Keep in mind:
- Attributes of a JSON-encoded object are unordered - do not rely on fields appearing in any given order
- Your JSON parser should tolerate unexpected or missing fields.
Events streamed through the connection are delimited by consecutive \n characters:
event: data
data: {"checksum":"4d2640dc846186ecada995653da4c7c0","created_at":"2017-09-26T15:44:18.132Z","deleted_at":null,"end_time":"2017-09-26T15:43:43Z","id":"bbe980134497830de89c5e9c93cc1acd","log_id":"914247830","metrics":[{"type":"body_weight","origin":"device","unit":"kg","value":57.832},{"type":"body_fat","origin":"device","unit":"percent","value":0.0}],"offset_origin":"source","source":{"type":"nokia","device":null},"start_time":"2017-09-26T15:43:43Z","type":"measurement","user":{"organization_id":"59b02bbdf758800001e13e75","user_id":"59b02bdef758800001e13e8c","uid":"vpt12345sp"},"user_notes":[],"utc_offset":-18000,"version":"1.0"}
event: data
data: {"checksum":"685d384861f8d56dfd82515c60a26ac4","created_at":"2017-09-26T15:48:18.701Z","deleted_at":null,"end_time":"2017-09-26T15:48:04Z","id":"ad1c3d1ee0afef55d6a45af29bdcd242","log_id":"914249460","metrics":[{"type":"diastolic","origin":"manual","unit":"mmHg","value":80.0},{"type":"systolic","origin":"manual","unit":"mmHg","value":134.0},{"type":"pulse","origin":"manual","unit":"bpm","value":78}],"offset_origin":"source","source":{"type":"nokia","device":null},"start_time":"2017-09-26T15:48:04Z","type":"measurement","user":{"organization_id":"59b02bbdf758800001e13e75","user_id":"59b02bdef758800001e13e8c","uid":"vpt12345sp"},"user_notes":[],"utc_offset":-18000,"version":"1.0"}
Data Resources
Each data event sent from the streaming API will contain a JSON encoded standard object which will include a "type" attribute defining the standard object type. The data object could be one of the following resources:
| Name | Description |
|---|---|
| Measurements | Provides point in time measurements, like height, weight, blood pressure, glucose, etc. |
| Nutrition | Provides a summary of your daily nutritional intake. Cake! |
| Sleep | Provides a record of your sweet, sweet dreams. |
| Summaries | A summary of your daily activity, like steps, calories and heart rate. |
| Workouts | Provides a list of all workout events. |
When Validic receives data from a data source (Fitbit, Garmin, etc), we will pass these events to the stream in near real-time, in the order, they were received from the source.
Unique identifiers & updates
Each data object will contain a unique id which defines a unique event type received processing data events. It is possible, particularly with summary data types, to receive the same event id multiple times. Since the data is transmitted through to the client in which it was received, the last record obtained will be the most up to date record for the event.
When receiving an updated event, a customer that is persisting this data should leverage the id to find the existing record in their system. Validic provides a checksum in each event. A difference in checksum values represents modified data for records with the same id. A final comparison on the created_at value can occur if there is concern about out of order processing as the most recent created_at time represents the most up to date record.
data: {
"category":"daily",
"checksum":"c9a1eb48cbf2504e6a7a1055238ae153",
"created_at":"2017-03-29T15:31:33.959Z",
"end_time":"2017-03-30T03:59:59Z",
"id":"97a2d220620bfa419f8bb0228c5fac73",
"log_id":"2017-03-29",
"metrics":[
{
"origin":"manual",
"type":"active_duration",
"unit":"min",
"value":0.0
},
{
"origin":"manual",
"type":"basal_energy_burned",
"unit":"kcal",
"value":897.0
},
{
"origin":"manual",
"type":"distance",
"unit":"km",
"value":1.76
},
{
"origin":"manual",
"type":"steps",
"unit":"count",
"value":2415
},
{
"origin":"manual",
"type":"energy_burned",
"unit":"kcal",
"value":1184.0
}
],
"offset_origin":"profile",
"segments":[
],
"source":{
"type":"fitbit"
},
"start_time":"2017-03-29T04:00:00Z",
"type":"summary",
"user":{
"organization_id":"58cb55c98SAMPLE01f75db3",
"uid":"demotestuser1",
"user_id":"58d96aSAMPLE001a25d40"
},
"utc_offset":-14400,
"user_notes": [],
"version":"1.0"
}
Poke
This event is sent every 5 seconds to maintain a heartbeat between the streaming API and the client.
event: poke
data: {"ts":"2017-01-01T12:00:00.000Z"}
SSE Libraries
There are many libraries already built to process Server Sent Events (SSE). Here are a few we suggest
- Ruby: https://github.com/conjurinc/sse-client-ruby
- Ruby: https://github.com/af83/em-eventsource
- Java: https://github.com/mariomac/jeasse
- Go: https://github.com/r3labs/sse
- .NET: [https://github.com/3ventic/EvtSource], [https://www.nuget.org/packages/3v.EvtSource/]
To learn more about the Stream API, please review the Inform Streaming API guide.
Updated about 2 years ago