posted on 08 May 2015 by Leah Barren
Disconnects from your PowerTrack stream can happen for a handful of reasons - whether they proactively planned or unplanned. Regardless of whether or not they were planned, any sort of disconnect can be surprising cause for data loss, but they don’t have to be. A basic understanding of the types of disconnects that you might encounter and how to quickly reconnect can mean the difference between a major issue and something that can be incorporated into your application design by reconnecting, or using Backfill or Replay. This article will go over the types of disconnects you might encounter, how to minimize their effects, and how to troubleshoot issues related to disconnects.
At the highest level:
Forced disconnects happen when Gnip actively closes your connection to the stream. These can happen for a variety of reasons, and when you are force disconnected from your stream then Gnip will send a zero byte chunk in accordance with HTTP chunked encoding practice. In all cases of forced disconnects, you should be able to reconnect to the stream immediately and you should be sure to have reconnect logic written into your code to prevent further data loss.
There are three types of forced disconnects that your app will need to be prepared for.
Gnip generally has regularly scheduled maintenance every two weeks. During this maintenance, sometimes customer streams will experience one or more disconnects. This will be accompanied by a “Gnip is closing for operational reasons” message. Regardless of whether or not you will experience a disconnect, we will post about all deploys, and if you can expect a disconnect, on status.gnip.com. You can learn how to sign up for notification emails about our deploys HERE. Typically, your application should be able to reconnect immediately, so make sure that you have reconnect logic written into your application.
Full Buffer Disconnect
A full buffer disconnect generally indicates that your application’s code isn’t keeping up with the amount of data that we’re streaming to you. This can happen after a major rule change, a big event, or simply because your application is having trouble consuming the stream. Full buffer disconnects are triggered when your stream hits a certain threshold of Tweets. If you are disconnected for a full buffer, data will not begin streaming from where you were disconnected. If you find that full buffer disconnects are happening frequently, feel free to reach out to the support team to assist you in making sure that your application is properly configured. You also might want to look into the Hosebird Client for integrating with our streaming APIs.
Too Many Connections
Each stream is configured to allow for a specified number of connections. This number is determined between you and your account manager, and is available in your account agreement. If you connect to your stream with more connections than are allowed, you will be force disconnected. Any extra connections are allowed for approximately one minute. If after one minute an extra connection exists, the most recent connection is forced disconnected. Allowing an extra connection for a minute enables the customer to, for example, spin up a new server and connect with it, then teardown a server that is being ‘retired.’
A client disconnect is essentially any disconnection that occurs which isn’t initiated by our servers. This could be caused by the code of your application, or this often occurs when something in the internet or network layer cuts off the connection. This can be just random internet blips. For example when a BGP update goes awry - usually caused by routers not capable of handling the sudden additional load put on them when a route fails. It can take some time for network operators to cooperate and reroute traffic. Or occasionally customers have firewalls set up with session limits that cut off the connections after a certain amount of time, which they need to create exceptions for. From our side, our servers just see the connection close, so we don’t have a way to see whether it was closed by the proactive actions of your app, or just something in the internet.
Please note that for any disconnect, forced or client-side, your console.gnip.com dashboard will have a message that displays the kind of disconnect that you experienced and a timestamp for the disconnect.
Occasionally, some customers have trouble reconnecting to their stream after they’ve terminated a connection. Assuming there are no operational issues posted on status.gnip.com, one reason might be that something within your code is keeping the connection alive. In these scenarios, we see something in the layer outside of your app persisting, because the connection wasn’t properly terminated. Generally we see similar behavior when the HTTP client portion of the code isn’t getting proactively closed. It might also be that there is simply some network latency or an internet blip preventing the request from going through.
If you have any questions about best practices surrounding disconnects, please reach out to the support team at email@example.com.