Compliance Firehose Overview
The Compliance Firehose is a realtime streaming API that delivers compliance events that occur on the Twitter platform. For an understanding of compliance events and how they are generated on Twitter, please reference our article, Honoring User Intent on Twitter.
Please note that the Compliance Firehose currently delivers compliance events related to the Twitter Platform only.
There are only two changes to highlight as you transition from version 1.0 to 2.0 of the Twitter Compliance Firhose:
Stream partitioning - 8 partitions, each containing 12.5% of volume of the Compliance stream
Output format - Activity Streams is no longer available as an output format option. The stream will only return data in original format.
Possible types of compliance events will include Tweet (or “status”) events and User events, for which there are multiple types described below. Refer to the ‘Recommended Action’ column to understand how to process each type of event in order to respect the privacy and intent of the end user. For examples of how these compliance events are represented in the JSON payload, please see our Twitter data format page.
|Original Message Type||Object||Permanent (Yes/No)||Recommended Action|
|delete||Status||Yes||Delete associated Tweet.|
|status_withheld||Status||Yes||Suppress associated Tweet in specific countries listed in the status_withheld message.|
|user_delete||User||No||Suppress or delete all Tweets by associated user.|
|user_undelete||User||No||All Tweets by associated user may be displayed again and treated as public.|
|user_protect||User||No||Suppress or delete all Tweets by associated user.|
|user_unprotect||User||No||All Tweets by associated user may be displayed again and treated as public.|
|user_suspend||User||No||Suppress or delete all Tweets by associated user.|
|user_unsuspend||User||No||All Tweets by associated user may be displayed again and treated as public.|
|scrub_geo||User||Yes||Delete all geodata provided by Twitter for all Tweets by the user prior to the specified Tweet in the scrub_geomessage. Note that subsequent Tweets by a user may contain geodata that may be used.|
|user_withheld||User||Yes||Suppress Tweets by associated user in specific countries listed in the user_withheld message.|
It is important to note that Tweet and User events are delivered independently and that each should be processed independently (i.e. a Tweet delete doesn’t imply a User event, and vice versa.) Several User events are not necessarily permanent and can toggle between states infinitely. These include: user_delete,user_undelete, user_protect, user_unprotect and user_suspend, user_unsuspend.
User_deletes are followed by status_deletes 30 days later only if the user has not selected to user_undelete their account. It is possible that a user_delete is reversed by the user and deletes for all of their tweets 30 days later do not occur.
User_suspend is an action that remains true unless the user is subject to an user_unsuspend event. These are not subject to any changes on a 30 day time period.
It is never suitable to display compliance events directly to users of your software or to otherwise incorporate them into your products or customer experiences. They are intended solely for maintaining compliance and honoring the actions of Twitter users.
To integrate the Compliance Firehose into your system, you will need to build an integration that can do the following:
- Establish a streaming connection to the Compliance Firehose stream API
- Handle high data volumes – de-couple stream ingestion from additional processing using asynchronous processes
- Reconnect to the stream automatically when disconnected for any reason
- Process compliance events that are relevant to Tweet and User data you have stored in accordance with the guidance presented above
For information on consuming a realtime data stream, see HERE.