This page is designed to help you migrate to the new Volume Streams 2.0. We’ve provided a summary and detailed description of the changes below to help with the transition from version 1.0 to 2.0.

Note: We will announce significant changes to any of the Volume Streams in the Change Log found on the Volume Streams 2.0 home page.

Summary of Changes 

  • New Endpoint URLs
  • Enrichments Support
  • Stream Partitioning
  • Backfill behavior

New Endpoint URLs  

Below are the new endpoint URLs for version 2.0 of the Volume Streams:

Decahose:

https://gnip-stream.twitter.com/stream/sample10/accounts/{ACCOUNT_NAME}/publishers/twitter/{STREAM_LABEL}.json

Firehose:

https://gnip-stream.twitter.com/stream/firehose/accounts/{ACCOUNT_NAME}/publishers/twitter/{STREAM_LABEL}.json

User Mention:

https://gnip-stream.twitter.com/stream/mentions/accounts/{ACCOUNT_NAME}/publishers/twitter/{STREAM_LABEL}.json

Enrichment Support  

  • Deprecating Gnip’s Language Classification enrichment in favor of Twitter’s language classification which supports far more languages.
  • Profile Geo is available in Volume Streams 2.0
  • The Enhanced URLs enrichment is available in Volume Streams 2.0 (with additional metadata)
  • A new, enhanced version of Klout is supported in Volume Streams 2.0

Stream Partitioning  

The use of partitioning in 2.0 is now required in order to consume the stream in full. To do this, you will need to connect to the stream with the partition={#} parameter. The total number of partitions vary by stream - see details below:

Decahose

There are 2 partitions, each containing 50% of volume of the Decahose stream. You will need to connect to both paritions in order to consume the entirety of the Decahose - see examples below:

https://gnip-stream.twitter.com/stream/sample10/accounts/{accountName}/publishers/twitter/{streamLabel}.json?partition=1

https://gnip-stream.twitter.com/stream/sample10/accounts/{accountName}/publishers/twitter/{streamLabel}.json?partition=2

Firehose

There are 20 partitions, each containing 5% of volume of the full Firehose. You will need to connect to all 20 paritions in order to consume the entirety of the Firehose - see examples below:

https://gnip-stream.twitter.com/stream/firehose/accounts/{accountName}/publishers/twitter/{streamLabel}.json?partition=1

https://gnip-stream.twitter.com/stream/firehose/accounts/{accountName}/publishers/twitter/{streamLabel}.json?partition=2

...

https://gnip-stream.twitter.com/stream/firehose/accounts/{accountName}/publishers/twitter/{streamLabel}.json?partition=20

User Mention

There are 8 partitions, each containing 12.5% of volume of the User Mention stream. You will need to connect to all 8 partitions in order to consumer the entirety of the stream:

https://gnip-stream.twitter.com/stream/firehose/accounts/{accountName}/publishers/twitter/{streamLabel}.json?partition=1

https://gnip-stream.twitter.com/stream/firehose/accounts/{accountName}/publishers/twitter/{streamLabel}.json?partition=2

...

https://gnip-stream.twitter.com/stream/firehose/accounts/{accountName}/publishers/twitter/{streamLabel}.json?partition=8



Note about Redundant Connections: If your stream is setup for redundant connections (e.g. two connections), then you can connect to each partition two times. If you’re setup for four total connections, then you can connect to each partition four times.


Backfill Behavior  

Backfill works a bit different in Volume Streams 2.0. You must specify the the number of minutes (1-5, whole numbers only) using the new ‘backfillMinutes’ parameter as part of your connection request (see example below).

Example request using Backfill:

https://gnip-stream.twitter.com/stream/firehose/accounts/{ACCOUNT_NAME}/publishers/twitter/{STREAM_LABEL}.json?partition=1&backfillMinutes=5

Replay  

Replay 2.0 is now available! It works the same as version 1.0, but includes support for the 2.0 PowerTrack operators and enhanced Enrichments.


Managing Compliance  

Important Note: Compliance will not be included in version 2.0 of the Firehose or Decahose streams; therefore, consumption of the Compliance Firehose 2.0 will be required to manage compliance.

Please reference the Compliance Firehose 2.0 documentation for more details.