The Gnip Historical Power Track product is a job-based system and usage is reliant on a multiple-request API. This API is solely intended for retrieving Historical data, and is not to be used as a platform for generating volume estimates. Generation of data via the Historical Power Track product requires two distinct steps: requesting a job quote/estimate (including projected cost, data volume, and time) and accepting or rejecting a job quote/estimate.
The process is initiated with a POST request that creates a job and kicks off an estimation process. Following such a request for a quote, the job status can be monitored with GET requests until a quote/estimate is available. Once a quote is generated, a subsequent PUT request can either accept the quote and initiate data generation or reject the quote and delete it from the queue. Once job execution has begun, the job’s status can once again be monitored for completion using GET requests.
Historical data are available for 15 days after the job has been accepted.
The API is structured in a way that it is easy to integrate into a customer’s existing workflow / tools. Additionally, we have separated the quoting of and acceptance of a quote into distinct steps as we recognize the price of a Historical Power Track job can be significant and for many of our customers, those requesting data may not have the authority within their organization to spend this money.
- Each Historical Power Track job supports up to 1000 rules.
- Historical Power Track supports the same rules and operators as its real-time counterpart. Details about Power Track rules and its Twitter operators are available at:
- For help with creating PowerTrack rules using the operators referenced above, please see THIS ARTICLE.
Create / Estimate Job
Creates a job and initiates estimation of cost, time to generate, and data volume. Note: this estimate is to be used as a general indicator of the parameters of the job you are about to run so that you may reject jobs which potentially fall far outside your expectations with regard to volume and time to completion. It is not to be used as a tool to estimate volumes for jobs you might decide to run in the future. See here for documentation.
Monitor Job Status
Monitor the status of a job be it generation of a quote/estimate or job execution/data generation. See here for documentation.
Accept / Reject Job (Execute/Delete)
Accept a job estimate / quote and kickoff the data generation process or reject it and remove it from your queue. Acceptance of a job estimate equates to signing a contract and you will be billed the amount of the quote. See here for documentation.
Number of Concurrently Running Jobs
A single Gnip account is permitted to have up to 10 jobs running concurrently at any one time. This includes jobs which are in the "opened", "estimating", "accepted", or "running" statuses. However, note that jobs which are in the idle states of "quoted", "rejected", "delivered", "failed", or "paused" do not count toward this limit.
Gnip Enrichments are available in historical Twitter data from the dates specified below, moving forward:
|Gnip Language Classification||03/26/2012|
|Gnip Expanded URLs||03/26/2012|
|Basic Klout (Scores)||03/26/2012|
|Premium Klout (Topics)||08/01/2013|
Operators reliant on these enrichments are not supported for jobs with a timeframe prior to this date.
Geo-tagged data is suppressed from all Tweets prior to 9/1/2011 for Twitter Compliance reasons. As a result, all operators reliant on this geo data will not be supported for jobs with a timeframe prior to this date.
- The url_contains operator will still function prior to this date, but will only match against URLs as they are entered by a user into a Tweet and not the fully resolved URL (i.e. if a bit.ly URL is entered in the Tweet it can only match against the bit.ly and not the URL that has been shortened by bit.ly)
User Profile Data
All data prior to 1/1/2011 contains user profile information as it appeared in that user’s profile in September 2011. (e.g @jack’s very first Tweet in March 2006 contains his bio data from September 2011 that references his position as CEO at Square, which was not in existence at the time of the Tweet)
Followers and Friends Counts
All data prior to 1/1/2011 contains followers and friends counts equal to zero. As a result, any rules based on non-zero counts for these metadata will not return any results for a timeframe prior to this date.
Twitter's Language Classification
Twitter's native tweet-by-tweet language classification metadata is available in the archive beginning on March 26, 2013.
Example Code Snippets
The example code snippets at the following links are not officially supported by Gnip and only represent a basic framework of how to interact with the historical API. We encourage you to use it as a starting point and reference for building your own app.Python: