The Gnip 30-Day Search API provides low-latency, full-fidelity, query-based access to 30 days of historical data with minute granularity. Data is available within 30 seconds of generation on the Twitter platform. We’ve introudced several new operators and features with version 2.0, some of which are highlighted below with links to more details.

New Features 

Below is a list of some of the new operators and features available in 2.0. For more details, see “More Details” button.

  • New limit of 2048 characters per rule, with no limit on the number of positive clauses or negative clauses
  • New “has:{images, videos, symbols}” operator support
  • $ cashtag operator support
  • Enrichment - Enhanced URL Expansion
  • Enrichments and Matching rules in Original Format

More Details »


Request Types  

The 30-Day Search API supports two types of requests:

1. Search Requests (data)

Search requests to the data endpoint allow you to retrieve up to 500 results at a time (per request) for a given timeframe in the last 30 days. It can be used to retrieve the most recent results for a high volume query, all results for a small time slice, or all of the results in the last 30 days for a low volume query. The Search endpoint is ideal for allowing users to search for recent data for a query or topic. Searches can be refined to any timeframe in the last 30 days to analyze the data in that given timeframe.

Results provided via Search are compliant at the time of delivery.

2. Counts Requests (Tweet count)

Requests to the Counts endpoint provide the ability to retrieve historical activity counts (volume), which reflect the number of activities that match a given query during the requested timeframe. These requests may define bucket sizes for the generation of the counts, which define the time period of each count. The bucket size may be per day, hour, or minute.

These counts do not reflect any later compliance events (deletions, scrub geos), and some activities which are counted may not be available to retrieve with a Search request due to user compliance actions.


Rules & Filtering  

In addition to adding new operators, we’ve made some changes to a few of the existing operators.

New Operators:
  • has:images
  • has:videos
  • has:symbols
  • $ (cashtag)

For a complete list of supported operators, click the link below:

List of Operators »

Altered Operators:
  • url_contains:
  • twitter_lang:
  • has:lang:
  • bio_lang:

For more details on the altered operators and their replacements, click the link below:

Replaced Operators »


Data Updates and Mutability 

Unlike Gnip’s other historical products (Historical PowerTrack, Replay), some of the data within a Tweet is mutable (i.e. can be updated or changed after initial archival).

Mutable data falls into two categories:

  1. Metadata around a user/actor object.
    • user’s @handle
    • bio description
    • follower’s count, friends count, listed count, statuses count
  2. Tweet statistics (i.e. anything that can be changed on the platform by user action) See examples below:
    • retweet count
    • favorites count

In most of these cases, the Search API will return data as it exists on the platform at query-time, rather than Tweet generation time. However, in the case of queries using select operators (e.g. from, to, @, is:verified), this may not be the case. Data is updated in our index on a regular basis, with an increased frequency for most recent timeframes. As a result, in some cases the data returned may not exactly match the data that was queried for, but matches data at the time it was last indexed.

Note: this issue of inconsistency only applies to queries where the operator applies to mutable data (e.g. user and user-bio related operators and counts-based operators). We will not be supporting many such operators initially, but will offer more in the future. The most problematic example is indeed filtering for usernames, and the best workaround would be to use userIDs rather than @handles for queries including the from, to or @ operators.


Enrichments Support 

  • Support for Original Format: Original format streams will now include enrichments in the response payload, such as the “matching_rules” element.
  • Profile Geo Enrichment: there is no filtering support for Profile Geo, but it is included in the response payload.
  • Deprecation Gnip’s Language Classification: We’ve deprecated the Gnip language classification enrichment in favor of Twitter’s language classification which supports far more languages.
  • Enhanced URL Expansion - Tweets returned by the Search API now include the enhanced URL Expansion enrichment including the URL’s Title and Meta Description.