PowerTrack Rules

Introduction  

Products utilizing PowerTrack rules deliver social data to you based on filtering rules you set up. Rules are made up of one or more ‘clauses’, where a clause is a keyword, exact phrase, or one of the many PowerTrack Operators. Before beginning to build PowerTrack rules, be sure to review the syntax described below, look through the list of available operators, and understand the restrictions around building rules. You should also be sure to understand the nuances of how rules are evaluated logically, in the ‘order of operations’ section.

Multiple clauses can be combined with both ‘and’ and ‘or’ logic. ‘And’ logic is specified with a space between clauses, and ‘or’ logic is specified with an upper-case OR. See below for more details…

Realtime and Historical PowerTrack (as well as Replay) support two forms of rules:

• ‘Standard’ rules can be up to 1,024 characters long and contain up to 30 positive clauses (things you want to match or filter on) and 50 negative clauses (things you want to exclude and not match on).
• ‘Long’ rules can be up to 2,048 characters long with no limits on the number of positive and negative clauses.

     Reach out to your account representative to switch the form of rules your stream uses.

Building Rules with PowerTrack  

Keyword match

Keyword matches are similar to queries in a search interface (e.g. Google). For example, the following PowerTrack rule would match activities with ‘happy’ in the text body.

 happy

ANDing terms with white space

Adding another keyword is the same as adding another requirement for finding matches. For example, this rule would only match activities where both ‘happy’ and ‘party’ were present in the text, in either order – having a space between terms operates as boolean AND logic. Note that if you include an explicit AND it will be treated as a regular keyword.

 happy party

ORing terms with upper-case OR

Many situations actually call for boolean OR logic, however. This is easily accomplished as well. Note that the OR operator must be upper-case and a lower-case ‘or’ will be treated as a regular keyword.

 happy OR party

Negating terms

Still other scenarios might call for excluding results with certain keywords (a boolean NOT logic). For instance, activities with ‘happy’, but excluding any with ‘birthday’ in the text.

 happy -birthday

Grouping with parentheses

These types of logic can be combined using grouping with parentheses, and expanded to much more complex queries.

 (happy OR party) (holiday OR house) -(birthday OR democratic OR republican)

This is just the beginning though – while the above examples rely simply on tokenized matching for keywords, PowerTrack also offers a operators to perform different types of matching on the text.

Exact match

 "happy birthday"

Substring match

 contains:day

Proximity match

 "happy birthday"~3

Further, other operators allow you to filter on unique aspects of social data, besides just the text. For example:

The user who is posting a Tweet

 from:user

Geo-tagged Tweets within 10 miles of Pearl St. in Boulder, CO

 point_radius:[-105.27346517 40.01924738 10.0mi]

Putting it all together

These can be combined with text filters using the same types of logic described above.

 (happy OR party) (holiday OR house OR "new year's eve") point_radius:[-105.27346517 40.01924738 10.0mi] lang:en -(birthday OR democratic OR republican)

Using various operators:

 (gnip OR from:688583 OR @gnip) ("powertrack operators" OR "streaming code"~4) contains:help bio_contains:developer has:links url_contains:github source:web (friends_count:1 OR followers_count:2000 OR listed_count:500 OR statuses_count:1000 OR is:verified OR klout_score:50) (country_code:US OR bio_location:CO OR bio_location_contains:Boulder OR time_zone:"Mountain Time (US & Canada)") -is_retweet (lang:en OR twitter_lang:en)

Also see below for converting this to JSON.

Boolean Syntax 

The examples in the previous section, utilized various types of boolean logic and grouping. See the table below for additional detail regarding the syntax and requirements for each.

Note that Operators may be either positive or negative.

Positive Operators  define what you want to include in the results. E.g. the ‘is:retweet’ operator says “I only want retweets.”

Negative Operators  define what you want to exclude from the results, and are created by using the Boolean NOT logic described above. E.g. ‘-is:retweet’ says “Exclude retweets from my rule, I only want to receive original Tweets, no retweets”

If using ‘standard’ rules, a single ‘standard’ PowerTrack rule can support up to 30 positive operators, and up to 50 negative operators, subject to a maximum length of 1,024 characters as documented here. If using ‘long’ rules, there is no limit on the number of positive and negative clauses, subject to a maximum length of 2,048 characters.

Order of Operations 

When combining AND and OR functionality in a single rule, the following order of operations will dictate how your rule is evaluated.

1. Operators connected by AND logic are combined first
2. Then, Operators connected with OR logic are applied

Example:

• apple OR iphone ipad would be evaluated as apple OR (iphone ipad)
• ipad iphone OR android would be evaluated as (iphone ipad) OR android

To eliminate uncertainty and ensure that your rules are evaluated as intended, group terms together with parentheses where appropriate. For example:

• (apple OR iphone) ipad
• iphone (ipad OR android)

Punctuation, Diacritics, and Case Sensitivity 

In PowerTrack Operators, characters with accents or diacritics are treated the same as normal characters and are not treated as word boundaries. For example, a rule of cumpleaños would only match activities containing the word cumpleaños and would not match activities containing cumplea, cumplean, or os.

All Operators are evaluated in a case-insensitive manner. For example, the rule Cat will match all of the following: cat, CAT, Cat.

Rule Tags 

As described here, each rule can be created with a tag. Each rule can have only one tag, with a maximum of 255 characters. Tags have no effect on rule uniqueness, for example identical rules with different tags will be ignored. It is not possible to have two separate rules with the same value on the same stream or job. Tags have no effect on filtering, but can create an identification for each rule which can be referenced by your app for business logic. Tags should be included with the JSON formatted rule at the time of creation via the API, as described in our documentation.

Putting Rules in JSON Format 

In order to add or delete a rule from a PowerTrack stream via the API, the rules must utilize JSON format. Essentially, this requires putting each rule into the following structure:

 {"value":"insert_rule_here"}

Rules with Double-quotes

If the ‘rule’ contains double-quote characters (“) associated with exact-match or other operators, they must be escaped using a backslash to distinguish them from the structure of the JSON format. For example, if your rule is:

 "social data" @gnip

The JSON formatted rule would be:

 {"value":"\"social data\" @gnip"}

Rules with Double-quote String Literals

To include a double-quote character as a string literal within an exact-match, it must be double-escaped. For example, for a rule matching on the exact phrase ‘Toys “R” Us’, including the double-quotes around R, the plain-text representation of this would look like the following:

  "Toys \"R\" Us"

Translating this to JSON format, you should use the following structure:

  {"value":"\"Toys \\\"R\\\" Us\""}

More examples:

{"value":"happy"}

{"value":"happy party"}

{"value":"happy -birthday"}

{"value":"happy OR party"}

{"value":"from:twitterdev"}

{"value":"(happy OR party) (holiday OR house) -(birthday OR democratic OR republican)"},

{"value":"\"happy birthday\"~3","tag":null},{"value":"(happy OR party) (holiday OR house OR \"new year's eve\") point_radius:[-105.27346517 40.01924738 10.0mi] lang:en -(birthday OR democratic OR republican)"}

 {"value":"(gnip OR from:688583 OR @gnip) (\"powertrack operators\" OR \"streaming code\"~4) contains:help bio_contains:developer has:links url_contains:github source:web (friends_count:1 OR followers_count:2000 OR listed_count:500 OR statuses_count:1000 OR is:verified OR klout_score:50) (country_code:US OR bio_location:CO OR bio_location_contains:Boulder OR time_zone:\"Mountain Time (US & Canada)\") -is_retweet (lang:en OR twitter_lang:en)"}

Rules with Tags

To include an optional Tag with your rule, as described above, simply include an additional “tag” field with the rule value:

 {"value":"\"social data\" @gnip","tag":"RULE-TAG-01"}

Formatting for API Requests

When adding or deleting rules from the PowerTrack stream via the API, multiple JSON formatted rules should be comma delimited, and wrapped in a JSON “rules” array, as shown below:

  {"rules":[{"value":"from:gnip"},{"value":"\social data\" @gnip","tag":"RULE-TAG-01"}]}

List of Operators 

Below is a list of all operators supported in PowerTrack. Note that while many operators work across multiple different data sources, others are specific to a specific source. See the Sources column for the data sources that a specific operator applies to.

Or, for a list of all the operators available for a specific source, see one of the following links.

• Disqus
• IntenseDebate
• Twitter
• Wordpress

Restrictions 

1. Stop words are not allowed as stand-alone terms in queries. If you need to find a phrase that contains a stop word, either pair it with an additional term, or use the exact match operators such as “on the roof”. As long as there is at least one required and allowed term in the rule, it will be allowed. Please note that this list of stop words is subject to change, but the current stop words we use are: "a", "an", "and", "at", "but", "by", "com", "from", "http", "https", "if", "in", "is", "it", "its", "me", "my", "or", "rt", "the", "this", "to", "too", "via", "we", "www", "you"

2. Rules cannot consist of only negated terms/operators. For example, ‘-cat -dog’ is not valid.

3. Realtime and Historical PowerTrack (as well as Replay) support two forms of rules:
   • ‘Standard’ rules:
     • The entire string for a rule may be no more than 1024 characters, including all operators and spaces, with no single term exceeding 128 characters.
     • Rules may contain no more than 30 positive operators (things you want to match or filter on). If you exceed this limit when trying to create a rule, you will receive a 422 error, with a message indicating that you have exceeded one of the clause limits.
     • Rules may contain no more than 50 negative clauses. If you exceed this limit when trying to create a rule, you will receive a 422 error, with a message indicating that you have exceeded one of the clause limits.
   • ‘Long’ rules can be up to 2,048 characters long, with no single term exceeding 128 characters, and with no limits on the number of positive and negative clauses.
     
     
          Reach out to your account representative to switch the form of rules your stream uses.
     
     

4. Negated ORs are not supported. Such as: apple OR -lang:en

5. Each rule may only have 1 tag. However, the tag is simply treated as a string, and may contain up to 255 characters, including - ; and other punctuation.

6. Geo rules with a radius greater than 25mi are not supported. Geo rules with a bounding box comprised of any edge greater than 25mi are not supported.

7. A rule keyword or input can start with either a digit (0-9) or any non-punctuation character. Current punctuation characters are defined as the ASCII characters below. Any keyword or input that needs to start with or contains punctuation must be “quoted”. A keyword can not have a colon or parentheses unless you quote it.

! % & \ ' ( ) * + - . / ; < = > ? \\ , : # @ \t \r \n " [] _
and the Unicode ranges:
U+007B -- U+00BF
U+02B0 -- U+037F
U+2000 -- U+2BFF
U+FF00 -- U+FF03
U+FF05 -- U+FF0F