Enhanced URLs


Overview

Twitter Data Products’ enhanced URL enrichment automatically expands shortened URLs that are included in the body of a Tweet, and includes the resulting URL as metadata within the payload. In addition to that, this new enrichment also provides HTML page metadata from the Title and Description of the destination page. This new HTML page metadata was not previously available through Gnip products and provides better filtering and additional insight into linked content.

Access

Twitter Data Products’ enhanced URL enrichment is only available in a new version of PowerTrack, which is currently in limited Beta. There are no plans to add the new enrichment to the current version of PowerTrack.

Tweet Payload

  • The URL Unwind enrichment delivers the following pieces of metadata as part of the JSON payload:
    • Shortened URL
    • Expanded URL
    • Expanded HTTP Status
    • (NEW) Expanded URL HTML Title (300 character limit)
    • (NEW) Expanded URL HTML Description (1000 character limit)

PowerTrack Filtering

New PowerTrack operators filter and provide a tokenized match on the following pieces of URL metadata:

  • url:
    • Example: “url:tennis”
    • Tokenized match on any Expanded URL that includes the word tennis
    • Could also be used as a filter to include or exclude links from specific website using something like “url:npr.org”
  • url_title:
    • Example: “url_title:tennis”
    • Tokenized match on any Expanded URL HTML Title that includes the word tennis
    • Matches on the HTML Title data included in the payload, which is limited to 300 characters.
  • url_description:
    • Example: “url_description:tennis”
    • Tokenized match on any Expanded URL HTML Description that includes the word tennis
    • Matches on the HTML Description included in the payload, which is limited to 1000 characters.

Tweet Payload Example

Our current design is to include this additional metadata as part of the gnip.urls section of the JSON payload, highlighted below:

        {
            "gnip": {
                "urls": [
                    {
                        "url": "http://t.co/tzUB3cEtKR",
                        "expanded_url": "http://espn.go.com/video/clip?id=12994775&utm_source=twitterfeed&utm_medium=twitter",
                        "expanded_status": 200
                        "expanded_url_title": "Who has the best bullpen in the AL? - ESPN Video",
                        "expanded_url_description": "Chris Singleton and Doug Glanville break down why the Royals have the best bullpen in the American League.",
                    }
                ],
                "klout_score": 40,
                "klout_profile": {
                    "topics": [
                        {
                            "klout_topic_id": "7008943767379250621",
                            "displayName": "Sports",
                            "link": "http://klout.com/topic/id/7008943767379250621"
                        },
                        {
                            "klout_topic_id": "8582816108322836738",
                            "displayName": "Football",
                            "link": "http://klout.com/topic/id/8582816108322836738"
                        }
                    ],
                    "klout_user_id": "44754526062629965",
                    "link": "http://klout.com/user/id/44754526062629965"
                },
                "language": {
                    "value": "en"
                }
            }
        }