Twitter entities  

Introduction

Entities provide metadata and additional contextual information about content posted on Twitter. The entities section provides arrays of common things included in Tweets: hashtags, user mentions, links, stock tickers (symbols), Twitter polls, and attached media. These arrays are convenient for developers when ingesting Tweets, since Twitter has essentially pre-processed, or pre-parsed, the text body. Instead of needing to explicitly search and find these entities in the Tweet body, your parser can go straight to this JSON section and there they are.

Beyond providing parsing conveniences, the entities section also provides useful ‘value-add’ metadata. For example, if you are using the Enhanced URLs enrichment, URL metadata include fully-expanded URLs, as well as associated website titles and descriptions. Another example is when there are user mentions, the entities metadata include the numeric user ID, which are useful when making requests to many Twitter APIs.

Every Tweet JSON payload includes an entities section, with the minimum set of hashtags, urls, user_mentions, and symbols attributes, even if none of those entities are part of the Tweet message. For example, if you examine the JSON for a Tweet with a body of “Hello World!” and no attached media, the Tweet’s JSON will include the following content with entity arrays containing zero items:

"entities": {
    "hashtags": [
    ],
    "urls": [
    ],
    "user_mentions": [
    ],
    "symbols": [
    ]
  }

Notes:

• media and polls entities will only appear when that type of content is part of the Tweet.
• if you are working with native media (photos, videos, or GIFs), the Extended Entities object is the way to go.

Entities object

The entities and extended_entities sections are both made up of arrays of entity objects. Below you will find descriptions for each of these entity objects, including data dictionaries that describe the object attribute names, types, and short description. We’ll also indicate which PowerTrack Operators match these attributes, and include some sample JSON payloads.

A collection of common entities found in Tweets, including hashtags, links, and user mentions. This entities object does include a media attribute, but its implementation in the entiites section is only completely accurate for Tweets with a single photo. For all Tweets with more than one photo, a video, or animated GIF, the reader is directed to the extended_entities section.

Entities data dictionary

The entities object is a holder of arrays of other entity sub-objects. After illustrating the entities structure, data dictionaries for these sub-objects, and the Operators that match them, will be provided.

{
  "hashtags": [
    {
      "indices": [
        32,
        38
      ],
      "text": "nodejs"
    }
  ]
}

{
  "media": [
    {
      "display_url": "pic.twitter.com/5J1WJSRCy9",
      "expanded_url": "https://twitter.com/nolan_test/status/930077847535812610/photo/1",
      "id": 9.300778475358126e17,
      "id_str": "930077847535812610",
      "indices": [
          13,
          36
      ],
      "media_url": "http://pbs.twimg.com/media/DOhM30VVwAEpIHq.jpg",
      "media_url_https": "https://pbs.twimg.com/media/DOhM30VVwAEpIHq.jpg"
      "sizes": {
          "thumb": {
               "h": 150,
               "resize": "crop",
               "w": 150
          },
          "large": {
              "h": 1366,
              "resize": "fit",
              "w": 2048
          },
          "medium": {
              "h": 800,
              "resize": "fit",
              "w": 1200
          },
          "small": {
              "h": 454,
              "resize": "fit",
              "w": 680
          }
      },
      "type": "photo",      
      "url": "https://t.co/5J1WJSRCy9",
    }
  ]
}

{
  "urls": [
    {
      "indices": [
        32,
        52
      ],
      "url": "http://t.co/IOwBrTZR",
      "display_url": "youtube.com/watch?v=oHg5SJ…",
      "expanded_url": "http://www.youtube.com/watch?v=oHg5SJYRHA0"
    }
  ]
}

{"urls": [
      {
        "url": "https://t.co/D0n7a53c2l",
        "expanded_url": "http://bit.ly/18gECvy",
        "display_url": "bit.ly/18gECvy",
        "unwound": {
          "url": "https://www.youtube.com/watch?v=oHg5SJYRHA0",
          "status": 200,
          "title": "RickRoll'D",
          "description": "http://www.facebook.com/rickroll548 As long as trolls are still trolling, the Rick will never stop rolling."
        },
        "indices": [
          62,
          85
        ]
      }
    ]
}

{
  "user_mentions": [
    {
      "name": "Twitter API",
      "indices": [
        4,
        15
      ],
      "screen_name": "twitterapi",
      "id": 6253282,
      "id_str": "6253282"
    }
  ]
}

{
  "symbols": [
    {
      "indices": [
        12,
        17
      ],
      "text": "twtr"
    }
  ]
}

{"polls": [
      {
        "options": [
          {
            "position": 1,
            "text": "I read documentation once."
          },
          {
            "position": 2,
            "text": "I read documentation twice."
          },
          {
            "position": 3,
            "text": "I read documentation over and over again."
          }
        ],
        "end_datetime": "Thu May 25 22:20:27 +0000 2017",
        "duration_minutes": 60
      }
    ]
  }

Hashtag object  

The entities section will contain a hashtags array containing an object for every hashtag included in the Tweet body, and include an empty array if no hashtags are present.

The PowerTrack # Operator is used to match on the text attribute. The has:hashtags Operator will match if there is at least one item in the array.

"indices":[32,38]

"text":"nodejs"

Media object  

The entities section will contain a media array containing a single media object if any media object has been ‘attached’ to the Tweet. If no native media has been attached, there will be no media array in the entities. For the following reasons the extended_entities section should be used to process Tweet native media:
+ Media type will always indicate ‘photo’ even in cases of a video and GIF being attached to Tweet.
+ Even though up to four photos can be attached, only the first one will be listed in the entities section.

The has:media Operator will match if this array is populated.

"display_url":"pic.twitter.com/rJC5Pxsu"

"expanded_url": "http://twitter.com/yunorno/status/114080493036773378/photo/1"

"id":114080493040967680

"id_str":"114080493040967680"

"indices":[15,35]

"media_url":"http://pbs.twimg.com/media/DOhM30VVwAEpIHq.jpg"

"media_url_https":"https://p.twimg.com/AZVLmp-CIAAbkyy.jpg"

{
  "sizes": {
    "thumb": {
      "h": 150,
      "resize": "crop",
      "w": 150
    },
    "large": {
      "h": 1366,
      "resize": "fit",
      "w": 2048
    },
    "medium": {
      "h": 800,
      "resize": "fit",
      "w": 1200
    },
    "small": {
      "h": 454,
      "resize": "fit",
      "w": 680
    }
  }
}

"source_status_id": 205282515685081088

"source_status_id_str": "205282515685081088"

"type":"photo"

"url":"http://t.co/rJC5Pxsu"

Media size objects

All Tweets with native media (photos, video, and GIFs) will include a set of ‘thumb’, ‘small’, ‘medium’, and ‘large’ sizes with height and width pixel sizes.  For photos and preview image media URLs, Photo Media URL formatting specifies how to construct different URLs for loading different sized photo media.

Sizes object 

"thumb":{"h":150, "resize":"crop", "w":150}

"large":{"h":1366, "resize":"fit", "w":2048}

"medium":{"h":800, "resize":"fit", "w":1200}

"small":{"h":454, "resize":"fit", "w":680}

Size object 

Photo Media URL Formatting

Photo media on Twitter can be loaded in different sizes.  It is best to load the smallest size image that is larger enough to fit into a particular image viewport.  To load different sizes, the Size Object and media_url (or media_url_https) need to be combined in a particular format.  We'll use the media entity example object already provided for our example in constructing a photo media URL.

The media_url or media_url_https on their own can be loaded, which will result in the medium variant being loaded by default.  It is preferable, however, to provide a fully formatted photo media URL when possible.

There are three parts of a photo media URL:

We take these three parts (base URL, format and name) and combine them into the photo media URL to load.  There are 2 formats for loading images this way, legacy and modern.  All image loads should stop using the legacy format and use the modern format.  Using the modern format will result in better CDN hit rate for the caller, thus improving load latencies by being less likely to have to generate and load the media from the Data Center.

URL object 

The entities section will contain a urls array containing an object for every link included in the Tweet body, and include an empty array if no links are present.

The has:links Operator will match if there is at least one item in the array. The url: Operator is used to match on the expanded_url attribute. If you are using the Expanded URL enrichment, the url: Operator is used to match on the unwound.url (fully unwound URL) attribute. If you are using the Enhanced URL enrichment, the url_title: and url_decription: Operators are used to match on the unwound.title and unwound.description attributes.

"display_url":"bit.ly/2so49n2"

"expanded_url":"http://bit.ly/2so49n2"

"indices":[30,53]

"url":"https://t.co/yzocNFvJuL"

If you are using the Expanded and/or Enhanced URL enrichments, the following metadata is available under the unwound attribute:

"url":"https://blog.twitter.com/en_us/topics/insights/2016/using-twitter-as-a-go-to-communication-channel-during-severe-weather-events.html"

200

"title":"Using Twitter as a ‘go-to’ communication channel during severe weather"

"description":"Using Twitter as a ‘go-to’ communication channel during severe weather"

User mention object  

The entities section will contain a user_mentions array containing an object for every user mention included in the Tweet body, and include an empty array if no user mention is present.

The PowerTrack @ Operator is used to match on the screen_name attribute. The has:mentions Operator will match if there is at least one item in the array.

"id":6253282

"id_str":"6253282"

"indices":[4,15]

"name":"Twitter API"

"screen_name":"twitterapi"

Symbol object  

The entities section will contain a symbols array containing an object for every $cashtag included in the Tweet body, and include an empty array if no symbol is present.

The PowerTrack $ Operator is used to match on the text attribute. The has:symbols Operator will match if there is at least one item in the array.

"indices":[12,17]

"text":"twtr"

Poll object

The entities section will contain a polls array containing a single poll object if the Tweet contains a poll. If no poll is included, there will be no polls array in the entities section.

Note that these Poll metadata are only available with the following Enterprise APIs:

• Volume streams (Decahose )
• Real-time PowerTrack
• Historical PowerTrack
• Twitter Search APIs (Full-Archive Search and 30-Day Search)

{"options": [
          {
            "position": 1,
            "text": "I read documentation once."
          }
      ]
}

"end_datetime": "Thu May 25 22:20:27 +0000 2017"

"duration_minutes": 60

Retweet and Quote Tweet details

From the Twitter API perspective, Retweet and Quote Tweets are special kinds of Tweets that contain the original Tweet as an embedded object. So Retweets and Quote Tweet objects are parents of a child 'original' Tweet (and thus double the size). Retweets have a top-level "retweeted_status" object, and Quoted Tweets have a "quoted_status" object. For consistency, these top-level Retweet and Quote Tweet objects also have a text property and associated entities. However, the entities at the top level can differ from the entities provided by the embedded 'original' entities. In case of Retweets, new text is prepended to the original Tweet body. For Quoted Tweets, new text is appended to the Tweet body.

In general, the best practice is to retrieve the text, entities, original author and date from the original Tweet in retweeted_status whenever this exists. An exception is getting Twitter entities that are part of the additive Quote. See below for more details and tips.

Retweets

An important detail with Retweets is that no additional Twitter entities can be added to the Tweet. Users can not add hashtags, URLs or other details when they Retweet. However, the Retweet (top-level) text attribute is composed of the original Tweet text with “RT @username: ” prepended.  

In some cases, especially with accounts with long user names, the combination of these new characters and the original Tweet body can easily exceed the original Tweet text length limit of 140 characters. In order to preserve support for 140 character based display and storage, the top-level body truncates the end of the Tweet body and adds an ellipsis (“…”). Consequently, some top-level entities positioned at the end of the original Tweet might be incorrect or missing, for instance in the case of a truncated hashtag or URL entry.

This Tweet,  https://twitter.com/FloodSocial/status/907974220298125312, has the following Tweet text:

               Just another test Tweet that needs to be exactly 140 characters with trailing URL and hashtag http://wapo.st/2w8iwPQ  #Testing

In the above example, both the URL and hashtag were affected. Since the hashtag was completely truncated and the URL partially truncated, these are missing from the the top-level entities. You will also notice the additional user_mentions top-level entity coming from the “RT @floodsocial: ” prefix on the text field.

However, the Tweet text and entities in retweeted_status perfectly reflect the original Tweet with no truncation or incorrect entities, hence our recommendation to rely on the nested retweeted_status object for Retweets.

Quote Tweets

Quote Tweets were introduced in 2016, and differ from Retweets in that when you "quote" a Tweet you are adding new content "on top" of a shared Tweet. This new content can include nearly anything an original Tweet can have, including new text, hashtags, mentions, and URLs.

Quote Tweets can contain native media (photos, videos, and GIFs), and will appear under the entities object.

Since Twitter entities can be added, the Quote entities are likely different from the original entities.

In this example, a new URL and hashtag were positioned at the end of the Quote Tweet.

This Tweet, https://twitter.com/FloodSocial/status/907983973225160704, has the following Tweet text:

                  strange and equally tragic when islands flood... trans-atlantic testing of quote tweets | @thisuser @thatuser http://bit.ly/2vMMDuu  #testing

In this case, the top-level entities do not reflect the Quote details. 

However, the Tweet text and entities in extended_tweet perfectly reflect the Quote Tweet with no truncation or incorrect entities, hence our recommendation to rely on the nested extended_tweet object for Quote Tweets.

Entities for user object

Entities for User Objects describe URLs that appear in the user defined profile URL and description fields. They do not describe hashtags or user_mentions. Unlike Tweet entities, user entities can apply to multiple fields within its parent object — to disambiguate, you will find a parent nodes called url and description that indicate which field contains the entitized URL.

In this example, the user url field contains a t.co link that is fully expanded within the entities/url/urls[0] node of the response. The user does not have a wrapped URL in their description.

JSON example

Entities for Direct Messages

Entities for Direct Messages are very similar to entities for Tweets. However, there are a few differences concerning the media entities.

Unlike media shared in Tweets, media shared in Direct Messages requires authorization to view. This authorization can be presented via an authenticated twitter.com session or by signing a request with the User’s access token using OAuth 1.0A.

Also, in Tweets, media URLs are only in the media entities, but in Direct Messages, media URLs are in both media and URLs entities.

JSON example

Next steps

Explore other Tweet JSON objects and data dictionaries:

• Extended Entities object and data dictionary
• Tweet object and data dictionary
• User object and data dictionary
• Tweet geo objects and data dictionaries