X entities  

Introduction

Entities provide metadata and additional contextual information about content posted on X. The entities section provides arrays of common things included in Posts: hashtags, user mentions, links, stock tickers (symbols), X polls, and attached media. These arrays are convenient for developers when ingesting Posts, since X has essentially pre-processed, or pre-parsed, the text body. Instead of needing to explicitly search and find these entities in the Post body, your parser can go straight to this JSON section and there they are.

Beyond providing parsing conveniences, the entities section also provides useful ‘value-add’ metadata. For example, if you are using the Enhanced URLs enrichment, URL metadata include fully-expanded URLs, as well as associated website titles and descriptions. Another example is when there are user mentions, the entities metadata include the numeric user ID, which are useful when making requests to many X APIs.

Every Post JSON payload includes an entities section, with the minimum set of hashtags, urls, user_mentions, and symbols attributes, even if none of those entities are part of the Post message. For example, if you examine the JSON for a Post with a body of “Hello World!” and no attached media, the Post's JSON will include the following content with entity arrays containing zero items:

"entities": {
    "hashtags": [
    ],
    "urls": [
    ],
    "user_mentions": [
    ],
    "symbols": [
    ]
  }

Notes:

• media and polls entities will only appear when that type of content is part of the Post.
• if you are working with native media (photos, videos, or GIFs), the Extended Entities object is the way to go.

Entities object

The entities and extended_entities sections are both made up of arrays of entity objects. Below you will find descriptions for each of these entity objects, including data dictionaries that describe the object attribute names, types, and short description. We’ll also indicate which PowerTrack Operators match these attributes, and include some sample JSON payloads.

A collection of common entities found in Posts, including hashtags, links, and user mentions. This entities object does include a media attribute, but its implementation in the entiites section is only completely accurate for Posts with a single photo. For all Posts with more than one photo, a video, or animated GIF, the reader is directed to the extended_entities section.

Entities data dictionary

The entities object is a holder of arrays of other entity sub-objects. After illustrating the entities structure, data dictionaries for these sub-objects, and the Operators that match them, will be provided.

{
  "hashtags": [
    {
      "indices": [
        32,
        38
      ],
      "text": "nodejs"
    }
  ]
}

{
  "media": [
    {
      "display_url": "pic.twitter.com/5J1WJSRCy9",
      "expanded_url": "https://twitter.com/nolan_test/status/930077847535812610/photo/1",
      "id": 9.300778475358126e17,
      "id_str": "930077847535812610",
      "indices": [
          13,
          36
      ],
      "media_url": "http://pbs.twimg.com/media/DOhM30VVwAEpIHq.jpg",
      "media_url_https": "https://pbs.twimg.com/media/DOhM30VVwAEpIHq.jpg"
      "sizes": {
          "thumb": {
               "h": 150,
               "resize": "crop",
               "w": 150
          },
          "large": {
              "h": 1366,
              "resize": "fit",
              "w": 2048
          },
          "medium": {
              "h": 800,
              "resize": "fit",
              "w": 1200
          },
          "small": {
              "h": 454,
              "resize": "fit",
              "w": 680
          }
      },
      "type": "photo",      
      "url": "https://t.co/5J1WJSRCy9",
    }
  ]
}

{
  "urls": [
    {
      "indices": [
        32,
        52
      ],
      "url": "http://t.co/IOwBrTZR",
      "display_url": "youtube.com/watch?v=oHg5SJ…",
      "expanded_url": "http://www.youtube.com/watch?v=oHg5SJYRHA0"
    }
  ]
}

{"urls": [
      {
        "url": "https://t.co/D0n7a53c2l",
        "expanded_url": "http://bit.ly/18gECvy",
        "display_url": "bit.ly/18gECvy",
        "unwound": {
          "url": "https://www.youtube.com/watch?v=oHg5SJYRHA0",
          "status": 200,
          "title": "RickRoll'D",
          "description": "http://www.facebook.com/rickroll548 As long as trolls are still trolling, the Rick will never stop rolling."
        },
        "indices": [
          62,
          85
        ]
      }
    ]
}

{
  "user_mentions": [
    {
      "name": "Twitter API",
      "indices": [
        4,
        15
      ],
      "screen_name": "twitterapi",
      "id": 6253282,
      "id_str": "6253282"
    }
  ]
}

{
  "symbols": [
    {
      "indices": [
        12,
        17
      ],
      "text": "twtr"
    }
  ]
}

{"polls": [
      {
        "options": [
          {
            "position": 1,
            "text": "I read documentation once."
          },
          {
            "position": 2,
            "text": "I read documentation twice."
          },
          {
            "position": 3,
            "text": "I read documentation over and over again."
          }
        ],
        "end_datetime": "Thu May 25 22:20:27 +0000 2017",
        "duration_minutes": 60
      }
    ]
  }

Hashtag object  

The entities section will contain a hashtags array containing an object for every hashtag included in the Post body, and include an empty array if no hashtags are present.

The PowerTrack # Operator is used to match on the text attribute. The has:hashtags Operator will match if there is at least one item in the array.

"indices":[32,38]

"text":"nodejs"

Media object  

The entities section will contain a media array containing a single media object if any media object has been ‘attached’ to the Post. If no native media has been attached, there will be no media array in the entities. For the following reasons the extended_entities section should be used to process Post native media:
+ Media type will always indicate ‘photo’ even in cases of a video and GIF being attached to Post.
+ Even though up to four photos can be attached, only the first one will be listed in the entities section.

The has:media Operator will match if this array is populated.

"display_url":"pic.twitter.com/rJC5Pxsu"

"expanded_url": "http://twitter.com/yunorno/status/114080493036773378/photo/1"

"id":114080493040967680

"id_str":"114080493040967680"

"indices":[15,35]

"media_url":"http://pbs.twimg.com/media/DOhM30VVwAEpIHq.jpg"

"media_url_https":"https://p.twimg.com/AZVLmp-CIAAbkyy.jpg"

{
  "sizes": {
    "thumb": {
      "h": 150,
      "resize": "crop",
      "w": 150
    },
    "large": {
      "h": 1366,
      "resize": "fit",
      "w": 2048
    },
    "medium": {
      "h": 800,
      "resize": "fit",
      "w": 1200
    },
    "small": {
      "h": 454,
      "resize": "fit",
      "w": 680
    }
  }
}

"source_status_id": 205282515685081088

"source_status_id_str": "205282515685081088"

"type":"photo"

"url":"http://t.co/rJC5Pxsu"

Media size objects

All Posts with native media (photos, video, and GIFs) will include a set of ‘thumb’, ‘small’, ‘medium’, and ‘large’ sizes with height and width pixel sizes.  For photos and preview image media URLs, Photo Media URL formatting specifies how to construct different URLs for loading different sized photo media.

Sizes object 

"thumb":{"h":150, "resize":"crop", "w":150}

"large":{"h":454, "resize":"fit", "w":680}

"medium":{"h":800, "resize":"fit", "w":1200}

"small":{"h":1366, "resize":"fit", "w":2048}

Size object 

Photo Media URL Formatting

Photo media on X can be loaded in different sizes.  It is best to load the smallest size image that is larger enough to fit into a particular image viewport.  To load different sizes, the Size Object and media_url (or media_url_https) need to be combined in a particular format.  We'll use the media entity example object already provided for our example in constructing a photo media URL.

The media_url or media_url_https on their own can be loaded, which will result in the medium variant being loaded by default.  It is preferable, however, to provide a fully formatted photo media URL when possible.

There are three parts of a photo media URL:

We take these three parts (base URL, format and name) and combine them into the photo media URL to load.  There are 2 formats for loading images this way, legacy and modern.  All image loads should stop using the legacy format and use the modern format.  Using the modern format will result in better CDN hit rate for the caller, thus improving load latencies by being less likely to have to generate and load the media from the Data Center.

URL object 

The entities section will contain a urls array containing an object for every link included in the Post body, and include an empty array if no links are present.

The has:links Operator will match if there is at least one item in the array. The url: Operator is used to match on the expanded_url attribute. If you are using the Expanded URL enrichment, the url: Operator is used to match on the unwound.url (fully unwound URL) attribute. If you are using the Exhanced URL enrichment, the url_title: and url_decription: Operators are used to match on the unwound.title and unwound.description attributes.

"display_url":"bit.ly/2so49n2"

"expanded_url":"http://bit.ly/2so49n2"

"indices":[30,53]

"url":"https://t.co/yzocNFvJuL"

If you are using the Expanded and/or Enhanced URL enrichments, the following metadata is available under the unwound attribute:

"url":"https://blog.twitter.com/en_us/topics/insights/2016/using-twitter-as-a-go-to-communication-channel-during-severe-weather-events.html"

200

"title":"Using Twitter as a ‘go-to’ communication channel during severe weather"

"description":"Using Twitter as a ‘go-to’ communication channel during severe weather"

User mention object  

The entities section will contain a user_mentions array containing an object for every user mention included in the Post body, and include an empty array if no user mention is present.

The PowerTrack @ Operator is used to match on the screen_name attribute. The has:mentions Operator will match if there is at least one item in the array.

"id":6253282

"id_str":"6253282"

"indices":[4,15]

"name":"API"

"screen_name":"api"

Symbol object  

The entities section will contain a symbols array containing an object for every $cashtag included in the Post body, and include an empty array if no symbol is present.

The PowerTrack $ Operator is used to match on the text attribute. The has:symbols Operator will match if there is at least one item in the array.

"indices":[12,17]

"text":"twtr"

Poll object

The entities section will contain a polls array containing a single poll object if the Post contains a poll. If no poll is included, there will be no polls array in the entities section.

Note that these Poll metadata are only available with the following Enterprise APIs:

• Volume streams (Decahose )
• Real-time PowerTrack
• Historical PowerTrack
• X Search APIs (Full-Archive Search and 30-Day Search)

{"options": [
          {
            "position": 1,
            "text": "I read documentation once."
          }
      ]
}

"end_datetime": "Thu May 25 22:20:27 +0000 2017"

"duration_minutes": 60

Retweet and Quote Tweet details

From the X API perspective, Retweet and Quote Tweets are special kinds of Posts that contain the original Post as an embedded object. So Retweets and Quote Tweet objects are parents of a child 'original' Post (and thus double the size). Retweets have a top-level "retweeted_status" object, and Quoted Tweets have a "quoted_status" object. For consistency, these top-level Retweet and Quote Tweet objects also have a text property and associated entities. However, the entities at the top level can differ from the entities provided by the embedded 'original' entities. In case of Retweets, new text is prepended to the original Post body. For Quoted Posts, new text is appended to the Post body.

In general, the best practice is to retrieve the text, entities, original author and date from the original Post in retweeted_status whenever this exists. An exception is getting X entities that are part of the additive Quote. See below for more details and tips.

Retweets

An important detail with Retweets is that no additional X entities can be added to the Post. Users can not add hashtags, URLs or other details when they Retweet. However, the Retweet (top-level) text attribute is composed of the original Post text with “RT @username: ” prepended.  

In some cases, especially with accounts with long user names, the combination of these new characters and the original Post body can easily exceed the original Post text length limit of 140 characters. In order to preserve support for 140 character based display and storage, the top-level body truncates the end of the Post body and adds an ellipsis (“…”). Consequently, some top-level entities positioned at the end of the original Post might be incorrect or missing, for instance in the case of a truncated hashtag or URL entry.

This Post,  https://twitter.com/FloodSocial/status/907974220298125312, has the following Post text:

               Just another test Post that needs to be exactly 140 characters with trailing URL and hashtag http://wapo.st/2w8iwPQ  #Testing

In the above example, both the URL and hashtag were affected. Since the hashtag was completely truncated and the URL partially truncated, these are missing from the the top-level entities. You will also notice the additional user_mentions top-level entity coming from the “RT @floodsocial: ” prefix on the text field.

However, the Post text and entities in retweeted_status perfectly reflect the original Post with no truncation or incorrect entities, hence our recommendation to rely on the nested retweeted_status object for Retweets.

Quote Tweets

Quote Tweets were introduced in 2016, and differ from Retweets in that when you "quote" a Pos you are adding new content "on top" of a shared Post. This new content can include nearly anything an original Post can have, including new text, hashtags, mentions, and URLs.

Quote Tweets can contain native media (photos, videos, and GIFs), and will appear under the entities object.

Since X entities can be added, the Quote entities are likely different from the original entities.

In this example, a new URL and hashtag were positioned at the end of the Quote Tweet.

This Post, https://twitter.com/FloodSocial/status/907983973225160704, has the following Post text:

                  strange and equally tragic when islands flood... trans-atlantic testing of quote tweets | @thisuser @thatuser http://bit.ly/2vMMDuu  #testing

In this case, the top-level entities do not reflect the Quote details. 

However, the Post text and entities in extended_tweet perfectly reflect the Quote Tweet with no truncation or incorrect entities, hence our recommendation to rely on the nested extended_tweet object for Quote Tweets.

Entities for user object

Entities for User Objects describe URLs that appear in the user defined profile URL and description fields. They do not describe hashtags or user_mentions. Unlike Post entities, user entities can apply to multiple fields within its parent object — to disambiguate, you will find a parent nodes called url and description that indicate which field contains the entitized URL.

In this example, the user url field contains a t.co link that is fully expanded within the entities/url/urls[0] node of the response. The user does not have a wrapped URL in their description.

JSON example

Next Steps

Explore the other sub-objects that a Post contains:

• Post object and data dictionary
• User object and data dictionary
• Extended Entities object and data dictionary
• Post geo objects and data dictionaries