Advanced filtering with geo data

Note: the payload also contains a "geo" object that has the coordinates in reverse order [LAT, LONG], but it is deprecated and we recommend using the "coordinates" object instead.

Place data is rendered in the root-level "place" object of a Tweet payload. Places can have the following types: POI, neighborhood, city (most common), country, and admin. Here's an example below:

2. Profile location (account level)

Prevalence: ~30-40% of Tweets contain some profile location information.

The user object contained within a Tweet payload may also contain geographic information. Users can customize their profile through several fields, including a "location" field. If present, the Profile Geo Enrichment (only available with enterprise APIs) adds structured geodata relevant to the user provided location value by geocoding and normalizing location strings where possible.

Note: The user provided location field accepts any valid string, even if it isn't a real location or place. Users may choose to input a fictional place or something generic that isn't tied to an actual location. In these instances, the profile geo enrichment will not add any structured metadata (as you might expect) and the "derived" object will not be present in the payload. 

The user defined location is rendered in the "user.location" object of a Tweet payload (see below):

For enterprise API customers that have Profile Geo enabled, the additional metadata is found in the "user.derived" object of a Tweet payload (see below):

Example at the region/state level:

Example at the locality (city) level:

A more nuanced example with sub_region (county level):

London is the capital and largest city of both England and the United Kingdom. Below shows how it is represented in the profile geo part of the payload:

In summary, these are the two types of geographical metadata that you can expect to find in Tweets when working with the API. It's important to keep in mind that both types of geo metadata are elective, and controlled by the user. Additionally, Twitter does not validate the location information provided at the Tweet or account level against IP address or other sources of signal; therefore, the geo metadata may not represent where a user was when a Tweet was sent, or where they're geographically based (profile location).

List of Operators

The tables below list the available operators for each type of geographical metadata as well as a brief description and which field in the payload the operator matches on.
 

Tweet location

Profile location

*The only operator that doesn't use/require the enterprise Profile Geo enrichment.
 

Proxy geo operators

The table below contains operators that can proxy for geographic information:

Note: The keyword and exact phrase operators above can be used to match explicit mentions of a place in a Tweet.
 

Building effective filters

Upfront decisions

Before you create any queries or rules, you'll need to decide up front whether or not you want to filter on geo-tagged Tweets, profile location, or a combination of both. This really depends on your use case and how you plan to use the Tweets collected from the API. For example, if you want to plot Tweets on a map for visual display, it may be best to focus on geo-tagged Tweets. However, if you want to follow conversations or analyze sentiment of a topic and how it varies across regions, the comparatively large volume of Tweets with profile location may suit you better.

Another factor to consider is the availability of the data. As mentioned in the review section (<anchor link>), approximately 1-2% of Tweets are geo-tagged; but 30-40% of Tweets published contain some profile location information. Given the limited volume of geo-tagged Tweets, it can be difficult to satisfy a use case that's focused solely on Tweets with geo dat In that case, you may want to explore filtering on profile location in addition to, or in place of, geo-tagged Tweets.
 

Get place IDs

Some of the examples below will utilize a "place ID" (e.g., 96683cc9126741d1). There are two ways to programmatically retrieve Twitter place information:

1. GET geo/reverse_geocode - takes a latitude and a longitude and retrieves up to 20 places (note: the place ID returned can be used as the value with the 'place:' operator)
2. GET geo/search - search by lat/long or query (free form text) to get a list of all the valid places (note: the place ID returned can be used as the value with the 'place:' operator)

Let's use the GET geo/search endpoint to get the place ID for the “United States,” which we'll use in the first real world example below:

Request:

twurl '/1.1/geo/search.json?query=United%20States&granularity=country&max_results=2'

Response:

Two results were returned and we can quickly see that we want the "id": "96683cc9126741d1" field from the "United States" place.
 

Real-world examples

In this section, we'll use a real world use case and walk through how you can apply geo filters to a rule at different levels of specificity: country, state/province, city, and neighborhood level. Geo data is often times just one element of a query or rule that you're looking to build, so for the purposes of this section, we'll use the following scenario:

Use case: Collect Tweets about the Tokyo Summer Olympic Games with varying levels of granularity. More specifically, we will use the following elements to compose the rule:

1. Mentions: @Tokyo2020, @Olympics
   1. This will capture both @ mentions and replies to the specified handle.
2. Hashtags: #Tokyo2020
3. Keywords: olympics, "olympic games", "olympic team"
4. Exclude Retweets: -is:retweet
   1. Note: A Retweet cannot be tagged with a location; therefore, we do not need to add this negation clause to rules that target geo-tagged Tweets. It will only be added in the rules that target profile geo.
5. Geo filters: these will change based on each level of granularity below.

The conditions above will be combined together, as clauses, to form the following rule value (see documentation on building rules):

The results count above represents Tweet volume for a 3-month period (March-June 2021) for the query before any geo filters are applied. In the sections below, the results are also shared for each filter option to illustrate the impact of the geo filters on the volume of matching Tweets over the same period.

Next, we're going to add geo filters at varying levels of granularity – starting with the country level.
 

How to filter for Tweets at the country level

Now, let's say that we want to limit our Tweet results to those that are in a specific country. We must first identify the operators that provide the ability to filter at the country level:

1. place_country:
2. profile_country:
3. place:

We'll use the United States as an example. The following are a few options to accomplish this:

Now, let's add in the additional clauses to create a rule value that encompasses our use case and compare the results (note: the results represent Tweet volume that matches each query from March-June 2021):

You'll notice that the results increase significantly (20x) when using the profile_country operator instead of place_country. This is because significantly more Tweets contain profile location compared to Tweet-level location. Additionally, very few Tweets matched the specific place ID for the "United States." The place ID has to be an exact match; it doesn't encompass all of the places such as states and cities that roll up to the US as the place_country operator does.
 

How to filter for Tweets at the state/province level

In most cases, "region" represents the state/province level, so the following two operators can help us collect Tweets at the state/province level:

1. place:
2. profile_region:

We'll use California as our example state that we want to limit our results to. The following are a few options to accomplish this:

Now, let's add in the additional clauses to create a rule value that encompasses our use case and compare the results (note: the results represent Tweet volume from March-June 2021):

As seen above, the results increased significantly again with the profile location clause (profile_region) instead of the Tweet-level location clause (place:CA). This is expected as 30-40% of Tweets contain profile location information; whereas, only 1-2% of Tweets are geo-tagged. While there were 3x more Tweets that matched the specific place ID for California, it was still nominal compared to the other filter options.

Important notes about filtering for the "state" level:

• The "place:CA" clause will NOT match Tweets tagged at the neighborhood level or tagged with a point of interest (POI). These would need to be captured by using bounding boxes (covered in the next section) or with additional place operator clauses that reference the city name.
• For example, "place:Los Angeles" will match on this pattern: "place.full_name": "{Neighborhood}, Los Angeles"
• The clause, "place:California", will only match Tweets tagged at the state-level  ("place.full_name":"California, USA"). This won't include a majority of Tweets tagged at the city level because the place object doesn't contain the word, "California" ("place.full_name":"San Francisco, CA")
   

How to filter for Tweets at the city level

Locality generally refers to “city level". The following operators can help us collect Tweets at the city level:

1. place:
2. point_radius:
3. bounding_box:
4. profile_locality:

To continue with the West Coast (US) example, we'll use San Francisco as the example city here. The following are a few options to help us filter results to San Francisco, CA:

Now, let's add in the additional clauses to create a rule value that encompasses our use case and compare the results (note: the results represent Tweet volume from March-June 2021):

How to filter for Tweets at the neighborhood level

Neighborhood level filtering may be useful for very granular filtering where city-level is too broad. The following operators can help us collect Tweets at the neighborhood level:

1. place:
2. point_radius:
3. bounding_box:
4. bio_location:

 

To continue with our theme, we'll use neighborhoods in the San Francisco area for the examples below. The following are a few options to help us filter results to the SoMa neighborhood in San Francisco (note: the results represent Tweet volume from March-June 2021):

Note: given the level of granularity here and the low likelihood of Tweets matching both the geo clause and use case clauses (e.g., the Olympics), we’ve run the numbers without the use case clauses: