Migration Guide: Moving from User Streams/Site Streams to Account Activity API

As of August 23rd, 2018, we retired both Site Streams and User Streams. Please make sure to migrate over to the Account Activity API.

Please review this announcement to learn more.

This guide is designed to help you migrate from legacy User Streams and Site Streams APIs to their replacement, the Account Activity API. Below you will find a summary of the changes, new features list, as well as key differences and considerations to help with the transition. For guidance in migrating from basic DM endpoints, please refer to the Direct Message migration guide.

Summary of changes

The Account Activity API will deliver you events for authenticated and subscribed accounts via webhooks as opposed to a streaming connection like with User Streams and Site Streams.

Deprecated APIs

GET user

GET site  (including control streams: GET site/c/:stream_id,  GET site/c/:stream_id/info.json,  GET site/c/:stream_id/friends/ids.json,  POST site/c/:stream_id/add_user.json,  POST /site/c/:stream_id/remove_user.json)

Replacement APIs

Enterprise Account Activity API - All Activities

Differences & migration considerations

API format: The new Account Activity API operates differently than User Streams and Site Streams. You will need to alter your web app to receive data with webhooks. You can find more information on webhooks here.

Data Available: Another key difference you will notice is in regards to the data being delivered. X will no longer send events from people that you follow on X (aka your home timeline). This was an intentional change and is not something we plan to alter going forward.

Reliability: Unlike streaming, webhooks enable confirmation of delivery and options to retry POSTed activities that do not make it to the webhook URL.  This gives more assurance that the app is receiving all applicable activities, even if there are brief disconnections or periods of downtime.

New features

The Account Activity API offers many new features, most notably that data is now delivered via webhooks as opposed to streaming. Webhooks offer many benefits compared to streaming, but the most prominent are speed and reliability. The API will send you data in the form of JSON events as they become available and you will no longer need to maintain an active connection or poll the endpoint. This limits the need for redundancy features and increases efficiency overall. More information on webhooks can be found in the technical documentation.

Managing user subscriptions

The Account Activity API allows multiple subscriptions for a single registered webhook.  This allows multiple user subscriptions activities to be delivered to the same location, similar to the Site Streams architecture, with webhooks.  This means you can track subscriptions, as they pertain to your subscription limits, independently from the webhook connection.  This also allows scalability from only one or a few subscriptions to thousands of subscriptions for a single webhook.

How to Migrate

Follow the steps below to easily migrate from the Site Streams API to the Account Activity API

Step 1: Decide on a Package

Depending on how you are currently operating with User Streams or Site Streams, you should consider moving to either the enterprise or premium version of the Account Activity API.  Consider the number of applications or authorized users you are currently supporting and scale appropriately to the volume and reliability needed.  When deciding on the package that best suits your needs, some things worth considering are:

• Number of webhooks needed
• Current/projected subscriptions/authorized users managed on your application
• Current number of X client applications
• The level of support you'd prefer from X (forum support or managed enterprise level 1:1 support)
• Price of each package

Step 2: Check the Setup of your X app in the developer portal

The X app currently used for User Streams or Site Streams will be listed for the owning user within the developer portal. This X app can also be used for Account Activity API to retain authorized users for that application.  A new app can also be created, and users can be re-authorized for this new app if desired.  If you are creating a new app on behalf of a business, it is recommended that you create the app with a corporate X @handle account.

• Enable “Read, Write and Access direct messages” on the permissions tab of your X app page. 
  *Note that changing these settings is not retroactive, any authorized users will keep the authorization settings from the time at which they were authorized. If a user has not already given you read, write and direct message access, you will need to have that user re-authorize your application.
• If you are unfamiliar with X Sign-in and how user contexts work with the X API review Obtaining Access Tokens.
• Generate access tokens for the X app owner at the bottom of the “Keys and Tokens” tab. On this same tab take note of your Consumer Key, Consumer Secret, Access Token and Access Token Secret. You will need these to use the API.
• Generate a bearer token using your Consumer Key and Consumer Secret for application-only API methods.

Step 3: Setup & Configure Your Webhooks

• Create a web app with an endpoint to use as your webhook to receive events (e.g. https://your_domain.com/webhook/twitter or https://webhooks.your_domain.com).
• Use your Consumer Key, Consumer Secret, Access Token and Access Token Secret when creating your webhook, Note that your endpoint must return a JSON response with a response_token that is a base64 encoded HMAC SHA-256 hash created from the crc_token and your app Consumer Secret.
  
• Review Securing Webhooks documentation taking special note of the Challenge Response Check (CRC) requirements.
• Make sure your webhook supports POST requests for incoming events and GET requests for the CRC.
• Make sure your webhook has low latency (<3 seconds to respond to POST requests)

Step 4: Validate Your Webhook Setup

• Webhook APIs will secure your webhooks in two ways:

               - Require challenge response checks to validate that the webhook owner is the web app owner.

               - A signature header in each POST request for your web app to validate the source.

• In order to verify that you are both the owner of the web app and the webhook URL, X will perform a Challenge Response Check (CRC), which is not to be confused with a cyclic redundancy check.
• A GET request with a parameter named crc_token will be sent to your webhook URL. Your endpoint must return a JSON response with a response_token that is a base64 encoded HMAC SHA-256 hash created from the crc_token and your app Consumer Secret.
• The crc_token should be expected to change for each incoming CRC request. The crc_token should be used as the message in the calculation, where your Consumer Secret is the key.
• In the event that the response is invalid, events will cease to be sent to the registered webhook.

Step 5: Create Subscriptions for Each User Stream or Site Streams Authorized User

Converting to the Account Activity API from User Streams:

• Generate a list of your current user subscriptions on User Streams
• Set up your new Account Activity API subscriptions using the request:  POST account_activity/all/:env_name/subscriptions
• Confirm your Account Activity API subscriptions using the request:  GET account_activity/all/:env_name/subscriptions/list
   

Converting to the Account Activity API from Site Streams: (using control streams):

• Generate a list of your current subscriptions on Site Streams using the request:  GET /1.1/site/c/:stream_id/info.json
• Set up your new Account Activity API subscriptions using the request:  POST account_activity/all/:env_name/subscriptions
• Confirm your Account Activity API subscriptions using the request:  GET account_activity/all/:env_name/subscriptions/list
   

Registering a Webhook and Creating Subscriptions (not migrating from Site Streams or User Streams)

• Register your webhook URL with your app using POST webhooks and receive a webhook_id.
• Use the returned webhook_id to add user subscriptions with POST webhooks/:webhook_id/subscriptions/all.

The Account Activity dashboard (sample Account Activity API application)

We've created a sample app to make testing the Account Activity API a little quicker:   

• Download the Account Activity Dashboard sample application here (it uses Node.js)
• Follow the instructions on the README to install and launch the app
• Once the application has been launched, you can use the UI to easily set up your webhook and create a new subscription

Available Activities

Deprecated streaming message types 

Deprecated event types