Discrepancies with Fetching User Tweets (Twitter Public Data)

There are some quirks about how the Twitter Public Data connector has to fetch the "User tweets" data that can make it appear as if there are discrepancies. Below are the most common causes of issues and how to address them.

Quick links to the sections below:

Row Count Requested Does Not Match What is Returned

This section is for the case when you have selected a specific "# of rows per item" value but the number of tweets returned is drastically lower.

Example Demonstrating the Issue

For example, say you had a setup  - getting 50 tweets for a specific account over the last 90 days. And it returns 3 tweets... The account is full of tweets, how could this be? (Account data blurred to protect client)

What is happening is that Retweets (marked with RT) and replies (marked with a @) are counted in the fetch limit (which maxes out around 2000 items per account). So the connector requests the first x item of the type you want from the time period, and if that happens to be 3, then it returns 3. If you have an account that doesn't post their own content much but does retweet and reply a lot, the spaces will be taken up by those other items.

If we turn those on in the settings to see them, 

you'll see it does actually fetch 50 items (this query starts at row 3), starting from now and counting backwards.

In this situation, this account replies to basically everyone, and their first 50 spots don't even leave the current day. This is rather extreme, as most accounts have a mix of activity, but it does show the issue quite easily.

Resolving/Working Around the Issue

Try adjusting the date range to a longer period and increasing the rows per item. This will make the query take longer to process, but you will be more likely to find what you are looking for.

Also enable those settings to see if the bulk of the rows are taken up by RT and @ tweets.

Example tweet results with orange highlights on the "@" character for replies and "RT" for retweets by the account

If the account you are checking is more extreme about retweet and replies, there's not much you can do as the API can only the serve about the first ~2000 items for that account (regardless of what type the are).

Not Getting Tweets from Full Date Range

This section is for the case when you requested a certain date range but are only seeing a small section of tweets.

Example Demonstrating the Issue

Let's say you wanted to get all the tweets an account made last month but only see the most recent 10. You know there's more content than that for the month, and it's only getting the end of the month.

Resolving/Working Around the Issue

This is simply a configuration issue - the "# of rows per item" dictates what will be shown and the counter is always looking from now into the past. So it will find the most recent 10 tweets from that account, within the date range set.

If you want more tweets, then increase your "# of rows per item" value to be large enough to cover the whole month (or whatever your time period is).

"No Data" Even Though There are Tweets

This covers the case where there are tweets in the past for an account, but it keeps returning "no data" when you try to fetch them.

Resolving/Working Around the Issue

This issue also comes from the "# of rows per item" setting being too small. It will fetch the first 50 tweets from now, which does not extend far enough in time to capture these. The date range is more like a filter than an actual part of the request, so to fix this, increase the "# of rows per item" and then you should be able to get the older data.

Fetching Older Data/Historical Tweets

This section is for the case where you are trying to pull all tweets for an account, but it stops early.

Resolving/Working Around the Issue

This is caused by the fact the API can't serve more than about ~2000 objects for the account at once. So starting from now, it will go back as far as it can until it bumps that limit, and will be the end of the data. The more active an account is, the smaller the historical window will be, as every tweet, reply and retweet that account made takes up one spot.

Unless the account is not that active, it will be quite difficult to get all tweets due to the rate limiting on the API.

Did you find it helpful? Yes No

Send feedback
Sorry we couldn't be helpful. Help us improve this article with your feedback.