Hello and welcome in Keboola’s new Data Integration Tragicomedy Series! In the form of short blog posts we would like to present you a funny situations and desperate hours of debugging we spent on integration of many famous APIs.
Our first story is about Twitter Ads API. It really breaks my heart to write about it, because for many years Twitter was a thought leader and pioneer in terms of an API design and execution. But nowadays – not so much.
It all started with very common task: Download performance data for campaigns in Twitter Account, with month of history, segmented by days. Easy, right? With experiences from Adwords, Facebook Ads and others I thought it is a piece of cake. So, let’s dive in.
Don’t mess with the IDs.
Generally accepted best practice in the advertising systems similar to Twitter Ads is that you have some kind of ID for each account and campaign. And those IDs can be seen in URLs or in the source code of the page, so when you need to debug it, just compare campaign with the same ID in the UI and data downloaded from API. With Twitter Ads this is, well, awkward. You can see account’s ID in the URL, but no campaign. According to the comments from Twitter devs, you cannot see the ID used in API for a campaign in the UI. You need to do separate API call to get the names and IDs of all campaigns. Which makes matching campaigns data from UI and API based on campaign name kind of a hell, but hey, it gets better…
Really helpful error messages and consistency issues
Now that we have all campaign IDs loaded, we can get the historical performance data. For this Twitter has two approaches – synchronous and asynchronous. Synchronous is restricted to 7 days of data and you will get the data in the response. The endpoint looks like this: /stats/accounts/ with entity and metrics supplied. You need to specify IDs of the campaigns you want to get data for (maximum is 20).
You will get the data, process it and store. But where is the fun in doing 5 requests to get a month worth of history?
So, let’s move to asynchronous requests. Classic for generating reports – create a job, wait for the status to be done, downloaded result and store. The endpoint is very very similar: /stats/jobs/accounts/ with the same parameters. What could go wrong?
Everything. In the same time I moved from synchronous to asynchronous requests, suddenly the API started to respond with “Not Authenticated” error message. This is OAuth2 application owned by us with access rights to the account of a client, so first I thought it is really something about the authentication.
After some time spent checking all the different things that could go wrong with authentication I got cool idea – to compare my requests with the ones from twiurl, official tool from Twitter. Immediately, something caught my eye – all special characters were URL encoded.
There are 2 ways to get campaign data from Twitter Ads API. Synchronous endpoint “/stats/accounts/” and asynchronous one “/stats/jobs/accounts/”. Both have the same parameters. Although the synchronous works perfectly fine without URL encoding, asynchronous one returns “Not Authenticated” when the URL of the requests is not URL encoded.
And I am just sitting here, trying to ingest what just happened.
Thanks a lot for an attention folks. I hope that I save someone few minutes or hours with this little hint!
Did you encounter something similar? Don’t suffer quietly and let us know!
Stay tuned for more similar WTF stories, we have quite few up in our sleeve, don’t forget to subscribe to our blog!