Writing an API Wrapper for a Private API

Around 2010 I started playing a browser game called Grepolis. Back then it was very simple only having a browser client to play the game. When I started getting in to web related development I wrote a couple of scripts to gather the public data for the game that is still hosted in automatically updating gzipped csv files to make it easier for developers to use the data since even back then I thought that using gzipped csv files was not a very developer friendly way to share the game data. Since then I have tried this idea with a couple languages and eventually released a small lightweight NodeJS library that simply acted as a middle man to gather the data and transform it into JSON arrays. I quickly lost interest in this project because it's really simple and I didn't see a way to expand upon it at the time.

Then a couple weeks ago someone reached out to me via email and we chatted on Discord about the library. They asked me how I was able to reverse the Grepolis API which I explained that I didn't have to do for this as this data is known from a forum post many years ago. This sparked a new interest for me though because I thought "Grepolis has an app now, so they must have an API" and this thought gave me a way to expand upon my library and make it way more useful. Up until this point you could only read public data such as large lists of players, cities, battle points, etc. but if I could reverse the private API then I could create a full API wrapper for Grepolis which had not yet been done.

I set out on this task knowing a bit about sniffing communication between apps and their backend as I had done this once before using Charles Proxy but I never really liked that tool so I wanted to find something new to use that is easier and more robust. I landed on Proxyman which has all of the features I needed, in fact I only learned enough about the program to do what I needed which is sniffing the connection and decrypting the encrypted HTTPS traffic.

After configuring Proxyman on my computer and setting up the proxy on my iPhone I started seeing packets from en0.grepolis.com and en45.grepolis.com I play on the en45 "Hyperborea" world now so I knew this is what I was looking for. Inspecting the packet data it was clear that the structure for authentication and entering a game world went something like this:

1. Login to Grepolis via en0 which passes back a sessionid that we save for later
2. Login to game world with the sessionid from earlier, this gives a game session id (aka session_id) that we can use with the game server en45
3. Finalize game world login (not sure if this is required) this seems to just be something that the app does, i'm not sure that it provides value for the library other than giving us an initial town id initial_town_id because we do need a town id to start talking with game world endpoints
4. Start sending requests as if you're a user on the app!

I also noticed a typical pattern with this API at this point, now that I had a couple of requests written and tested I looked at the structure of each POST body to try and recognize these patterns to reduce redundancy in the codebase, and there was quite a bit of redundancy since one of the goals of this project is to not use any third party runtime libraries.

I noticed that there were two main sections for this API there are the non-world-specific ones such as login which use en0.grepolis.com and world specific ones that communicate with the game server itself, in this case en45.grepolis.com aside from the host being different the path is the same for both /api albeit with various args depending on the server and what you're trying to do. Some common args are ssid which is our session id token from login that's used to communicate with non-game-specific API's and gsid which is the game session token used to communicate with world specific servers. The body content typically had the following structure:

{
  method_name: String,
  class_name: String,
  params: {
    nlreq_id: Number,
    nl_skip: Boolean,
    ...etc
  }
}

and the headers for each request had some items of note as well:

{
  'Accept': '*/*',
  'Accept-Language': 'en-us',
  'x-flash-version': '33,1,1,345',
  'X-API-VERSION': '1',
  'Accept-Encoding': 'gzip, deflate, br',
  'Referer': 'app:/GrepolisApp-store.swf',
  'User-Agent': '',
  'Connection': 'keep-alive',
  'Content-Type': 'application/x-www-form-urlencoded'
}

With this information I decided to write a Util.js which contains generic GET and POST functions and I decided to match the class_name and method_name with the library to make it more understandable. This means that if we have a request like this:

{
  class_name: 'ApiMasterAuthentication',
  method_name: 'login'
}

You can call it in the library with ApiMasterAuthentication.login(...)

I ran in to a couple of endpoints that gave me some weird data so I assumed from the headers that some of the responses were gzipped so I wrote gzip decompression into some of the Util.js functions to properly handle this from the endpoints I noticed.

This project is still a work in progress but it's been a neat and interesting experience so far. You can check out the repository for updates as I go through the private API and write wrapper functions for it. I think with this article it's important to remember if you're writing an API that's open to the public no matter how you attempt to secure communication between your frontend and backend there is always a way around it and as such you should focus your backend development on writing secure API's.