Monday, October 24, 2016

GameDevMap Scraping with Python

GameDevMap.com is an excellent resource for finding game companies across the world. I wanted to see which companies had jobs that I might be interested in but doing it manually would be insurmountable- cue Python.

You can find the script on github

The script uses BeautifulSoup and urllib in Python 3.5.4. It gets all of the info listed on GameDevMap as well as a search for the company on duckduckgo.com (it has a nice sans-js mode). The search (ie: "careers Westwood Studios") is then combed for keywords, in this case, "jobs", "careers", "programmer", "engineer", "developer", "unity". Unity was a little harder because words like "opportunity" and "community" gave false positives. The search runs two versions and when it's done, writes to csv. The csv can then be imported into Google Sheets where it can be viewed and sorted.

Check out my sheets import here

A nice future improvement would be multithreading which would speed things up quite a bit. Feel free to do what you like with the script, let me know if it helps.

No comments:

Post a Comment