Friday, 16 November 2012

Anomalies in Steam Community data

In a recent post I introduced the Steam Community API, and showed how to retrieve gamer data and perform a few simple but fun analyses.

While writing the posting, I came across several problems associated with the data that's returned. If you're thinking about using Steam Community data, it's worth bearing these anomalies in mind because of the impact they'll have on downstream processing and further analysis.

Frustratingly, the quality of the data available through the Steam Community API is quite variable - in particular there are many discrepancies between global achievement data compared to achievement data for individual players. I also came across several global achievement rates that were clearly invalid, and in some cases found that global achievement records for games were totally missing.

The net result: it's hard to trust that the data that's returned. It is still possible to analyze returned data, but you're going to need strong validation, normalization (e.g. of player achievements against a 'gold standard'), and potentially multiple attempts to retrieve equivalent data to ensure what you have is accurate.

Below, you'll find a (non-exhaustive) list of data quality issues I came across, along with some examples, and a little discussion about the problems they introduced and workarounds I used.

Disclaimer: the issues described here reflect my experience while using the Steam API to retrieve data in bulk, to allow me to analyse data for a large number of gamers. Your experience may differ - if so, please let me know.

"Test" achievements

One of the most obvious problems you might find are spurious achievements associated with games. These include a few that can easily be filtered ("null" or empty names) as well as others that are more problematic - such as many 'test' achievements. For example:

TEST_ACHIEVEMENT
TestAchievement
Achievement_Test

Those readers familiar with the Valve games Portal and Portal 2 will realize why these can't be easily filtered - many valid achievement names include some variation of the word "test", e.g. portal_escape_testchambers.

I only spotted these when querying global achievement statistics, so it's possible that problem may just be a filtering issue on that particular API endpoint.

Another example can be seen when compare my (woeful) achievements for AaAaAA!!! - A Reckless Disregard for Gravity with the global achievement data. You should see one extra achievement testo2 which a vanishingly small number of people have achieved - more than likely because it's a leftover artifact from when the game was integrated into Steam.

Out of date achievement lists

Another closely related issue is that personal achievement lists can get out of step with global achievement lists, casting doubt on the reliability of any comparisons made between player achievements.

For example while processing the global record for The Legend of Grimrock, I noticed two additional achievements in comparison to my record:

FIND_ALL_TREASURES, complete_game_normal

It's worth noting that the FIND_ALL_TREASURES achievement appears in lower case in my record, but the complete_game_normal entry was missing completely. As a result, it's necessary to normalize all achievement records for players before making any comparisons, which unfortunately means making assumptions about why entries are missing (e.g. that the game hasn't been played recently) and how to fix the data.

Interestingly, since last viewing the global stats for this game, the data has become one of the ...

Missing global records

A more severe, though in some ways easier to handle issue is that global stats for some games are just missing - though this seems to be an intermittent issue.

The aforementioned Legend of Grimrock is currently one of them, as was Civilization V a couple of weeks ago. This seems to be an API specific issue, because the equivalent website for The Legend of Grimrock shows many achievements as I write this.

It seems that obtaining global stats for games is a bit of a hit-and-miss affair, so be careful with any apparently empty lists of achievements you may see and don't assume that such responses are correct when considering further processing.

Achievements with huge percentages

The final issue I've come across is that some of the global stats for achievements are simply incorrect. For example RAGE by iD Software has several achievements held by over 730,000% of players.

Thankfully this particular issue is easily detected, and offending achievements can be filtered easily.

Photo credit: wilhei55 / Foter / CC BY

Sunday, 4 November 2012

Harvesting Data from the Steam Community API

Introduction

The Steam community API is a web service that provides public access to information about Steam users, their games, achievements, and other related information. In this blog posting I'll describe some of the interesting data you can access, as well as how to model, retrieve, and process that data. I'll also show you how to generate a few fun, simple rankings and statistics for a group of steam gamers.
 
This is primarily a technical article, but it concludes with the results of a simple analysis performed over a small number of friends and aquaintances on Steam, which may be of interest to the non-technically inclined.

The examples shown here can be reproduced using the sample code found in this GitHub repository. It's a work in progress, but hopefully provides enough insight so you can either repeat the results or build your own equivalent.

Accessing the API

The first thing to know is that Steam community data is accessed using a RESTful web service, through a number of related endpoints. Many of the endpoints don't require authentication, but some require you to register for a key which you then provide as a parameter when interacting with the API.

You'll find links to the API documentation below - see the first link for details on how to get a key:

Steam Web API Documentation (high level)
Steam Web API Reference
Steam Web API Self documenting API endpoint
How to access Community Data 

The "Web API" supports both XML and JSON formats, while the closely related "Community Data" endpoints only support XML - it appears the latter are just public pages with an additional parameter of xml=1. In the rest of this posting, I provide XML examples for consistency, but the JSON resources seem to be equivalent. All of the URIs described below can be accessed using the HTTP GET verb, and in all cases appear to be browser-friendly (try clicking the examples).

There are also one or two client libraries available for different languages, notably steam condenser which is available for Java, PHP, and Ruby. Unfortunately I hit a bug caused (I believe) by changes to the behaviour of the steam API, and ultimately decided to using directly HTTP given that the API is relatively straightforward.

Available data

What kind of data can be accessed via the API? Some of the most interesting types of data are user profiles and user game lists, along with user achievements, which many users choose to make public. It's also possible to retrieve global achievement lists for games, which include percentages showing the proportion of players with the game who have a given achievement.
There's actually lots more information available, such as friend lists, statistics on play time, etc. But for now, let's focus on the above.

To make it easier to work with the data, it helps to establish a core domain model. That is, a set of concepts and relationships describing the problem domain. This helps with understanding the data, reasoning about how to process it, and further down the line how describe the data in code.

Given that we're interested in users, games, and achievements, the domain model is fairly simple:


Simple UML domain model for player data. Diagram courtest ObjectAid.

The above diagram was generated from core domain classes in the sample project, using a view-only UML modeling tool called ObjectAid. Aside from the three main concepts, you'll see relationships representing the fact that users have games, that games have achievements associated with them, and that users have acheivements either held or yet to be achieved. You'll also see a few attributes for key data such as steam ID, game name, etc.

Data retrieval

Before retrieving data from the various API endpoints, you'll need to find one or more Steam IDs. There are a few different ways of referring to Steam users, these include personas (nicknames), login account names, identifiers reported by game servers that start STEAM_, and 64 bit community IDs.

We're interested in the 64 bit steam community variants, which unfortunately require a little effort to obtain. A good starting point is your profile page which can be accessed via an "id" or via the unique profile ID - the behaviour of the "id" endpoint is ambiguous, but it appears to attempt to resolve users by their registered nicknames. I ended up viewing the page source on my friend list page or on specific profiles in order to obtain community IDs. It's also worth noting that the API endpoint for retrieving friend lists may be the most reliable method.

If you have a Steam ID in one of the other formats, you might look into one of the sites dedicated to converting between the various ID formats (example). However none of the sites I found generated 64 bit community IDs so friend lists may be the best option.

Once you have an ID or two, you can start to pull down some data. Below, you'll find an outline of key API interactions needed to get player data, game lists, and achievement information.

User profiles
First up, user profile data. You can get individual player summaries using a simple variant on the user profile link - just add xml=1 and you'll receive a computer readable version (example). Alternatively, use the Steam Web API to retrieve player summaries in batch, as follows:

URI:
http://api.steampowered.com/
ISteamUser/GetPlayerSummaries/v0002/?
key=[YOUR KEY HERE]&steamids=[STEAM 64 IDS]&format=[xml OR json]



For the sample stats shown at the end of this posting, I just needed the user's persona name which I retrieved using the second method shown above.

User game lists
A user's game list can also be retrieved using a simple variation on a web page URI. In this case, add /games?xml=1 to the end of a profile page URI (example) and you should have a complete list of games owned by the player.

The key data need you'll need to pull out of the response is the appID - that is, the unique identifier for the game. The response also includes other player data you might find useful, including game names, play time, etc.

User achievement lists
Once you've retrieved a list of unique game IDs for a particular player, it's possible to start retrieving something a little more interesting - individual achievements for those games.

Again, you can obtain a player's achievements for a game in two ways: using a profile page URI, and using the Web API. Helpfully, you'll find links to a player's game achievements in the game list response. Adapt these by adding xml=1 and you're away (example - warning: possible game spoilers for XCOM: Enemy Unknown). I actually used the alternative provided by the Web API, as follows:

URI:
http://api.steampowered.com/
ISteamUserStats/GetPlayerAchievements/v0001/?
appid=[GAME ID]&steamid=[STEAM 64 ID]&key=[YOUR KEY HERE]&format=[xml OR json]

The key information you'll need here are the unique identifiers for the achievements (apiname) and the flag indicating whether or not the player holds that achievement (achieved, values 1 or 0).

Global achievement data
The final endpoint that's worth reviewing retrieves global achievement lists and percentages for games. This data only appears to be available via an unauthenticated endpoint through the Web API (example), as follows:

URI:
http://api.steampowered.com/
ISteamUserStats/GetGlobalAchievementPercentagesForApp/v0002/?
gameid=[GAME ID]&format=[xml OR json]
  
Example response

This achievement data is useful both for cross-referencing with user achievement data, and for comparing individual achievements with global levels. The code in the sample project pulls down this data and uses it to validate and normalize user achievement lists.

Putting it all together

So by now, it's hopefully clear what kind of data you can access via the Steam Community API, and how to retrieve it using the various HTTP endpoints. But that's not quite enough to be able to start working with the data.

Below, you'll find an outline of the steps required to put together a cohesive data model that can then be analysed, persisted, and processed further:

For one or more Steam 64 identifiers:
- Retrieve user profile data, create a user record.
- Read the list of user games (capturing game IDs), associate them with the user.
- Read user achievements per game (capturing game ID, plus achievement ID and status) associate with user.

The end result should be a collection of users, each with an associated set of games, and for each user/game pair, a set of achievements both held and yet to be achieved. In the sample project, I use Java classes to hold user, game, and achievement entities, along with a few Collection objects to record game lists and achievement outcomes. The sample code also retrieves global achievement data for validation and normalization.

Popular games, Biggest Achievers

Using the above approach, I harvested data for eight Steam friends and aquaintances, then generated a list of the top games for the social group along with a ranking of the most accomplished players.

Image courtesy of smarnad / FreeDigitalPhotos.net
First, the top 10 games, according to popularity:

Pop.GameAch'ments
8Portal15
7Portal 250
6Amnesia: The Dark Descent0
6Counter-Strike: Global Offensive193
6Counter-Strike source148
6Counter-Strike source: Beta154
6Half-Life 2: Deathmatch0
6Half-Life 2: Lost Coast0
6Left 4 Dead 269
6Super Meat Boy49

No surprises there then. Portal and Portal 2 being highly popular, followed by a few of the top indie games and several stock Valve games. The only slightly puzzling thing being that Half-Life 3 sorry I mean 2 appears at position 11 (not shown), and is only shared by five players, while six have a copy of HL2: Deathmatch. The likely reason being that one person owns the Source Multiplayer pack which includes HL2: Deathmatch.

Next, who are the biggest achievers. Names have been anonymized to protect the innocent:

PlayerAchievmentsRatio
Chas1000/394925%
Olly at home DOTT789/655012%
Shreddies574/324518%
gryffindorpotential433/369712%
cryptoGoat193/198910%
HappyKittens40/6576%
unpronouncable29/11183%
Cuppa11/21741%

Congratulations Chas, the runaway winner with one thousand achievements, and the highest proportion of possible achievements held. Coming in close behind are Olly and Shreddies, Olly holding the higher number of achievements but Shreddies having achieved a higher relative proportion. Bringing up the rear, the wooden spoon award goes to Cuppa has both the lowest number of achievements overall, and the lowest overall proportion.

These particular stats are just for fun and are shouldn't be taken too seriously, but they do hint at some more compelling uses of the data. For example, it would be interesting to go beyond pure rankings and further analyse the achievements held by gamers from a particular social group. But that's the topic of a future post.

A final note on data reliability

One final thing to mention is that my experience with the quality of data exposed by the Steam Web API has been mixed. The data exposed by the by the community pages (i.e. the public pages, with xml=1 added) seems more reliable than the data provided Steam Web API. One reason might be that the Web API provides less filtering, while the profile page is designed to be human readable and thus may be subject to greater filtering or curation. 

Next time, I'll be discussing those issues in more detail as well as discussing the importance of, and problems associated with, obtaining reliable data.