Lineup display issue and resolution
First, our apologies in taking so long to post a response but we wanted to ensure we had all the facts and the issue was researched appropriately before putting out a statement.
Here’s a brief update on last night’s issues. We have a team working on both aspects of this:
1. Making sure that we settle contests in a way that none of condia’s opponents feel that they’ve been adversely effected by the contest results. This may take several more hours.
2. Making sure that this can’t happen again.
In the meantime, here’s a detailed technical explanation of what happened. Feel free to ask questions about it, but please understand that our focus right now is on getting the issue fixed so we may not get to answer follow up questions today.
The way our system works is that we have a main database which holds all the entries and updates a user has made to their lineups. This is updated directly when a user makes changes, and is used by the settlement system to determine outcomes of the contest. We always take the content of this database as definitive. According to this database, condia made changes to all his teams comfortably before the start of the games.
In order to be able to display all the pages on the site quickly to our volume of users we run several copies of this database – these are kept up to date from the main database using standard replication techniques. These generally reflect updates to the main database within a second, although they do occasionally run slightly later than that.
Live scoring is a more complex issue – calculating and displaying positions in tables on a large volume of data within a couple of seconds of getting an update can’t really be done if you need to get the data from a database every time. What we do for this is store all the table information in memory on another system (something between a copy of the database and a cache). This is fed continuously with updates from one of the copy databases – basically the live scoring system hits up the database every few seconds, and gets all changes with a timestamp later than the timestamp of the last change it processed, plus a 10 second grace period before that to catch anything that happened out of order.
Growth in the size of the database over the season reached a point over Christmas where the copy database that the live scoring system gets its data from is only just keeping up, and large changes like an export from condia can tip it over into being more than 20 seconds out of date for a minute or two. This appears to have led to condia’s updates not making it through to the live scoring system.
Since before the game the rosters that you can see as a user are delivered from the database, condia would have had no reason to be concerned – all his changes were showing correctly. Once the games started however the lineup he was seeing changed back to the earlier lineup. When the technical team became aware of the issue some time later, they restarted the live scoring system, which causes it to do a complete reload of data from the database. At this point, all of condia’s line-ups changed to reflect the changes which had been made hours before and not picked up.
We are making some changes today to stop this happening – firstly we are increasing the size of the database servers so they will be able to stay comfortably on top of the load, and secondly we are increasing the grace period, so if things do start to run slowly we will still pick up all changes.