Posto su evropski serveri doživeli ovih dana dosta problema (velikih lagova, i stalnog pada velikih server), za javnost World of Warcraft-a izašao je sam predsednik kompanije Blizzard-a Mike Morhaime. On je u svom pismu koji je ostavio za javnost, izašao igra�?ima u susret prikazavši nov na�?in kako bi se problem rešio kao i kompezacija za igubljeno vreme zbog problema koji su bili na serverima. Nadamo se da će biti jos ovakvih prilika da sam predsednik jedne velike kompanije izadje u slu�?ajiu novonastalih problema. Podsetimo samo da je pre nekoliko dana bila peticija u vezi WoW Europe servera protiv Blizzard kompanije. Pismo koje je Mike Morhaime poslao za javnost možete pro�?itati u daljem tekstu. Message from the President of Blizzard
The last couple of weeks have seen extended periods of unscheduled downtime and high levels of latency on our World of Warcraft servers in Europe. Several extraordinary, unrelated and external events combined to make this a particularly unacceptable and intolerable period and I would like to extend my apologies for the disruptions this has caused. I hope that by explaining what happened, and the steps that we took to find a resolution, I can reassure you of our ongoing commitment to providing the best service possible.
We experienced two particular trouble spots, one starting on August 31st and then again on September 12th.
On August 31st players were disconnected from the servers at one of our data centers due to a failure in our network and its redundant systems that connect the platform to the Internet. We immediately contacted our ISP partner who operates the data centers and resolved the situation through a controlled shutdown and restart of each server. This break in operations unfortunately touched all players in Europe as the login / authentication process was affected.
The next day during peak hours players connecting to the same data center experienced extreme network latency spikes. Our investigations discovered that a line failure between our network and our ISP had caused a 50% reduction in bandwidth. We contacted our ISP to move traffic over to a redundant network, allowing players to continue playing. However, the latency remained until the equipment was fully restored on the September 5th.
A fairly trouble-free week followed.
On September 12th we detected a dramatic increase in temperature at different data centers. We then brought down the servers to avoid any damage to the hardware. Our ISP quickly identified and rectified a failure in the air-conditioning system; however, before re-opening the servers, we brought our hardware manufacturer on-site to check for any damage.
Also on the same day an apparent ongoing peering/routing issue between several Internet providers from across Europe hit the region in France where some of our servers are located. This resulted in latency spikes, players unable to complete authentication, and being stuck at the ‘connected’ screen. Due to the complexity of the problem we were not able to gain an immediate solution. We have now confirmed with each of the Internet providers that the traffic that was previously routed over the troubled areas is now routed through a different path in Europe. It appears that this has resolved the problem, as the latency spikes and authentication problems have not occurred since late evening on September 14th.
While the issues of the last two weeks have now been addressed and had stemmed, on multiple occasions, from unrelated equipment and connectivity issues, this is no consolation for the disruptions that you have had to endure. We remain committed to providing the best playing conditions possible, and I want to apologize on behalf of everyone at Blizzard for being unable to do so during this period. Therefore, in addition to the 2 days compensation and rest time already credited, I would like to extend this by an additional 3 day credit to all player accounts in Europe. These will be credited during next week.
I also want to reassure you that our European and US teams remain committed to pulling together and working around the clock to solve any such problems when they occur. The constant flow of information between all departments on either side of the ocean ensures that problems are identified and solutions found as quickly as possible.
In parallel, we are constantly working on new features to enhance the game. Most recently we deployed new code with patch 1.7 to better balance the load and thus reduce overall latency on servers running instances. This has resulted in better performance of the new Arathi Basin battleground and in popular dungeons such as Molten Core and Zul’Gurub.
We will continue to make such improvements and focus on maintaining the stability of the platforms while endeavoring to ensure that you are kept up to date on any issues that might occur.
Mike Morhaime
President, Blizzard Entertainment
The last couple of weeks have seen extended periods of unscheduled downtime and high levels of latency on our World of Warcraft servers in Europe. Several extraordinary, unrelated and external events combined to make this a particularly unacceptable and intolerable period and I would like to extend my apologies for the disruptions this has caused. I hope that by explaining what happened, and the steps that we took to find a resolution, I can reassure you of our ongoing commitment to providing the best service possible.
We experienced two particular trouble spots, one starting on August 31st and then again on September 12th.
On August 31st players were disconnected from the servers at one of our data centers due to a failure in our network and its redundant systems that connect the platform to the Internet. We immediately contacted our ISP partner who operates the data centers and resolved the situation through a controlled shutdown and restart of each server. This break in operations unfortunately touched all players in Europe as the login / authentication process was affected.
The next day during peak hours players connecting to the same data center experienced extreme network latency spikes. Our investigations discovered that a line failure between our network and our ISP had caused a 50% reduction in bandwidth. We contacted our ISP to move traffic over to a redundant network, allowing players to continue playing. However, the latency remained until the equipment was fully restored on the September 5th.
A fairly trouble-free week followed.
On September 12th we detected a dramatic increase in temperature at different data centers. We then brought down the servers to avoid any damage to the hardware. Our ISP quickly identified and rectified a failure in the air-conditioning system; however, before re-opening the servers, we brought our hardware manufacturer on-site to check for any damage.
Also on the same day an apparent ongoing peering/routing issue between several Internet providers from across Europe hit the region in France where some of our servers are located. This resulted in latency spikes, players unable to complete authentication, and being stuck at the ‘connected’ screen. Due to the complexity of the problem we were not able to gain an immediate solution. We have now confirmed with each of the Internet providers that the traffic that was previously routed over the troubled areas is now routed through a different path in Europe. It appears that this has resolved the problem, as the latency spikes and authentication problems have not occurred since late evening on September 14th.
While the issues of the last two weeks have now been addressed and had stemmed, on multiple occasions, from unrelated equipment and connectivity issues, this is no consolation for the disruptions that you have had to endure. We remain committed to providing the best playing conditions possible, and I want to apologize on behalf of everyone at Blizzard for being unable to do so during this period. Therefore, in addition to the 2 days compensation and rest time already credited, I would like to extend this by an additional 3 day credit to all player accounts in Europe. These will be credited during next week.
I also want to reassure you that our European and US teams remain committed to pulling together and working around the clock to solve any such problems when they occur. The constant flow of information between all departments on either side of the ocean ensures that problems are identified and solutions found as quickly as possible.
In parallel, we are constantly working on new features to enhance the game. Most recently we deployed new code with patch 1.7 to better balance the load and thus reduce overall latency on servers running instances. This has resulted in better performance of the new Arathi Basin battleground and in popular dungeons such as Molten Core and Zul’Gurub.
We will continue to make such improvements and focus on maintaining the stability of the platforms while endeavoring to ensure that you are kept up to date on any issues that might occur.
Mike Morhaime
President, Blizzard Entertainment
