Quantcast StorefrontBacktalk » Blog Archive » Some Retail Sites Struggled With Uptime Right Through New Year’s
advertisement
advertisement

Some Retail Sites Struggled With Uptime Right Through New Year’s

Written by Fred J. Aun
January 8th, 2009
Like this story? Share it
To share this story with people in your social network, please click on the network icons below.

Retailer Web site problems didn’t end with the passing of Black Friday and Cyber Monday. Some of the biggest names in retail, including Wal-Mart, Target, Lowe’s, Gap and Blockbuster, found themselves with cranky Web sites in the days leading up to, and beyond, Christmas.

Most of the site problems were minor in scope, but a few were long and probably costly. When it came to duration, none appeared to match the outage that hit the Publix Super Markets site on January 2.

Publix.com crashed shortly after 2 PM New York time on that Friday and didn’t come back until almost 7 PM, according to site monitoring company Sitemorse. Even after the Publix site resurfaced, its troubles weren’t over: Sitemorse said the site went offline for another half-hour at 7:15 PM.

Asked to provide some insight into the cause of the problem, a Publix spokeswoman said she couldn’t help. “I am unaware of any outages that would have resulted in disruption of service to our customers,” said Publix Director of Media and Community Relations Maria Brous. She noted that even if she was aware of site problems, Publix “would not comment on IT incidences.”

Let’s imagine there were people who wanted to order, say, aspirin on that Friday-after-New Year’s afternoon. Having no luck getting on the Publix site, perhaps they surfed over to CVS.com. If they did so between 2:20 PM and 4:14 PM New York time, their headaches might have throbbed a little harder because the CVS site was also dead in the water during that period, Sitemorse reported.

Speaking of headaches, it’s likely that the folks in charge of the Blockbuster site know a thing or two about migraines. Their site went offline several times on December 20, according to site monitoring company Pingdom.

Although the Blockbuster problems were short-lived, about 10 minutes in duration each, they weren’t the site’s only issues. Pingdom said Blockbuster.com went down again, for about six minutes, on December 29, and yet again, for about 12 minutes, on January 2.

Blockbuster Blocked
Attempts to visit Blockbuster.com yielded either HTTP 503 (service unavailable) messages or an apology: “We’re sorry, Blockbuster.com is temporarily unavailable while we make some changes to the site. We’re working hard to bring you an even better Blockbuster Online experience. We apologize for any inconvenience this may cause. Check back soon!”

Maybe the IT crew at Blockbuster can take comfort in knowing it wasn’t alone in having a crummy day December 20. Their colleagues over at Foot Locker found themselves with a site that crashed at least three times that Saturday morning, according to Pingdom.

“We got a few different error messages from them,” Pingdom Analyst Peter Alguacil said. “A few HTTP error 504 (gateway timeout), a few HTTP error 503 (service unavailable). But most of the time the Web site was just extremely slow in responding.”

December 20 also brought a trio of brief outages at the Foot Locker site. Pingdom said Footlocker.com went down three times shortly after midnight that Saturday. “As far as I can see, the issue with Foot Locker doesn’t look to be traffic related, but rather some kind of backend problem,” Alguacil said. “It might well have been caused by some work being done, but it’s hard to tell. They might have been doing something that affected the way their site works with Akamai, since the 503 and 504 errors we got were from Akamai servers.”

He acknowledged that trying to determine the causes of site problems is often “extremely difficult.” Alguacil always takes into account the times when incidents take place.

Weekend Work
“Considering the timeframe of the Foot Locker problem, it is quite likely that they were doing some form of work on the site and ran into trouble,” he said, noting that the overnight period between Saturday and Sunday “is a pretty common choice for that, since less commerce than usual is expected to be going on, in case something goes wrong.”

Target.com became unavailable for about an hour on December 31, Sitemorse said. That was the same day the Lowe’s site had some significant issues. Sitemorse reported Lowes.com to be unavailable at 3:08 PM New York time and said the site didn’t recover until 4:36 PM.

The Gap’s site slipped up on the day after Christmas, Pingdom said. It remained offline (to at least some shoppers) for about a half-hour, about the same duration as an outage, spotted by Sitemorse, that hit Walmart.com on December 30.

Other outages of note: CDW’s site went out for about 48 minutes on December 19. Kroger’s collapsed temporarily on Christmas Eve and again on January 4.

Given the results of a nationwide survey from E-commerce development company Guidance, none of this will come as a surprise to many online shoppers. Guidance found that 36 percent of online shoppers “ran into roadblocks” while online during the holidays, “ranging from molasses-like Web site response to fruitless efforts to check out to outright system crashes.”

The good news is that about 64 percent of those surveyed said they encountered no problems doing their online holiday shopping. But Guidance found that 37 percent of the respondents said they “skipped Internet shopping altogether” for the 2008 holidays and that “a small percentage” of the people in that category said they didn’t bother shopping online due to frustration from problems they encountered during prior attempts.

According to Guidance, 13 percent of the people who had online shopping problems this year said they gave up because the site they were shopping was too slow. Eight percent said a site froze or crashed, 7 percent couldn’t complete a purchase on their first attempt, 6 percent encountered a site that was temporarily down and 4 percent said “a purchase they thought they had completed actually didn’t go through.”

No Excuse For Hiccups?

Guidance owner and CTO John Provisor said there’s really no excuse for big-time retailer Web site outages, even during peak holiday periods. “It seemed that, as a whole, the performance of many sites was poor given that retailers should have had plenty of time to estimate and prepare for the traffic,” he said. “Maybe they were surprised by the volume of the traffic, but it’s really a message to the retailers to do a lot better preparation. They should prepare months in advance with advanced stress testing that simulates that kind of traffic.”

Matt Poepsel, vice president of performance strategies at site performance company Gomez, disagreed with Provisor’s unflattering assessment of site performance during the holidays. “The nature of how the retailers responded this holiday season was pretty good overall,” he asserted. “Certainly, the month of December seemed to be business as usual. That hasn’t always been the case.”

Retailers, having gone through a “gradual learning curve” over the years, are generally adept at maintaining site uptime, Poepsel said. “We’ve seen availability numbers hold up very well,” he said. Although Poepsel acknowledged there were some high-profile site flameouts, he said retailers in general made big investments in infrastructure and that they are benefitting from more durable hardware and applications.

Mid-December retail site problems can often be attributed to a “chronic buildup” of issues combined with “the stress of trying to maintain high levels of volume over a long period of time,” Poepsel said. Provisor agreed, noting that retailer databases, after dealing with a blitz of orders after Thanksgiving, tend to get clogged with data, logs, backups and algorithms.

“It really magnifies the demand on the infrastructure,” Provisor said. “The solution is proper cleansing and pruning of your legacy and archive systems. I have seen systems that have data logs that take up all the capacity of a system. If they don’t have proper pruning and purging, they’re going to see a slowdown.”


advertisement

Leave a Reply

Newsletter

Quickly catch-up on the latest in E-Commerce and Retail Tech with our free weekly newsletter, with urgent bulletins as news merits.
advertisement

Most Recent Comments

What’s The Rush For New PCI Call Center Requirements?

And I have not heard anyone mention the impact on companies who provide quality improvement services. Many merchants hire quality improvement companies to review their audio recordings to provide guidance on how to improve their sales staff’s effectiveness in customer service and sales retention. PCI Council needs to rethink this requirement until there is a widely available commercially viable solution. Read more...
Another ridiculous decision where regulators don't think critically enough about the unintended consequences of their decision. This will be a huge problem for the credit and collections industry. We have to keep all recorded calls for other reasons not related to cc information. We can't purge all of our calls and we don't have the technology to not record part of the conversation. Even if we did, I am not sure we could afford it. Read more...
This "clarification" is causing a lot of panic with large FS clients who now appear to be non-compliant after spending 7 figure sums on their compliance programs. The only alternative to call recording would now appear to be some sort of IVR/push button type interrupt to take card data away from the contact centre. The council is a position to force that sort of process and technology change and this may backfire on them and the vendors that lobbied hard for this clarification. Read more...
PCI council has made a one-sided decision; They should have done a much more in-depth research that could have provided more insight on what regards to the implications of such decision. Read more...

Will Old OS Cause PCI Violation? No, But Marketing Still Says So

This is an interesting issue, because there's more to it than what's apparent on the surface. PA-DSS requires supported and patched operating systems and other software components (e.g., databases, libraries, Java, etc.) per PA-DSS 7.1.b and 8.1, and the option for compensating controls simply isn't there. Merchants can make use of compensating controls for most PCI DSS requirements, but only when legitimate constraints exist and only in ways that meet the intent and rigor of the requirement and go above and beyond the other PCI DSS requirements. Read more...
Why would one automatically upgrade to a "new" OS -- some of the older versions of certain OS-es are more stable and more robust than the crap being peddled today. This is yet another clear example of PCI SSC being out of touch with reality. Rather than requiring a "current" OS, the requirement should be to demonstrate the OS in use is stable and robust, and is adequately hardened against threats. Read more...
There are compensating controls that encrypt the swipe at the driver level as it enter the PC, there are hardware encrypting card swipes so the cardholder data is already encrypted before it comes to the PC -- either of these, especially the second, would remove the OS entirely from a cardholder data risk profile. Read more...
In my opinion, the only thing the vendor did wrong was they didn’t know of that FAQ entry. Even if they did, it changes nothing about the need for merchants to update software that no longer receives updates. Read more...

MasterCard Blinks, Drops Dec. 31 Level 2 PCI Deadline

Reciprocity between MasterCard and Visa was always been a factor in Acquirer merchant level assignments. The brief removal of reciprocity generated a great deal of interest in being able to be classified at a lower level in MasterCard's world. Nevertheless the return of the reciprocity language in the December changes did not effectively create any new Level 2 merchants, but it DID dash the hopes of a lot of them.... :-( Read more...
Let's given them credit??? For being idiotic in the first place? Not on your life! Everyone has just had to scramble and include the costs of the previously announced M/C requirement in their 2010 budgets, and start negotiating with the QSAs for the additional services. All for naught! Read more...
"A bunch of Level 3 and Level 4 merchants just became Level 2s". Is this an accurate statement? MasterCard & Visa have historically included the caveat "or is a Level X in another brand" in their level setting criteria. MasterCard appeared to back way from this in the June pronouncement, and have simply returned to the status quo. Have Acquirers have been tracking and reporting merchants at separate levels by brand? Read more...
I stick by my comment (quoted in the column) about a bunch of L3 and L4 merchants becoming L2s and requiring an onsite. To me, what made MasterCard's original requirement for an onsite assessment for L2s palatable was that they took away their reciprocity provision. That is, they seemed to focus on larger merchants with over a million MasterCard trans/year. With reciprocity in place, a lot of smaller merchants are pulled into the onsite requirement. Rather than causing confusion, I think reciprocity will lead to additional work for processors and acquirers. Read more...

Retailers Sue POS Vendor, Questions Raised Where PCI Duties Stop

I would add a couple more questions: "did the breach involve the use of the default passwords?" (The story doesn't say.) And "were the default passwords used by Computer World to remotely administer the store systems?" "where is the PCI auditor in all this?" Did the restaurant group think they didn't need an audit because Radiant was (mis)representing Aloha as PCI compliant? How is a retailer or even a PCI auditor to know otherwise? A PCI auditor is not necessarily a qualified computer forensic investigator capable of finding the card data on the hard drives. They can only base a decision on information given to them by others. Read more...
There are so many holes in the process it will be difficult to pin blame on just one constituent. It is ridiculous that the technology exists to better secure these transactions (PIN, EMV, etc) yet banks won't use them. Only the banks or government can force this change, and retailers will suffer until then. Read more...
A major issue in this case will be if the restaurants had any support agreements in place with Computer World and if so what those agreements say. In my experience many single unit/small operators choose to skip the support agreements in favor of a "pay as you go" arrangement. In this scenario I can't imagine how the POS VAR can be held responsible for a system they don't own nor exclusively manage. Read more...
There is a big difference in having the POS installation guide say "make sure you set this password because the security of your CHD depends on this" vs. a POS application not storing the CHD in the first place. Traditionally only the merchant was liable for breaches and PCI related fees (fines). Maybe dragging some of the vendors into the liability mud fight will open the eyes of some of these vendors. Read more...

Should Credit Card Transactions Be Free? There May Be A Way

Here in the Netherlands, where the population is notoriously penny-pinching, credit card acceptance is amazingly low. It's both a result of the consumer not wanting to pay interest on everyday purchases as well as merchants not giving up a slice of the action. It is both legal and common to pass the processing fee onto the customer as a surcharge. Now things are moving to leave the credit cards behind: mobile phone payments are becoming more and more common here, and the transaction fees are minimal. Parking and entertainment (movie/concert tickets, nightclubs) have been amongst the first, and it's rapidly gaining momentum because the market has been hungry for the convenience at a price it is willing to pay. Read more...
"Free" is an illusion. Don't charge one person but charge double to someone else. I am very skeptical on anyone who says that advertising will create valid cashflow. Just look at the advertising struggles in a TiVo world. And if you sell your customers data, just be warned that the one group that might have issue with that are you customers (which to me is very important to cashflow. Read more...
Another factor not mentioned here is the impending costs that the processors and issuers are going to incur when someone decides on an end-to-end encryption method, and it then becomes government mandated. I can guarantee that this is a when question and not an if question. The back-end networks are pretty antiquated right now, and it's going to cost billions to replace everything. The cost of tech may be going down, but the cost of replacing millions of servers and hardware, and creating new, proprietary, software is still really expensive. Read more...
Accepting credit cards are not "risk-free" for merchants, contrary to Jim's comments above. Chargebacks are an expense - both in terms of actual transaction reversals and costs associated with managing the process. Chargeback rules and expenses can be everything from a thorny issue to an onerous expense for some merchants, especially for convenience stores that allow customers to pay for gasoline at the pump, or other retailers that allow in-store self-checkout options. Read more...
I've wondered for years why the price of transactions has been so high. Phone companies long ago started offering unlimited calling for flat rates because they understood that in many cases it cost more to report on the transactions (calls) than it did to fulfill them. Read more...
If a home-owner defaults on the mortgage, who is taking the risk? The bank making the loan to the consumer or the person selling the house? It is obviously the bank that takes this risk and is rewarded for that risk through interest rate charges. In my mind, we have mixed together two distinct and unrelated transactions. Read more...
The one big factor not mentioned in this article is who will take over the risk ? Taking credit cards is risk free to merchants and the issuing Banks take the risk if a customer defaults on the payments ! If you had a "interchange free" payment system will the merchants assume the risk ? Also, if there isn't enough profit for the issuing banks they will stop issuing credit cards which will in turn kill our economy. Read more...

The Dangerous Out-Of-Scope PCI Charade

If tokens are ever deemed in-scope, then where does the line stop? I ask this because it would mean that all timestamps, sequential number, random numbers or any other piece of information that may or may not be used to generate a token is within scope -- all data a POS uses and stores, not just payment data. Read more...
Having the ability to do both Tokenization and End to End Encryption (not mere point to point) can have tremendous scope and risk reduction benefits and agility to adapt to change in this fast moving compliance landscape. Being able to have both on tap from a single platform is a solid approach to avoiding the pitfalls. Read more...
But the consumer walks into a particular retail chain, gives their payment card to someone wearing that chain's uniform and the card is swiped. If, six months later, there's a breach and that card was misused, it's the retailer who will in the spotlight. They're the deep pocket and, therefore, the target. If the consumer is angry and wants to cut off business, it will hit the retailer. Therefore, if the retailer is going to end up being blamed no matter what, they have to stay involved. Read more...
True, that someone may be storing a token-to-PAN cross reference. But that would be the bank, not the retailer. If the bank is not sure they can keep their data secure, then there are bigger problems to be addressed than bringing tokens into scope. Read more...
Good general point, Steve, but for the record, not all tokenization is done the same way. Many tokens are associated with lookup lists that allow for them to re-matched to the card data if it's needed, such as for a chargeback. A token doesn't have to be decryptable (is that a word?) for there to be a way to access the original data. Read more...
The out-of-scope argument is very valid but in reference to tokens, the premise of temporarily out-of-scope or abruptly deemed in-scope is flawed. Conway was quoted “anything that could be made unreadable can, in various ways, be made readable again,” this statement is true when talking about encryption technologies (all encryption technologies) but not so with true tokens. True tokens are in no way related to the original data other than as a reference key. Read more...