Quantcast StorefrontBacktalk » Blog Archive » Dark For 36 Hours: Burlington’s Web Gambit
advertisement
advertisement

Dark For 36 Hours: Burlington’s Web Gambit

Written by Evan Schuman and Fred J. Aun
November 18th, 2009
Like this story? Share it
To share this story with people in your social network, please click on the network icons below.

Leaving online shoppers out in the cold with no warnings or explanations (or coats, if that’s what they wanted to buy), Burlington Coat Factory took its Web site offline all day Wednesday (Nov. 18)—plus about 12 hours split between Tuesday and Thursday—for a planned outage as the $3.5 billion clothing retailer performed an extensive hardware and database upgrade.

In what a senior company official conceded was an oversight, the 430-store, New Jersey-based chain failed to publish any ahead-of-time advisories before yanking its E-Commerce site’s plug in the wee hours. Nor did it post much in the way of an explanatory statement during the long downtime period that followed.

“The Burlington Coat Factory site is temporarily unavailable. Please check back later” was the extent of the message presented to visitors while the site was AWOL. Burlington Coat Factory Supervisor of Web Development Jack Follansbee acknowledged the blunder and said that the retailer should have told visitors when its site was expected to return.

“The messaging on the site could clearly have been better,” Follansbee said. “It was an omission. We should have done something (a status page) a little more customized.”

The outage had long been planned to start Tuesday night and last all day Wednesday and into the early morning hours of Thursday, Follansbee said. “Wednesday is actually the lowest revenue day for our Web site,” he said.

The need for the upgrade seems to have been demonstrated earlier in the week, something visitors to www.coat.com might have learned the hard way, as the site began experiencing sporadic, short-lived outages as early as Monday, according to site monitoring companies Dotcom-Monitor and Alert-Bot. “On Monday, it went down for five minutes with an HTTP 503 server-too-busy outage,” said AlertBot Sales and Marketing Manager Justin Noll, adding that the site was offline for brief periods up until shortly after 2:00 A.M. (New York time) Wednesday. At that point, the lights went out completely.

Follansbee said the site was actually taken down at 10:00 P.M. (New York time) Tuesday and was targeted to be back up at 2:00 A.M. (New York time) Thursday. (Editor’s Note: It seems as though Burlington Coat missed its target to be back up by 2 AM. According to site monitor services, Coat.com returned to the living at about 9:50 AM Thursday.)

Follansbee said he was not surprised the site was sputtering before the patient was euthanized. “We’ve been trying to migrate off obsolete hardware,” he said. Any intermittent outages were “related to getting off the infrastructure we are replacing. The deteriorating performance of the old hardware was one of the reasons for the upgrade.”

The effort was to perform a major upgrade to the chain’s Oracle Application Server in addition to “a complete hardware and software refresh of our E-Commerce platform,” including an operating system upgrade and “getting off some old disk arrays that we’ve been trying to retire,” Follansbee said. “Everything is being refreshed.”

Internal projections had the database upgrade alone taking 19 hours. “We needed a full 24-hour outage to do the upgrade, to do the tidying up,” he said, adding that some of these upgrades were attempted last year—under a different executive’s jurisdiction—and had “failed.” Why had it failed. “Nobody seems to remember,” Follansbee said.

The changes will not deliver any immediate functionality improvements but will enable such improvements shortly, including an improved capability to handle the accounting backend operations needed for selling stored-value cards online.

Although Burlington Coat Factory’s situation clearly doesn’t qualify as mere site maintenance, one of Alert-Bot’s Noll’s pet peeves is the way some major retailers regularly take their E-Commerce sites offline for nightly tweaking.

“These major brands should not have scheduled maintenance,” he said, noting there is technology available that allows companies to update sites without incurring extensive downtime. “You don’t need to take down your Web site to do updates, but you see these huge, Fortune 500 companies doing it. I don’t know if they’re skimping on IT spending, but I can’t see why they are updating every night for two or three hours. When we do consulting, we tell them the most amount of downtime we consider acceptable is 20 minutes per month for scheduled maintenance.”

That said, the Burlington changes included extensive upgrades to key hardware plus multiple pieces of strategic software. By completely taking down the site, the E-Commerce group was theoretically able to complete all of its major changes in barely more than one day. (We stress: “In theory.”) An attempt to perform these changes while maintaining site functionality would have almost certainly caused the upgrades to take far longer, an undesirable luxury right before the crucial holiday shopping season. Noll’s point has merit in general, but Burlington’s effort seems like a valid exception.


advertisement

Leave a Reply

Newsletter

Quickly catch-up on the latest in E-Commerce and Retail Tech with our free weekly newsletter, with urgent bulletins as news merits.
advertisement

Most Recent Comments

What’s The Rush For New PCI Call Center Requirements?

And I have not heard anyone mention the impact on companies who provide quality improvement services. Many merchants hire quality improvement companies to review their audio recordings to provide guidance on how to improve their sales staff’s effectiveness in customer service and sales retention. PCI Council needs to rethink this requirement until there is a widely available commercially viable solution. Read more...
Another ridiculous decision where regulators don't think critically enough about the unintended consequences of their decision. This will be a huge problem for the credit and collections industry. We have to keep all recorded calls for other reasons not related to cc information. We can't purge all of our calls and we don't have the technology to not record part of the conversation. Even if we did, I am not sure we could afford it. Read more...
This "clarification" is causing a lot of panic with large FS clients who now appear to be non-compliant after spending 7 figure sums on their compliance programs. The only alternative to call recording would now appear to be some sort of IVR/push button type interrupt to take card data away from the contact centre. The council is a position to force that sort of process and technology change and this may backfire on them and the vendors that lobbied hard for this clarification. Read more...
PCI council has made a one-sided decision; They should have done a much more in-depth research that could have provided more insight on what regards to the implications of such decision. Read more...

Will Old OS Cause PCI Violation? No, But Marketing Still Says So

This is an interesting issue, because there's more to it than what's apparent on the surface. PA-DSS requires supported and patched operating systems and other software components (e.g., databases, libraries, Java, etc.) per PA-DSS 7.1.b and 8.1, and the option for compensating controls simply isn't there. Merchants can make use of compensating controls for most PCI DSS requirements, but only when legitimate constraints exist and only in ways that meet the intent and rigor of the requirement and go above and beyond the other PCI DSS requirements. Read more...
Why would one automatically upgrade to a "new" OS -- some of the older versions of certain OS-es are more stable and more robust than the crap being peddled today. This is yet another clear example of PCI SSC being out of touch with reality. Rather than requiring a "current" OS, the requirement should be to demonstrate the OS in use is stable and robust, and is adequately hardened against threats. Read more...
There are compensating controls that encrypt the swipe at the driver level as it enter the PC, there are hardware encrypting card swipes so the cardholder data is already encrypted before it comes to the PC -- either of these, especially the second, would remove the OS entirely from a cardholder data risk profile. Read more...
In my opinion, the only thing the vendor did wrong was they didn’t know of that FAQ entry. Even if they did, it changes nothing about the need for merchants to update software that no longer receives updates. Read more...

MasterCard Blinks, Drops Dec. 31 Level 2 PCI Deadline

Reciprocity between MasterCard and Visa was always been a factor in Acquirer merchant level assignments. The brief removal of reciprocity generated a great deal of interest in being able to be classified at a lower level in MasterCard's world. Nevertheless the return of the reciprocity language in the December changes did not effectively create any new Level 2 merchants, but it DID dash the hopes of a lot of them.... :-( Read more...
Let's given them credit??? For being idiotic in the first place? Not on your life! Everyone has just had to scramble and include the costs of the previously announced M/C requirement in their 2010 budgets, and start negotiating with the QSAs for the additional services. All for naught! Read more...
"A bunch of Level 3 and Level 4 merchants just became Level 2s". Is this an accurate statement? MasterCard & Visa have historically included the caveat "or is a Level X in another brand" in their level setting criteria. MasterCard appeared to back way from this in the June pronouncement, and have simply returned to the status quo. Have Acquirers have been tracking and reporting merchants at separate levels by brand? Read more...
I stick by my comment (quoted in the column) about a bunch of L3 and L4 merchants becoming L2s and requiring an onsite. To me, what made MasterCard's original requirement for an onsite assessment for L2s palatable was that they took away their reciprocity provision. That is, they seemed to focus on larger merchants with over a million MasterCard trans/year. With reciprocity in place, a lot of smaller merchants are pulled into the onsite requirement. Rather than causing confusion, I think reciprocity will lead to additional work for processors and acquirers. Read more...

Retailers Sue POS Vendor, Questions Raised Where PCI Duties Stop

I would add a couple more questions: "did the breach involve the use of the default passwords?" (The story doesn't say.) And "were the default passwords used by Computer World to remotely administer the store systems?" "where is the PCI auditor in all this?" Did the restaurant group think they didn't need an audit because Radiant was (mis)representing Aloha as PCI compliant? How is a retailer or even a PCI auditor to know otherwise? A PCI auditor is not necessarily a qualified computer forensic investigator capable of finding the card data on the hard drives. They can only base a decision on information given to them by others. Read more...
There are so many holes in the process it will be difficult to pin blame on just one constituent. It is ridiculous that the technology exists to better secure these transactions (PIN, EMV, etc) yet banks won't use them. Only the banks or government can force this change, and retailers will suffer until then. Read more...
A major issue in this case will be if the restaurants had any support agreements in place with Computer World and if so what those agreements say. In my experience many single unit/small operators choose to skip the support agreements in favor of a "pay as you go" arrangement. In this scenario I can't imagine how the POS VAR can be held responsible for a system they don't own nor exclusively manage. Read more...
There is a big difference in having the POS installation guide say "make sure you set this password because the security of your CHD depends on this" vs. a POS application not storing the CHD in the first place. Traditionally only the merchant was liable for breaches and PCI related fees (fines). Maybe dragging some of the vendors into the liability mud fight will open the eyes of some of these vendors. Read more...

Should Credit Card Transactions Be Free? There May Be A Way

Here in the Netherlands, where the population is notoriously penny-pinching, credit card acceptance is amazingly low. It's both a result of the consumer not wanting to pay interest on everyday purchases as well as merchants not giving up a slice of the action. It is both legal and common to pass the processing fee onto the customer as a surcharge. Now things are moving to leave the credit cards behind: mobile phone payments are becoming more and more common here, and the transaction fees are minimal. Parking and entertainment (movie/concert tickets, nightclubs) have been amongst the first, and it's rapidly gaining momentum because the market has been hungry for the convenience at a price it is willing to pay. Read more...
"Free" is an illusion. Don't charge one person but charge double to someone else. I am very skeptical on anyone who says that advertising will create valid cashflow. Just look at the advertising struggles in a TiVo world. And if you sell your customers data, just be warned that the one group that might have issue with that are you customers (which to me is very important to cashflow. Read more...
Another factor not mentioned here is the impending costs that the processors and issuers are going to incur when someone decides on an end-to-end encryption method, and it then becomes government mandated. I can guarantee that this is a when question and not an if question. The back-end networks are pretty antiquated right now, and it's going to cost billions to replace everything. The cost of tech may be going down, but the cost of replacing millions of servers and hardware, and creating new, proprietary, software is still really expensive. Read more...
Accepting credit cards are not "risk-free" for merchants, contrary to Jim's comments above. Chargebacks are an expense - both in terms of actual transaction reversals and costs associated with managing the process. Chargeback rules and expenses can be everything from a thorny issue to an onerous expense for some merchants, especially for convenience stores that allow customers to pay for gasoline at the pump, or other retailers that allow in-store self-checkout options. Read more...
I've wondered for years why the price of transactions has been so high. Phone companies long ago started offering unlimited calling for flat rates because they understood that in many cases it cost more to report on the transactions (calls) than it did to fulfill them. Read more...
If a home-owner defaults on the mortgage, who is taking the risk? The bank making the loan to the consumer or the person selling the house? It is obviously the bank that takes this risk and is rewarded for that risk through interest rate charges. In my mind, we have mixed together two distinct and unrelated transactions. Read more...
The one big factor not mentioned in this article is who will take over the risk ? Taking credit cards is risk free to merchants and the issuing Banks take the risk if a customer defaults on the payments ! If you had a "interchange free" payment system will the merchants assume the risk ? Also, if there isn't enough profit for the issuing banks they will stop issuing credit cards which will in turn kill our economy. Read more...

The Dangerous Out-Of-Scope PCI Charade

If tokens are ever deemed in-scope, then where does the line stop? I ask this because it would mean that all timestamps, sequential number, random numbers or any other piece of information that may or may not be used to generate a token is within scope -- all data a POS uses and stores, not just payment data. Read more...
Having the ability to do both Tokenization and End to End Encryption (not mere point to point) can have tremendous scope and risk reduction benefits and agility to adapt to change in this fast moving compliance landscape. Being able to have both on tap from a single platform is a solid approach to avoiding the pitfalls. Read more...
But the consumer walks into a particular retail chain, gives their payment card to someone wearing that chain's uniform and the card is swiped. If, six months later, there's a breach and that card was misused, it's the retailer who will in the spotlight. They're the deep pocket and, therefore, the target. If the consumer is angry and wants to cut off business, it will hit the retailer. Therefore, if the retailer is going to end up being blamed no matter what, they have to stay involved. Read more...
True, that someone may be storing a token-to-PAN cross reference. But that would be the bank, not the retailer. If the bank is not sure they can keep their data secure, then there are bigger problems to be addressed than bringing tokens into scope. Read more...
Good general point, Steve, but for the record, not all tokenization is done the same way. Many tokens are associated with lookup lists that allow for them to re-matched to the card data if it's needed, such as for a chargeback. A token doesn't have to be decryptable (is that a word?) for there to be a way to access the original data. Read more...
The out-of-scope argument is very valid but in reference to tokens, the premise of temporarily out-of-scope or abruptly deemed in-scope is flawed. Conway was quoted “anything that could be made unreadable can, in various ways, be made readable again,” this statement is true when talking about encryption technologies (all encryption technologies) but not so with true tokens. True tokens are in no way related to the original data other than as a reference key. Read more...