Category Archives: Security

How much does it cost to build and run a PCI compliant business?

I’ve written a few posts about PCI security.  I get quite a few questions about the cost of building and managing a PCI environment.  In a nutshell, it is expensive.  Below, I’ll outline the expenses that I am familiar with as it relates to my experience, but you should note that there are more expensive versions because there are plenty of places that you might be required to spend significantly more to satisfy the folks that you are doing business with and exactly what you are doing with credit card information.

ExpensiveI suppose the first thing worth pointing out is that are initial or one time costs and then ongoing costs.  Just to state a few things that are probably obvious, the initial costs are all the things you must do or buy to get from where ever you are today with the security program that you have in place to the point that an auditor is signing your PCI AOC (Attestation of Certification).  I could argue that in some cases, it might be less expensive to start from scratch then to transition from an existing infrastructure.  I’ve worked at many startups and I make significant investments in security, workflow and documentation that I suspect are unusual for startups and therefore I believe my starting point was significantly further along than your average scrappy startup.  In any case, here is a description of our costs:

  1. We spent about 6 Full-Time-Equivalent (FTE) months modifying our existing 65 page Word document of Information Security Policies and Standards while porting it to a wiki so that it was more consistent with our workflows and documentation tools, not to mention that it enable us to simplify PCI requirements such as audit-able document/change control and integration with other important tools such as ticketing systems, logging systems, etc.
  2. We spent about 6 FTE months enhancing and modifying our existing HR, Engineering and System Operations workflows to meet requirements.  This included additional documentation to support the Information Security Policies and Standards.  Examples of these modifications include increases to insurance types and amounts, employee background checks, security training for 100% of the employees, fully-documented engineering workflows, review/approval workflows for modifying any 3rd party software components to our platform, monitoring security alerts, secure access key rotations, business continuity planning, etc..
  3. We spent about 2 FTE months building a security training program for both every employee as well as more elaborate training for our technical team including online training pertaining to secure programming and QA practices.
  4. We spent about 3 FTE months modifying and enhancing our technical infrastructure and automation tooling to accomodate PCI requires such as vulnerability scanning, code reviews, etc…
  5. We spent about 6 months revisiting or deploying new technology throughout our cloud based platform.  This includes deploying egress firewalls, redistributing application components to ensure that each server had a single role, deploying an isolated/secure software repository within our PCI environment, OS hardening (removing all components, accounts that are not actually used by our platform), deploy various intrusion detection, logging, monitoring and alerting solutions.
  6. We spent about 4 FTE months scanning for vulnerabilities and external penetration testing which revealed mostly false positives but a few issues that then cost us a full engineering cycle (12 folks) working for 4 weeks to clean up.
  7. We spent about 4 FTE months building out a real business continuity plan and actually testing every type of failure and recovery that we could identify.
  8. Then we invited the auditor in for the audit.  The initial audit took was about 2 FTE month of work spread out over 4 calendar months for two reasons.  First time audit meant that we needed to establish a shared understanding of terminology, environment, compliance issued, etc… Plus, even after all the above work with the help of a security consulting firm, the auditor discovered issues that they were not comfortable with so we spent time revisiting documentation, workflows and technology tools.
  9. We increased our cloud fees by 25%, primarily because we needed to buy additional instances to separate server roles – you can’t have a single command and control server for all your utility, admin stuff.
  10. Finally, we spent $250K on the following services and products – security consulting expertise, PCI auditor, security scanning services, computer based training, code review software and intrusion detection/file monitoring software and anti-virus software.

That was we spent to get our first certification.  In summary, it was approximately 36 FTE months of work, let’s say at a fully loaded cost of $150K per year = $450K total labor.  It was $250K/year of software and service cost and another $50K/year of increased cloud fees.  In total, approximately $750K of cost to get us to the starting line in year one.

In addition, I estimate that it takes us 1 FTE to run our security program – support audits, run/manage vulnerability scans, manage customer/partner security interface, manage training program for all employees, manage a security management workflow, monitor alerts and address issues, etc…  PCI also easily adds at least a 25% drag on our entire technology organization – secure code reviews, vulnerability scanning, training, change control, secure builds and transfers, etc…  So for us, this is approximately $450K of ongoing labor cost and the same $300K/year of software, services and cloud fees.

Basically, $750K to get PCI compliant and $750K/year in ongoing cost.  Pretty expensive for a startup.

Writing a PCI Compliant Information Policy and Standard

I mentioned in my Securing a Cloud Based Platform post that we embarked upon an effort to build a PCI compliant security program; we really needed to improve upon several areas:

  1. We needed to improve our documentation
  2. We needed to modify existing or add new technologies
  3. We needed to improve existing or add new workflow

As we learned, most of the PCI standard is about documenting what you do and doing what you document.  Sure, there are plenty of standards stating the minimum password length, key rotation standards and data retention policies – I’ll get to some of the bits and bytes soon enough.  But let’s start with documentation, which is where I started.  What I didn’t initially understand was that there is a juicy, riveting 113 page Reporting on Compliance (“ROC”) Reporting Instructions for PCI DSS v2.0.  This is basically the auditor’s instruction book which completely describes exactly how an auditor will gather evidence, ask questions, etc…  This is the auditor’s playbook and there are 12 requirements.

So, why do I mention that the auditor’s instructions are available, because if you need to write an Information Policy and Standard document then you should organize it to correspond exactly to the PCI ROC’s organization.  Why?  Because if you don’t then you get to have the fun that I enjoyed for several months which includes providing the auditor with a map of where my policy documentation states the required language.  My policy documentation is well over 100 pages, so the exercise of mapping my 100 pages to the 113 pages of auditor instructions was a truly ridiculous waste of time.  If I had to do it over again, the first 12 sections of my policy would map directly to the 12 PCI requirements and I would store extra beyond these basic policies.  In fact, I think that it is silly that the PCI council doesn’t produce a compliant security policy document (at least not one that I could find anyway) that organizations can simply adopt, or at least use as a starting document.

Securing a Cloud Based Platform

When I’m building business platforms, security is a considerable issue.  Security is definitely one of the technology issues that can get extremely expensive, timing consuming and complicated.  Like so many business decisions, security solution are investments that need to be within the maturity context of the information that you are securing and the resources that you have available.  Two years ago, I started a new business, Linkable Networks, and our technology platform, MyLinkables, integrates into several points along the financial transaction rails as we are providing users with the ability to link purchase discounts to the credit cards that they already have in the wallet today.  Our initial integration strategy was to avoid collecting any credit card information, instead we used a fairly common technique called “tokenization”; simply put, our partner collects authentication details from the users and issues randomly generated, unique identifiers to us that we use to anonymously exchange details without ever collecting, transmitting or storing actual credit card information on our infrastructure.  It worked surprisingly well, given the messy data issues and numerous edge cases in this complicated scenario.  Wearing our favorite start-up hats, we were up and running within 3 months of writing our first lines of real code.

We deployed using some of the typical secure practices that you would expect in a 3 month start-up – we used Amazon Web Service’s security groups, we changed a few default passwords, we closed down ports, we password protected our interfaces and services.  We definitely took protecting user information seriously and we definitely wanted to avoid security problems; we also didn’t have very interesting information and we were favoring speed in our new business.  As our business matured, we grew our platform and eventually made an important decision to build out a more mature security program for a few reasons.  We wanted to more securely protect information, we wanted to signal that we really did take security seriously and we wanted to accelerate the security portion of our business discussions with partners and customers by providing audited third-party reports of compliance.  It was a big decision and a serious investment – lots of money, tons of time to get into place and ongoing drag.

I called it a security “program” because it is in no way a project that you complete – unfortunately for the nibble minded start-up inclined.  There are many standards and best practices available – there are entire industries formed around writing standards, providing services, auditing and issuing certifications.  We decided to pursue PCI Level-1 certification since many of our partners are familiar with these standards and our business is centered around credit cards.  We learned quite a bit about securing a cloud based platform that is PCI compliant, including a number of gotchas that tripped up security experts, consultant and auditors.  In the next few posts, I’ll describe some of the things that we learned and some of the challenges of this security program, as it relates to certifying a cloud based architecture.