Share this article
Engineering at Vanta: How we imported our AWS environment into Terraform
We use Terraform to manage the majority of our infrastructure as code. Terraform enables us to automate provisioning and easily review changes. Since we didn’t manage all of our infrastructure in Terraform—or monitor our AWS environment for unmanaged resources—our list of unmanaged resources continued to grow.
Resources that aren’t managed via Terraform are hard to keep track of and can potentially have security implications. Resources such as Route53 records, S3 buckets, and KMS keys were causing some problems when not managed in Terraform:
- We needed to ask a couple engineers to perform actions on our production environment that involved these resources. One example was to add DNS records for domain verification. These changes didn’t reap the benefits from GitOps, such as review and revertability.
- Resources not in Terraform differed across staging and production environments, and we didn’t have a deep grasp about how they differed.
- It was possible for an engineer to make temporary changes to resources and forget to revert them. This was a particular problem for our staging environment.
- Any static analysis we applied to our Terraform configuration wouldn’t apply to these unmanaged resources.
Moving existing resources to Terraform
The ability to create and destroy resources is well-supported in Terraform, but importing resources isn’t as intuitive. The team at Segment worried about inadvertently destroying resources when getting started, so they created a separate AWS account for Terraform and gradually removed resources from their old AWS account while creating the equivalent resource in their Terraform-managed AWS account. For our use case, this felt too heavy-handed. We already had an environment that was mostly, but not wholly, Terraform-managed.
Within the same AWS environment, we also considered creating an equivalent resource in Terraform, followed by the deletion of the corresponding, manually-created resource. For resources that had to have particular identifiers, such as S3 buckets or Route53 records, this was a non-starter—we would have had to delete the manually-created resources first and incur some downtime. For resources that didn’t have this problem, it wasn’t clear how we could mechanically verify that the resource described in Terraform was identical to the resource we created manually beforehand.
Thankfully, Terraform already has a command to add already existing resources to Terraform state without creating or destroying anything: terraform import. The import command will add a resource to state given configuration, but it will not generate that configuration in your .tf files for you. A subsequent terraform apply after the resource is imported will change the resource to reflect the configuration, so it is important to make sure that the terraform configuration reflects the current state of the resource.
We then broke up this problem into a few steps:
- Regularly monitor our environments for resources that are not managed via Terraform
- Import these resources via the import command
- Add an observability monitor to maintain the invariant that all resources that should be managed via Terraform are managed via Terraform
Thankfully, the monitoring tooling already existed. We ran driftctl as a Github Action on a regular cadence to determine what resources are in our AWS environment but not our Terraform state. The team at driftctl is also quite responsive. One caveat, however, is that driftctl doesn’t yet monitor all possible AWS resources. Here is the current list of resources that they support.
Next, we focused on the import command timing. If running the import out-of-band of your continuous deployment (CD) pipeline, a terraform apply at the wrong time could:
- Delete your newly imported resources
- Attempt to create redundant copies of the resources that you intend to import
If applying Terraform changes automatically in CD, it is important to atomically merge the new Terraform configuration into the CD pipeline and perform the Terraform import. Only then will the subsequent terraform apply become a safe operation.
In our case, we temporarily disabled applying any Terraform changes via CD before running our import step.
Since this was disruptive to our existing CD workflows, we wanted to go through this process as few times as possible. We aimed to perform dry runs of imports in development. Once we were sure that we described our configurations and import commands correctly, running our actual imports should be as simple as running a dry run script with the --do-it flag enabled.
Making Terraform imports dry-runnable
To verify whether our Terraform imports match the resource, we wanted to have a clean terraform plan. This terraform plan in turn needs to be compared against a Terraform state with imported resources. Terraform workspaces provide tooling to manage duplicate, temporary state files. After finishing the dry run, we can delete the workspace.
Be careful when working with state files. State files can store sensitive information, such as database passwords.
Our workflow looked something like this:
We pieced together a script to abstract away these details so that developers only focus on the address-id pairs (and the accompanying Terraform configuration).
Determining Terraform configuration
We’ve now reduced importing resources in Terraform to a process of guess-and-check. Even if we are completely incorrect in our configuration, the dry run’s terraform plan will give you a diff you can interpret as the reversed instructions to import a resource as is.
Let’s say we were trying to import a Route53 zone. We wrote up an attempt at Terraform configuration and now run our dry run script:
Oops! This configuration attempt forgot to associate the resource with a VPC, as it is a private hosted zone. The comment attribute was also missing.Even this process can feel somewhat tedious, so we used Terraformer to generate resource configurations for us. These configuration attribute values are string literals, so they still require some tweaking to instead reference attributes to already-Terraform-managed resources.
What resources are we responsible for?
Not all resources should be managed via Terraform. Here are some examples of resources driftctl marked as drifted which we decided not to import:
- IAM roles, policies, and policy attachments generated by AWS. Some examples include roles with the prefixes AmazonSSMRoleFor, AWSServiceRoleFor, AWS_InspectorEvents, and AWSReservedSSO . The AWS console’s policies page can give you some hints about whether certain IAM roles are customer managed versus AWS managed.
- Resources managed via a separate cloudformation stack
- Default AWS Network ACL rules
- AWS IAM Access keys, which cannot be changed if the resource is imported.
Thankfully, driftctl supports a .driftignore file that can ignore resources. Once we imported all of our resources, we found that deciding whether to manage a new resource in Terraform became easier. We now have more temporal context when a resource suddenly appears in driftctl that wasn’t there before, so we can make a guess about why the resource was created.
What we learned
After importing, deleting, or ignoring all the resources in our production and staging AWS environments that were not managed via Terraform, we added a Datadog Monitor to notify us via Slack if a new resource appears in driftctl. This monitoring gives us the added benefit of learning when AWS automatically creates certain resources in our AWS environment.
Making changes with Terraform can feel risky. Making changes replicable in a development environment helps us build confidence in the safety of these changes. Since Terraform imports are primarily changes in Terraform state, we were able to simulate imports by creating throwaway duplicates of our Terraform state. This enabled us to find a solution that was more effective than trying to create and then import a resource in a development AWS environment.
Ready to join the team?
Want to improve how we manage our infrastructure? Vanta’s engineering team is hiring. Join us!
Determine if you need to comply with GDPR
Not all organizations are legally required to comply with the GDPR, so it’s important to know how this law applies to your organization. Consider the following:
Do you sell goods or services in the EU or UK?
Do you sell goods or services to EU businesses, consumers, or both?
Do you have employees in the EU or UK?
Do persons from the EU or UK visit your website?
Do you monitor the behavior of persons within the EU?
Document the personal data you process
Because GDPR hinges on the data you collect from consumers and what your business does with that data, you’ll need to get a complete picture of the personal data you’re collecting, processing, or otherwise interacting with. Follow these items to scope out your data practices:
Identify and document every system (i.e. database, application, or vendor) that stores or processes EU- or UK-based personally identifiable information (PII).
Document the retention periods for PII in each system.
Determine whether you collect, store, or process “special categories” of data, including:
Determine whether your documentation meets the GDPR requirements for Records of Processing Activities, that include information on:
Determine whether your documentation includes the following information about processing activities carried out by vendors on your behalf:
Determine your legal grounds for processing data
GDPR establishes conditions that must be met before you can legally collect or process personal data. Make sure your organization is meeting the conditions listed below:
For each category of data and system/application, determine the lawful basis for processing based on one of the following conditions:
Review and update current customer and vendor contracts
For your organization to be fully GDPR compliant, the vendors you use must also maintain the privacy rights of your users’ and those rights should be reflected in your contracts with customers:
Review all customer and in-scope vendor contracts to determine that they have appropriate contract language (i.e. Data Protection Addendums with Standard Contractual Clauses).
Determine if you need a Data Protection Impact Assessment
A Data Protection Impact Assessment (DPIA) is an assessment to determine what risks may arise from your data processing and steps to take to minimize them. Not all organizations need a DPIA, the following items will help you determine if you do:
Identify if your data processing is likely to create high risk to the rights and freedoms of natural persons. Considering if your processing involves any of the following:
Clearly communicate privacy and marketing consent practices
A fundamental element of GDPR compliance is informing consumers of their data privacy rights and requesting consent to collect or process their data. Ensure your website features the following:
A public-facing privacy policy which covers the use of all your products, services, and websites.
Notice to the data subject that include the essential details listed in GDPR Article 13.
Have a clear process for persons to change or withdraw consent.
Update internal privacy policies
Ensure that you have privacy policies that are up to the standards of GDPR:
Update internal privacy notices for EU employees.
Have an employee privacy policy that governs the collection and use of EU and UK employee data.
Determine if you need a data protection officer (DPO) based on one of the following conditions:
Review compliance measures for external data transfers
Under GDPR, you’re responsible for protecting the data that you collect and if that data is transferred. Make your transfer process compliant by following these steps:
If you transfer, store, or process data outside the EU or UK, identify your legal basis for the data transfer. This is most likely covered by the standard contractual clauses.
Perform and document a transfer impact assessment (TIA).
Confirm you comply with additional data subject rights
Ensure you’re complying with the following data subject rights by considering the following questions:
Do you have a process for timely responding to requests for information, modifications, or deletion of PII?
Can you provide the subject information in a concise, transparent, intelligible, and easily accessible form, using clear and plain language?
Do you have a process for correcting or deleting data when requested?
Do you have an internal policy regarding a Compelled Disclosure from Law Enforcement?
Determine if you need an EU-based representative
Depending on how and where your organization is based, you may need a representative for your organization within the European Union. Take these steps to determine if this is necessary:
Determine whether an EU representative is needed. You may not need an EU-rep if the following conditions apply to your organization:
If the above conditions do not apply to you, appoint an EU-based representative.
Identify a lead data protection authority (DPA) if needed
GDPR compliance is supervised by the government of whatever EU member-state you’re operating in. If you’re operating in multiple member-states, you may need to determine who your lead data protection authority is:
Determine if you operate in more than one EU state.
If so, designate the supervisory authority of the main establishment to act as your DPA.
Implement employee training
Every employee in your organization provides a window for hackers to gain access to your systems and data. This is why it's important to train your employees on how to prevent security breaches and maintain data privacy:
Provide appropriate security awareness and privacy training to your staff.
Integrate data breach response requirements
GDPR requires you to create a plan for notifying users and minimizing the impact of a data breach. Examine your data breach response plan, by doing the following:
Create and implement an incident response plan which includes procedures for reporting a breach to EU and UK data subjects as well as appropriate data authorities.
Establish breach reporting policies that comply with all prescribed timelines and include all recipients (i.e. authorities, controllers, and data subjects).
Implement appropriate security measures
Have you implemented encryption of PII at rest and in transit?
Have you implemented pseudonymization?
Have you implemented appropriate physical security controls?
Have you implemented information security policies and procedures?
Can you access EU or UK PII data in the clear?
Do your technical and organizational measures ensure that, by default, only personal data that are necessary for each specific purpose of the processing are processed?
Streamline GDPR compliance with automation
GDPR compliance is an ongoing project that requires consistent upkeep with your system, vendors, and other factors that could break your compliance. Automation can help you stay on top of your ongoing GDPR compliance. The following items can help you streamline and organize your continuous compliance:
Explore tools for automating security and compliance.
Transform manual data collection and observation processes via continuous monitoring.
Download this checklist for easy reference
GDPR compliance FAQs
In this section, we’ve answered some of the most common questions about GDPR compliance:
What are the seven GDPR requirements?
The requirements for GDPR compliance are based on a set of seven key principles:
- Lawfulness, fairness, and transparency
- Purpose limitation
- Data minimization
- Accuracy
- Storage limitations
- Integrity and confidentiality
- Accountability
These are the seven requirements you must uphold to be GDPR compliant.
Is GDPR compliance required in the US?
GDPR compliance is mandatory for some US companies. GDPR compliance is not based on where your organization is located but whose data you collect, store, or process. Regardless of where your organization is based, you must comply with GDPR if you are collecting or processing data from EU residents.
What are the four key components of GDPR?
The four components of GDPR include:
- Data protection principles
- Rights of data subjects
- Legal bases for data processing
- Responsibilities and obligations of data controllers and processors
Safeguard your business with GDPR compliance
If your organization collects data from EU residents, GDPR compliance is mandatory for you. It’s important to follow the steps listed above to protect your business from heavy fines and to respect the data privacy rights of consumers.
Vanta provides compliance automation tools and continuous monitoring capabilities that can help you get and stay GDPR compliant. Learn more about getting GDPR compliance with Vanta.
Pre-work for your SOC 2 compliance
Choose the right type of SOC 2 report:
Do you sell goods or services to EU businesses, consumers, or both?
Do you sell goods or services to EU businesses, consumers, or both?
Do you sell goods or services to EU businesses, consumers, or both?
Determine the framework for your SOC 2 report. Of the five Trust Service Criteria in SOC 2, every organization needs to comply with the first criteria (security), but you only need to assess and document the other criteria that apply. Determining your framework involves deciding which Trust Service Criteria and controls are applicable to your business using our Trust Service Criteria Guide.
Estimate the resources you expect to need. This will vary depending on how closely you already align with SOC 2 security controls, but it can include several costs such as:
Compliance software
Engineers and potentially consultants
Security tools, such as access control systems
Administrative resources to draft security policies
Auditing for your compliance certification
Choose the right type of SOC 2 report:
Do you sell goods or services to EU businesses, consumers, or both?
Do you sell goods or services to EU businesses, consumers, or both?
Do you sell goods or services to EU businesses, consumers, or both?
Work toward SOC 2 compliance
Begin with an initial assessment of your system using compliance automation software to determine which necessary controls and practices you have already implemented and which you still need to put in place.
Review your Vanta report to determine any controls and protocols within the “Security” Trust Service Criteria that you do not yet meet and implement these one by one. These are multi-tiered controls across several categories of security, including:
CC1: Control Environment
CC2: Communication and Information
CC3: Risk Assessment
CC4: Monitoring Activities
CC5: Control Activities
CC6: Logical and Physical Access Controls
CC7: System Operations
CC8: Change Management
CC9: Risk Mitigation
Using Vanta’s initial assessment report as a to-do list, address each of the applicable controls in the other Trust Services Criteria that you identified in your initial framework, but that you have not yet implemented.
Using Vanta’s initial assessment report, draft security policies and protocols that adhere to the standards outlined in SOC 2.
Vanta’s tool includes thorough and user-friendly templates to make this simpler and save time for your team.
Run Vanta’s automated compliance software again to determine if you have met all the necessary criteria and controls for your SOC 2 report and to document your compliance with these controls.
Complete a SOC 2 report audit
Select and hire an auditor affiliated with the American Institute of Certified Public Accountants (AICPA), the organization that developed and supports SOC 2.
Complete a readiness assessment with this auditor to determine if you have met the minimum standards to undergo a full audit.
If your readiness assessment indicates that there are SOC 2 controls you need to address before your audit, complete these requirements. However, if you have automated compliance software to guide your preparations and your SOC 2 compliance, this is unlikely.
Undergo a full audit with your SOC 2 report auditor. This may involve weeks or longer of working with your auditor to provide the documentation they need. Vanta simplifies your audit, however, by compiling your compliance evidence and documentation into one platform your auditor can access directly.
When you pass your audit, the auditor will present you with your SOC 2 report to document and verify your compliance.
Maintain your SOC 2 compliance annually
Establish a system or protocol to regularly monitor your SOC 2 compliance and identify any breaches of your compliance, as this can happen with system updates and changes.
Promptly address any gaps in your compliance that arise, rather than waiting until your next audit.
Undergo a SOC 2 re-certification audit each year with your chosen SOC 2 auditor to renew your certification.
Download this checklist for easy reference
Prioritizing Your Security and Opening Doors with SOC 2 Compliance
Information security is a vital priority for any business today from an ethical standpoint and from a business standpoint. Not only could a data breach jeopardize your revenue but many of your future clients and partners may require a SOC 2 report before they consider your organization. Achieving and maintaining your SOC 2 compliance can open countless doors, and you can simplify the process with the help of the checklist above and Vanta s compliance automation software. Request a demo today to learn more about how we can help you protect and grow your organization.
Develop a roadmap for successful implementation of an ISMS and ISO 27001 certification
Implement Plan, Do, Check, Act (PDCA) process to recognize challenges and identify gaps for remediation
Consider ISO 27001 certification costs relative to org size and number of employees
Clearly define scope of work to plan certification time to completion
Select an ISO 27001 auditor
Set the scope of your organization’s ISMS
Decide which business areas are covered by the ISMS and which are out of scope
Consider additional security controls for business processes that are required to pass ISMS-protected information across the trust boundary
Inform stakeholders regarding scope of the ISMS
Establish an ISMS governing body
Build a governance team with management oversight
Incorporate key members of top management, e.g. senior leadership and executive management with responsibility for strategy and resource allocation
Conduct an inventory of information assets
Consider all assets where information is stored, processed, and accessible
- Record information assets: data and people
- Record physical assets: laptops, servers, and physical building locations
- Record intangible assets: intellectual property, brand, and reputation
Assign to each asset a classification and owner responsible for ensuring the asset is appropriately inventoried, classified, protected, and handled
Execute a risk assessment
Establish and document a risk-management framework to ensure consistency
Identify scenarios in which information, systems, or services could be compromised
Determine likelihood or frequency with which these scenarios could occur
Evaluate potential impact of each scenario on confidentiality, integrity, or availability of information, systems, and services
Rank risk scenarios based on overall risk to the organization’s objectives
Develop a risk register
Record and manage your organization’s risks
Summarize each identified risk
Indicate the impact and likelihood of each risk
Document a risk treatment plan
Design a response for each risk (Risk Treatment)
Assign an accountable owner to each identified risk
Assign risk mitigation activity owners
Establish target dates for completion of risk treatment activities
Complete the Statement of Applicability worksheet
Review 114 controls of Annex A of ISO 27001 standard
Select controls to address identified risks
Complete the Statement of Applicability listing all Annex A controls, justifying inclusion or exclusion of each control in the ISMS implementation
Continuously assess and manage risk
Build a framework for establishing, implementing, maintaining, and continually improving the ISMS
Include information or references to supporting documentation regarding:
- Information Security Objectives
- Leadership and Commitment
- Roles, Responsibilities, and Authorities
- Approach to Assessing and Treating Risk
- Control of Documented Information
- Communication
- Internal Audit
- Management Review
- Corrective Action and Continual Improvement
- Policy Violations
Assemble required documents and records
Review ISO 27001 Required Documents and Records list
Customize policy templates with organization-specific policies, process, and language
Establish employee training and awareness programs
Conduct regular trainings to ensure awareness of new policies and procedures
Define expectations for personnel regarding their role in ISMS maintenance
Train personnel on common threats facing your organization and how to respond
Establish disciplinary or sanctions policies or processes for personnel found out of compliance with information security requirements
Perform an internal audit
Allocate internal resources with necessary competencies who are independent of ISMS development and maintenance, or engage an independent third party
Verify conformance with requirements from Annex A deemed applicable in your ISMS's Statement of Applicability
Share internal audit results, including nonconformities, with the ISMS governing body and senior management
Address identified issues before proceeding with the external audit
Undergo external audit of ISMS to obtain ISO 27001 certification
Engage an independent ISO 27001 auditor
Conduct Stage 1 Audit consisting of an extensive documentation review; obtain feedback regarding readiness to move to Stage 2 Audit
Conduct Stage 2 Audit consisting of tests performed on the ISMS to ensure proper design, implementation, and ongoing functionality; evaluate fairness, suitability, and effective implementation and operation of controls
Address any nonconformities
Ensure that all requirements of the ISO 27001 standard are being addressed
Ensure org is following processes that it has specified and documented
Ensure org is upholding contractual requirements with third parties
Address specific nonconformities identified by the ISO 27001 auditor
Receive auditor’s formal validation following resolution of nonconformities
Conduct regular management reviews
Plan reviews at least once per year; consider a quarterly review cycle
Ensure the ISMS and its objectives continue to remain appropriate and effective
Ensure that senior management remains informed
Ensure adjustments to address risks or deficiencies can be promptly implemented
Calendar ISO 27001 audit schedule and surveillance audit schedules
Perform a full ISO 27001 audit once every three years
Prepare to perform surveillance audits in the second and third years of the Certification Cycle
Consider streamlining ISO 27001 certification with automation
Transform manual data collection and observation processes into automated and continuous system monitoring
Identify and close any gaps in ISMS implementation in a timely manner
Learn more about achieving ISO 27001 certification with Vanta
Book an ISO 27001 demo with Vanta
Download this checklist for easy reference
Download NowDetermine which annual audits and assessments are required for your company
Perform a readiness assessment and evaluate your security against HIPAA requirements
Review the U.S. Dept of Health and Human Services Office for Civil Rights Audit Protocol
Conduct required HIPAA compliance audits and assessments
Perform and document ongoing technical and non-technical evaluations, internally or in partnership with a third-party security and compliance team like Vanta
Document your plans and put them into action
Document every step of building, implementing, and assessing your compliance program
Vanta’s automated compliance reporting can streamline planning and documentation
Appoint a security and compliance point person in your company
Designate an employee as your HIPAA Compliance Officer
Schedule annual HIPAA training for all employees
Distribute HIPAA policies and procedures and ensure staff read and attest to their review
Document employee trainings and other compliance activities
Thoroughly document employee training processes, activities, and attestations
Establish and communicate clear breach report processes
to all employees
Ensure that staff understand what constitutes a HIPAA breach, and how to report a breach
Implement systems to track security incidents, and to document and report all breaches
Institute an annual review process
Annually assess compliance activities against theHIPAA Rules and updates to HIPAA
Continuously assess and manage risk
Build a year-round risk management program and integrate continuous monitoring
Understand the ins and outs of HIPAA compliance— and the costs of noncompliance
Download this checklist for easy reference
Download NowFEATURED VANTA RESOURCE
The ultimate guide to scaling your compliance program
Learn how to scale, manage, and optimize alongside your business goals.