Overview
Vertical Relevance highly recommends that the following foundational best practices be implemented to create a sustainable AWS Network that will support an enterprise-level organization.
Use AWS Organizations
To create a cross-account, cross-regional AWS Network Infrastructure in a manageable, scalable, and automated fashion, AWS Organizations must be configured. See our Accounts Foundation for details on AWS Accounts Best Practices. It is possible to create an AWS network that spans accounts not in the same organization however, architecting and managing such a network can be complex, tough to automate, and cumbersome to manage.
Form a Dedicated Network Team
Optimally, a company puts network management into the hands of the network team which in turn enables application developers to focus on building great applications. Network teams help ensure applications are communicating following best practices. They are the city traffic planners and are crucial for traffic communication and flow. Network design, implementation, and management is a major undertaking and will require dedicated support from network engineers as the company grows. Scalable cloud network architectures that span multiple accounts, regions, and are hybrid, require a dedicated network team for architecture and operations. Having a dedicated network team doesn’t mean that they must manage every single virtual network in AWS. Rather, they should oversee defining the larger network strategy while still providing some level of self-service networking products that developers can use to quickly deploy their required infrastructure.
When AWS Networks are allowed to be built without strict guidelines and proper planning, variations and security risks are introduced by default. It may also happen that certain functionality and design choices will make these networks incompatible with other future services and the cost of fixing it later will be exponentially higher.
Create a Dedicated Networking Account(s)
A Networking Account must be created under the AWS Organization as per the accounts foundations best practice described above. The Networking Account will be required to host all the shared networking resources such as Transit Gateway. It is also recommended that AWS Organizations Units are separated by Prod and Software Development Life Cycle (SDLC) where none of the SDLC accounts can reach Prod accounts. Creating separate network accounts for Prod and each stage of the SDLC will also help limit the blast radius. A Networking Account must be created under the AWS Organization as per the accounts foundations best practice described above. The Networking Account will be required to host all the shared networking resources such as Transit Gateway. It is also recommended that AWS Organizations Units are separated by Prod and Software Development Life Cycle (SDLC) where none of the SDLC accounts can reach Prod accounts. Creating separate network accounts for Prod and each stage of the SDLC will also help limit the blast radius.
The Networking Accounts should be registered as a delegated administrator account for managing CloudFormation StackSets. This will allow the networking account to spin up the networking solutions in an account other than the master account.
Register an Account as Delegated Administrator – This allows for an organization to have up to five registered delegated administrators with permissions to create and manage networking stack sets across the organization.
Enable Networking Account Access to Manage Shared Resources
AWS Resource Access Manager (AWS RAM) helps you securely share the AWS resources that you create in one AWS account with other AWS accounts. Enable organization-level sharing in AWS Resource Access Manager (RAM) in the management account. This allows Transit Gateway to be shared cross-account, so VPCs can be attached to it and communicate with each other.
Enable Trusted Access for CloudFormation Stacksets – This allows the CloudFormation Stack sets to deploy across the Organization with service-managed roles.
IP Management
When it comes to designing network architectures based on IPv4 protocol, there is one universal truth “never use overlapping IP addresses”. It is highly likely that two applications that currently don’t communicate, will be required to communicate in the future or may talk to a third application. Having overlapped CIDRs will require designing complex NAT solutions that are not scalable and add an unnecessary operational burden to network operators. We recommend implementing an IP Address Management (IPAM) solution to help manage a quickly growing list of CIDR space assignments and ensure that overlapping CIDRs are never used. AWS does not offer native IPAM solutions but third-party (paid or free) software like SolarWinds, Infoblox, Bluecat and Sapphire Ev10 are available for quick deployment in AWS Marketplace.
The network CIDR that most of our clients use is 10.0.0.0/8 which provides a total of 16,777,216 available IPv4 addresses.
Network Segments could be defined to further increase the number of available IP ranges. For example, a separate Dev, Staging, and Prod network could be created to ensure complete isolation and easier management of network space across these networks. Isolating the networks and network accounts by SDLC can significantly increase the available IP ranges for you.
Right-sizing each VPC CIDRs is vital to ensure that the IPs are preserved for the long term. We recommend using a t-shirt sizing approach to assign the CIDR space for different VPCs. See our recommended sizing approach below.
Note the first four IP addresses and the last IP address in each subnet CIDR block are reserved by AWS and not available for client use.
For modern networks using IPv6 CIDR blocks, IP management is no longer a necessity. IPv6 being a global space, you can request IPv6 CIDR block and AWS will provide a /56 CIDR space based on an AWS-owned pool of IPv6 CIDR spaces. These CIDR blocks will be always unique and will not cause any conflicts for your private networks. See IPv6 Addressing for VPC for further details.
Assessment for CIDR Assignments
The networking team should ask a range of questions from the application teams to grant them a proper allocation of CIDR spaces. The application teams in turn should reference their application architecture to determine these answers. Following is a sample list of questions that will help determine the CIDR spaces.
- How many IPs are required? – Simple test application with 3-tier architecture (ALBs, EC2s, and RDS/DynamoDB) may only require a “Small” size VPC as above whereas a large EKS Cluster hosting many microservices will likely require an XX-Large sized VPC.
- What Environment(s) will this application be deployed to?
- What type of workloads will be running in the said VPC? – For example, Running Lambda Functions in AWS VPC previously had scaling issues where each instance of lambda function would take up an IP space. However, AWS has resolved this limitation by introducing AWS Lambda VPC Networking with Hyperplane ENIs . Therefore, an application that previously required an XX-Large VPC Size might only require a Medium or Large sized VPC now.
- Architecture Review – In certain complex cases requiring high-number of IPs, it is recommended that the Networking Team review the application architecture to ensure that the needs are justified.
- Justification/Description – Similarly, a description of why certain CIDR number of CIDR spaces is required by the application team would be helpful for the networking team.
Define the Networking Strategy
Every organization has a varying range of network connectivity needs. To build a truly global network, we first must decide how we defined allowed network traffic patterns. There is also a large range of factors to consider including application architectures, cost, availability, scalability, ease of maintenance, and security/compliance requirements to name just a few. Some of these variables may be conflicting and therefore your network design should be implemented after careful consideration. Example If you’re a large company with Cloud and on-premises footprint, you may prefer to build with more complexity because it keeps costs lower at a high scale. In contrast, others may prefer to architect for simplicity and performance with a few added costs.
