AWS accounts provide excellent isolation boundaries for teams to develop and run services. With AWS Organizations, developers can logically group accounts and link them to a hierarchical OUs (Organization Units) structure. To find out what cool things are possible to do with AWS organizations, check the whitepaper "Organizing Your AWS Environment Using Multiple Accounts". The idea is pretty simple: organize, consolidate and link accounts in the OUs according to your org hierarchy (aka the org chart!), workloads (per application, service, environment), security (central logging, central VPC, central-anything-CISO-likes), shared services (CICD, deployments, secrets) and all other requirements. A pretty sensible and straightforward idea. 

What else do we need in addition to grouping, linking, and logical organization? Something will need to create, configure, and link these new AWS accounts, preferably as code. Developers and administrators will need a sign-in with SSO. 

Armed with these requirements, let's check what solutions and services are available to create and manage accounts and OUs with AWS Organizations. Ok, so where is all relevant information? AWS Organization is a fully managed service, so it must have all the above features, right? Wrong! If you start looking at how to create anything accounts-related with AWS Organization, there is a deep, deep, deep black hole. Why? Because Cloudformation doesn't support managing AWS Organization resources - four-plus years after AWS announced AWS Organizations service! AWS decided it was ok and doubled down on this Cloudformationless path: they built AWS Control Tower, AWS Account Management, and some other services with questionable Cloudformation support.

It is striking to have crucial resources in the organization managed without Cloudformation support, yet AWS kept building services to avoid it. Enter Control Tower, aka Rube Goldberg machinery of managing AWS accounts. The first iteration received the name "Landing Zones." AWS abandoned it and rolled out an improved service: "Control Tower," which still uses Landing Zones but only to set up and configure accounts. There are sometimes references to Landing Zones as "Account Factories," although the boundaries are blurry. The Control Tower is supposed to rule them all; however, it ended up in a weird state of being a mix of console-based click-ops or relying on third-party tools for automation. For example, there are four methods to provision accounts, two console-based and two automated.

The console methods are:

• Through the Account Factory console that is part of AWS Service Catalog.

• Through the Enroll account feature within AWS Control Tower

Here at Catenary, we don't do console. What are the other options?

• From your AWS Control Tower landing zone's management account, using Lambda code and appropriate IAM roles - with a link to a giant IAM policy and suggestion to "create Lambda functions in the AWS Control Tower management account"

• A mouthful thing called "AWS Control Tower Account Factory for Terraform (AFT)."

There is no simple option!

Overall, after looking at some of the solutions and reference implementations [1, 2, 3], I felt like this circus acrobat.

Thankfully, some brilliant people spotted and addressed this problem with a software framework. While AWS kept building these Rube Goldberg click-ops pieces of machinery, Olaf Conijn developed AWS Organization Formation (org-formation or ofn), a framework with the exact set of features to solve the aforementioned problems.

In the project's Github repository Olaf explains:

Just like with the resources within your AWS Account, managing AWS Organization resources as code allows you to apply changes automatically, reducing manual work, inconsistencies and mistakes.

If you are considering to use an account vending machine (e.g. AWS Control Tower) to create and manage new accounts within your organization: Do realize that the account vending machine allows you to quickly create organization resources but only has limited facilities when it comes to updating and maintaining these resources.

Well said. Creating resources is pretty easy, but keeping track of changes, and deploying changes in CICD will always be challenging.

I put together a quick-start deployment guide for org-formation. It is based on the original reference implementation repository, but it is simplified and adopted for those just getting familiar with AWS Organizations. I also wanted to go a little bit deeper into details on how to organize the repository, and where and how ofn stores code for the pipeline.
⁠
⁠The configuration is to create the following structure in AWS Organizations.

It is very simple yet fully functional and suitable for a simple app development workflow on AWS. The quick-start utilizes only a subset of configuration options possible with org-formation, however, accounts will have the following useful features:

• OUs and linked accounts with email, alias, and budget alarm for a threshold.
• SCP policy (restricting regions)
• CDK bootstrapped (usually a manual process), app accounts have trust configured with CICD account
• Automatic subdomain with DNS delegation. So the apex domain catenary.cloud can stay in the master account, and applications can have it's own hosted zones app-dev.catenary.cloud)
• Bootstrapped CICD pipeline to deploy config changes automatically
• IaC when everything is managed as code

There are even more features and goodies hidden under the hood, e.g. change-sets, simple cross-account Cloudformation links, state file in the S3 bucket, and parallel deployments. This is really great tool for keeping AWS accounts tidy. This approach would easily scale to hundreds or thousands of accounts.
⁠
If you are a cloud practitioner, DevOps engineer, a consultant I highly recommend getting familiar with the philosophy, design principles and implementation examples of the AWS Organization Formation tool.