The Catenary Team's blog is around-the-cloud, AWS CDK best practices, conference talks, whitepapers, coding and research.

Blog

Berenice Cotero, Ph.D.

Cloud computing and the advancement of tools for resource management and infrastructure automation have created unconventional forms of processing massive amounts of data, both efficiently, reliably and cost-effectively. The effects of cloud computing can be visible in everyday life. Academic research is one of the fields that can benefit greatly from cloud computing. Particularly, genomics projects have successfully adopted cloud computing in research, and some start-ups have been created to offer services for mapping gene information to find links to pathologies by using AWS. Beyond genomics research projects, there is a vast amount of geospatial, satellite imagery, weather, news, and other types of 

. To date, most compute-intensive research tasks are run in supercomputers; the implementation of cloud computing has not been as fast as in other areas. 

In this talk, Berenice demonstrated how to analyze a set of data, automate infrastructure and trigger analysis when a set of data is uploaded.

Automation of Data Analysis in academic research with AWS

Roman Naumenko

⁠AWS CDK is a large software project. When AWS released CDK, they introduced many new terms and definitions at the edge of software development, infrastructure configuration management, and software abstractions. What's the meaning of those terms from a software development perspective?

I created a table with core AWS CDK terminology, a brief software-related explanation of each term, and further links to look up relevant docs or code.

CDK framework using novel terminology. As you can see, the terms are often used interchangeably (CDK Toolkit = CDK CLI). The best approach to grasp these terms is to consider CDK as a software project. It provides classes and functions to generate Cloudformation templates that can deploy and manage anything on AWS: from a single AWS resource to sophisticated applications.

AWS builds CDK as a hierarchy of abstractions to make this happen. Low-level CDK classes provide direct access to all config options in Cloudformation resources. Then higher-level CDK classes encapsulate patterns of defaults for Cloudformation services. Of course, developers do not stop there: CDK "architecture" classes can create services containing a dozen AWS resources.

The practical way to get familiar and keep up with CDK jargon is to treat it as a software project. I highly recommend checking examples in the documentation or source code concerning specific terms. That's where AWS keeps the definitions pretty consistent, compared to marketing and landing pages.

Here are a few rhymes to remember core concepts:

⁠In the second part of this post, I will dig into the definitions of classes types and other CDK programmatic internals.

AWS CDK Glossary - Part I

In part one of "AWS CDK Glossary," we went through the base CD terminology: constructs, construct libraries, the App, and the Stack. In this post, I'd like to explore constructs' programmatic properties and relations between constructs.

Do you know how does CDK distribute code and libraries? The same way as any other programming language distributes libraries and packages. For example, 

. It supplies all L2 Constructs, App, Stack, and all other extra classes of CDK. From a packaging and distribution point of view, the CDK version 2 delivers almost everything required for development with a single 

CDK has a CLI - just like any other respectful programming framework. But unlike other frameworks, the parents gave it an enterprise-y name: the "CDK Toolkit" rather than a simple "CDK CLI." I've never heard anybody saying anything like, 

"let me quickly run the diff with CDK Toolkit."

Ok, what about the constructs implementation? My goal here is to look into the CDK source code and see how they named things. But first, let's talk about distinct ways AWS advertises and writes about CDK. They created various web portals (for example, "Constructs Hub") to provide users with valuable sets of abstractions or "construct libraries." Wait, we just talked about NPM modules? Why not distribute packages via a standard supported channel for a programming language?

I think all kinds of CDK "web portals" with libraries are a necessary evil. The first reason is that CDK itself supports (distributed for) multiple languages. The CDK itself is TypeScript; but libraries can be published in JavaScript, Python, Java, C#, and Go.  AWS uses 

 compiler to bundle classes in TypeScript for other languages. Is it the same experience with CDK constructs across all languages? No, it is very different. The documentation for extra languages is minimal. For example, 

 package links to the "source code," which points to the AWS CDK GitHub repository.

 compilation and distribution model in mind when you pick up the language for the CDK application:  The CDK itself has the source code, TypeScript. For all other languages - it is a compiled "runtime" from the TypeScript source code.

What is inside the CDK application? Here is a minimalistic example of a CDK application in TypeScript.

This is a simple yet fully functional CDK application that creates an S3 bucket.

The application uses classes such as: App, Stack, Construct, Bucket. What are those classes for? And why are there so many if it only creates a bucket? I peeked at the definitions in IDE and copied them right from the source code and comments.

: "Represents the building block of the construct graph.

⁠All constructs besides the root construct must be created within the scope of another construct"

: A construct which represents an entire CDK app. This construct is normally the root of the construct tree.

⁠You would normally define an App instance in your program's entrypoint, then define constructs where the app is used as the parent scope.

: A root construct that represents a single CloudFormation stack.

* @param scope Parent of this stack, usually an `App` or a `Stage`, but could be any construct.
* @param id The construct ID of this stack.
* @param props Stack properties.

 An S3 bucket with associated policy objects.

⁠This bucket does not yet have all features exposed by the underlying BucketResource.

: Synthesize this stage into a cloud assembly.

⁠Once an assembly has been synthesized, it cannot be modified. Subsequent calls will return the same assembly.

The App is a top-level construct, and they probably used 

 is an AWS resource; however, the phrase 

 are related to the CDK framework rather than the application code.

This is the glossary of all the above terms.

I created an entity relation diagram I created with Mermaid.js that highlights the tree hierarchy of constructs.

output.png

That is a lot to digest! The good news is that the CDK as a software project is well-organized and logical. Here are distilled core programming concepts from classes such as Stack, Construct, App:

And finally, the framework can produce a somewhat surprising number of deployment artifacts called "Cloud assembly." The method of creating artifacts is synth(), and constructs can implement their own synthesized methods.

AWS CDK Glossary - Part 2

What are cross-stack references, and why can they be convenient in CDK apps and libraries?

Here is a real-world example of a CDK application with a cross-stack reference. A stack deploys an application that backups up and restores configuration from a file. Developers can create environments and deploy their own stacks. However, they prefer to have backups in one location. Maybe in S3 bucket, to have CDK provision applications from a backup passed as a parameter.

Let's create a "producer" stack with an S3 bucket to store backups - this is a shared stack. Here is the code (in TypeScript):

In the application ("consumer") stack, create a new interface that extends the StackProps interface to information about the output. Pass bucket name to the "consumer" stack as an object of props in the stack constructor: 

Here is the code for the "consumer" stack:

The CDK will do quite a few things under the covers:

Deploy stacks with "cdk deploy --all": voilà, new application stacks can reference the same bucket!

What happens when the "consumer" stack does not need the bucket anymore? Perhaps developers found another way to manage an application.

In earlier versions of CDK, if the reference is removed as shown below and stacks deployed - Cloudformation would fail. Cloudformation has internal validations to prevent removing exports referenced in other stacks. 

 lists a few "restrictions apply to cross-stack references" - it might be a good idea to check those out before developing complex (for example chained) stacks references.

I tested this example code with CDK version 2.8.0, it removes references automatically. In earlier versions, a tedious manual sequence required:

 provides a minimalistic code to create cross-stack references. There is the 

 branch for testing cross-stack reference.

Conclusion: CDK handles cross-stack references in a convenient, development- and testing-friendly way. And this CDK feature works well in real-world scenarios when stacks share resources.

Cross-stack references in CDK

Today is Nemo's first day at the DevOps school! The day starts with a stunning trip across production AWS accounts, lovely CI/CD pipelines, and other DevOps gems.

Finally, they are ready for the exciting trip to the drop-off - also known as the on-prem trench.

The teacher asks how to implement a conditional parameter for the Cloudformation template that manages SSL certs for the critical application.

q.jpg

That is not a fun question. Usually, developers end up implementing something in YAML the half-Turing way. Check template below, it is not easy to grasp the logic and conditions in YAML.

nemo_not_happy.png

Ok, Nemo! Don't hurt yourself and welcome aboard of AWS CDK - where the whole class of problem like this simple does not exist.

welcome.png

Here is CDK app that creates same resource (source code is available in 

Project source code

`subjectAlternativeNames: []` can be an empty array - no problem. The Certificate construct will handle it.

 if you're a small Dev fish, the CDK will increase your productivity (along with curiosity). It makes writing Cloudformation a breeze with a familiar programming language. 

The real adventure begins when the CDK development pattern is applied to a Cloud coral reef with a massive ecosystem of templates, applications, Dev- and other types of fishes. There might be even some security sharks coming to a party!

Science question about Cloudformation

Hashicorp recently announced CDK for Terraform on AWS with 

Lets take it for a test drive! Actually for a comparison test drive: how does development feel in CDKTF compared to AWS CDK native library?

I initialized cdktf project in VS Code, then added "aws-cdk-lib" dependency. Then I picked up a random resource, Route53 record. Then I've added child constructs into the stack. On the screenshot below is the result of both CDK applications side-by-side. 

1 CDKTF_w_AWSCDK.png

First impression, looking at the construct annotations on the images below, 

 resource which linked to external help site. The page describes the resource in HashiCorp Configuration Language (HCL). As a developer, I'd be curious what's that 

 exactly for? We also noticed the mapping between configuration object is not strict, for example 

AWS CDK library does not expose such low level details. 

 is a class which extends some other classes. The class properties provide interfaces for configuration options. 

2 Route53_w_Arecord.png

3 ARecord_w_Route53Record.png

Lets look more closely into classes configuration options. This is where the library abstractions will start to break for developers. The "name" is a just a string in CDKTF's Route53Record class with the link to Terraform documentation site about 

AWS CDK library properly annotates it as "The domain name for this record".

4 name_w_recordName.png

5 recordName_w_name.png

AWS CDK starts to shine when the configuration options have specific types. For example, the 

 is not just a string as in CDKTF, but a particular type 

. Developers have to create const to define the zone correctly. There are few convenient methods in 

 class. It is pretty easy to find out which one should use. When developers write CDKTF, they will have no clue how to define 

, unless they research help sections of Terraform site.

Smashing types into strings is not a very good development experience for someone unfamiliar with Terraform syntax and resource definitions.

Next three images show how to define zone with types checking.

75 zone_with_zoneid.jpg

6 HostedZone.png

7 IHostedZone.png

 is a list of strings. Strings provide no context what kind of records they are. In the AWS CDK library, 

 class, which lists available options for the record target, including a particular AWS type of 

The CDKTF and CDK application classes look pretty similar; however, there are substantial differences in how libraries design, annotate, and abstract resources in the cloud. There is also almost no types enforcement in CDKTF, while AWS CDK requires  typed properties. Static types in AWS CDK is a massive boost to productivity: developers do not have to guess what "string" means, or what options class constructor needs.

AWS CDK composes classes and interfaces much better as well. In CDKTF there is one class for this example: 

 class and then maps everything to Terraform's 

 interface from the CDK library to create A record.

Static typing for programming infrastructure

AWS accounts provide excellent isolation boundaries for teams to develop and run services. With AWS Organizations, developers can logically group accounts and link them to a hierarchical OUs (Organization Units) structure. To find out what cool things are possible to do with AWS organizations, check the whitepaper "

". The idea is pretty simple: organize, consolidate and link accounts in the OUs according to your org hierarchy (aka the org chart!), workloads (per application, service, environment), security (central logging, central VPC, central-anything-CISO-likes), shared services (CICD, deployments, secrets) and all other requirements. A pretty sensible and straightforward idea. 

What else do we need in addition to grouping, linking, and logical organization? Something will need to create, configure, and link these new AWS accounts, preferably as code. Developers and administrators will need a sign-in with SSO. 

Armed with these requirements, let's check what solutions and services are available to create and manage accounts and OUs with AWS Organizations. Ok, so where is all relevant information? AWS Organization is a fully managed service, so it must have all the above features, right? Wrong! If you start looking at how to create anything accounts-related with AWS Organization, there is a deep, deep, deep black hole. Why? Because Cloudformation doesn't support managing AWS Organization resources - four-plus years after AWS announced AWS Organizations service! AWS decided it was ok and doubled down on this Cloudformationless path: they built AWS Control Tower, AWS Account Management, and some other services with questionable Cloudformation support.

It is striking to have crucial resources in the organization managed without Cloudformation support, yet AWS kept building services to avoid it. Enter Control Tower, aka Rube Goldberg machinery of managing AWS accounts. The first iteration received the name "Landing Zones." AWS abandoned it and rolled out an improved service: "Control Tower," which still uses Landing Zones but only to set up and configure accounts. There are sometimes references to Landing Zones as "Account Factories," although the boundaries are blurry. The Control Tower is supposed to rule them all; however, it ended up in a weird state of being a mix of console-based click-ops or relying on third-party tools for automation. For example, there 

 to provision accounts, two console-based and two automated.

Here at Catenary, we don't do console. What are the other options?

Overall, after looking at some of the solutions and reference implementations [

getting started control tower

control tower account factory

aws security reference examples 

collage_400.png

Thankfully, some brilliant people spotted and addressed this problem with a software framework. While AWS kept building these Rube Goldberg click-ops pieces of machinery, 

 developed AWS Organization Formation (org-formation or 

), a framework with the exact set of features to solve the aforementioned problems.

In the project's Github repository Olaf explains: 

Just like with the resources within your AWS Account, managing AWS Organization resources 

 allows you to apply changes automatically, reducing manual work, inconsistencies and mistakes.

If you are considering to use an account vending machine (e.g. 

) to create and manage new accounts within your organization: Do realize that the account vending machine allows you to quickly create organization resources but only has limited facilities when it comes to updating and maintaining these resources.

Well said. Creating resources is pretty easy, but keeping track of changes, and deploying changes in CICD will always be challenging.   

Quick start deployment guide for org-formation

 for org-formation. It is based on the original 

reference implementation for org-formation

 repository, but it is simplified and adopted for those just getting familiar with AWS Organizations. I also wanted to go a little bit deeper into details on how to organize the repository, and where and how 

 stores code for the pipeline.
⁠
⁠The configuration is to create the following structure in AWS Organizations.

output-1.png

 It is very simple yet fully functional and suitable for a simple app development workflow on AWS. The quick-start utilizes only a subset of configuration options possible with org-formation, however, accounts will have the following useful features:

There are even more features and goodies hidden under the hood, e.g. change-sets, simple cross-account Cloudformation links, state file in the S3 bucket, and parallel deployments. This is really great tool for keeping AWS accounts tidy. This approach would easily scale to hundreds or thousands of accounts.
⁠
If you are a cloud practitioner, DevOps engineer, a consultant I highly recommend getting familiar with the 

Alternative to Rube Goldberg machinery of AWS Control Tower

Today we talk about dreaded cloud lock-ins. While the fears of cloud lock-in have largely subsided, there are still vendors peddling it as a major concern for businesses.

To understand what lock-in even means, lets check dictionary first:

: an arrangement according to which a person or company is obliged to deal only with a specific company.

Ok, so this is an arrangement. And arrangements last; people avoid changes. So do companies which prefer to stick with what works. No reason exists for changes when a supplier or counterparty satisfies all requirements.

Over the years, a ridiculous list of anti-lock-in requirements formed for public cloud vendors. And by vendors, it is a specific dominant vendor because moving to the cloud currently means one-way street to AWS. The list goes like this:

Nobody believes any of this, of course. If they did - they would send money to build the anti-locked-in cloud and services.

Remember how Google announced a universal cloud library years back? It promised to have abstracted classes - Storage, Computer, etc for every major cloud vendor. And the idea was that the developers create resources and then swap quickly underlying cloud because cloud-specific API would be abstracted.

Whoever adopted that library probably ended up in tears, completely rewriting everything using native cloud SDKs.

Google ended up supporting the basic API of S3 on their cloud storage service; that’s about it for the compatibility.

Back to the arrangement of lock-ins. Even though it is a noun, the dictionary defines it as 

“the action, the process, or result of arranging or being arranged”

. See, the lock-in is not a state - it’s a 

. And companies are pretty good at changing or adapting processes.

Ok, I hear you asking: show me an actual case where the company considers lock-in a good option?

Glad you asked, there are growing number of examples out there (promise not to talk about Netflix lock-in because nobody cares anymore what Netflix is doing in the cloud, and nobody can repeat it anyways - although people keep trying hard. It’s called 

Here is an actual case. A university wants to use AWS exclusively, bypass all resellers, and go all-in public cloud, locking themselves entirely in. Because they are not a private company, purchasing department had to publish the “Sole Source Justification Form” and conduct a public hearing. Here's what they have to say about lock-in and using public cloud.

First, they clearly explain why they need a public cloud (and not many companies can do it easily without resorting to the digital transformation white noise). 

“Current faculty and students frequenting the Universities 

expect to be able to readily utilize modern Internet and cloud technologies as a part of their education and research endeavors

.” - re-read it; this is a great mission statement for anybody looking to build a business case for cloud adoption.

Then they gave the list of reasons for lock-in into AWS:

They also gave more detailed explanations why using AWS, and some of the responses are truly amazing! A few years ago, it would have been the standard boilerplate text about the hybrid cloud - at best. And here we are, when the purchasing department of Univercity explains why AWS Organization solves its problems at scale. And why direct access to API is better than any vendor’s cloud-management tooling.

I also ponder how they use Mechanical Turk (

university wide agreement regarding Amazon Mechanical Turk

Anyways, here it is - the form question and the answer.

Provide detail explaining the justification ... to explain why the requested supplies or services are the only ones available that can satisfy the agency or university requirements?



⁠The AWS Educate program, Mechanical Turk and the research credit programs as provided by AWS are unique offerings not provided by resellers. In addition, having a direct relationship with AWS ensures that IPHEC will be able to use all AWS tools now and into the future. This direct relationship with AWS will also provide IPHEC access to the financial Application Programming Interface (API) AWS provides, without having to go through subsidiary tools that built and maintained by resellers which may provide delays and due to the nature of the data, warrants itself to additional errors that requires more overhead costs and time loss.

What are the unique features of the supplies or services that are not available in any other product or by any other vendor? Provide specific quantifiable factors/qualifications:


⁠
⁠AWS uniquely bundles their offering and services in a way that no other vendor can provide. Although the bundle is unique and provides a unique direct relationship, as the owner or "manufacturer" of the offering and services they are the only provider that can provide their products and services in it's pure form.

Additionally, there are two key functions that are not available through other vendors, regardless of their application or form. These services are the Application Programming Interface (API) and AWS Organizations.

API is the billing application programming interface (API). AWS makes available a programmatic interface that provides access to their billing data. This is the preferred way to access billing data; it is easily accessible to scripts and applications, and it provides powerful built-in capabilities to query and filter billing data. If members had access to this API, members could easily inform customers and business officers:
⁠
⁠• Which resources are driving cost, the projects/services they support, and who created them.
⁠• Unexpected changes in daily spending.
⁠• Account compromise and unauthorized activity.
⁠
⁠Members cannot do fine grained queries directly without access to the billing API. Agreements between AWS and resellers prohibit exposing that API to downstream customers like the Universities. This creates a barrier that prevents members from effectively establishing tools that can provide real time financial alerting and controls to the degree that the campus business officers have requested.
⁠
⁠Second, is the AWS Organizations tool, Amazon’s solution for large-scale account management. IPHEC members operate well over 200 AWS accounts, and we have repeatedly run into scaling problems solved by the AWS Organizations tool. Among the most valuable features:
⁠
⁠• Organizations would allow us to create and enforce security policies that cannot be disabled within an account, enhancing our security capabilities and regulatory compliance.
⁠• Organizations would allow us to automate the creation of new accounts, which is currently a time-consuming manual operation.
⁠• Organizations enhances and consolidates AWS billing APIs. We could give administrators and business managers a single view of all spending across their department’s AWS accounts.

This is an ultimate testament to the benefits of the arrangements!

I hear you’re asking, “what about the managing riks?” Shouldn’t the university apply some risk management here? Well, they did. And not how the adepts of anti-lock-in think. The form asks what happens if the sole source is not approved.

You’d better know the possible answers if you’re a seasonal cloud practitioner. The answer was:
⁠

the university will lose invaluable unique features available to them and the costs will be significantly increased

In other words,  the lock-in ultimately provides unique features and low cost.

glocks.png

Right at the left upper corner of the “Cost of Useful Unit” graph. Overspend on arrangements - and it will be a useful yet very costly cloud. Preparing to escape lock-in will inevitably bring extra costs with nothing to show for it.

Cloud Goldilocks-in Benefits: Perspectives from Large Organizations

ChatGPT tackled the challenge of writing a simple AWS Cloud Development Kit (CDK) app from scratch in Typescript, creating a basic app with VPC, private subnets, and a NAT Gateway. Through a series of prompts, we guided the AI through the app creation process, addressing any errors or issues encountered.

In our 20-minute interaction, we faced multiple errors, such as outdated CDK versions, deprecated properties, and incorrect imports. ChatGPT provided instructions for updating the CDK version, replacing deprecated properties, and using correct imports. Some issues took several interactions to resolve. At some point ChatGPT started making bogus excuses like "this could be due to local cach issues with npm" - well, that did sound like a developer's complaint.

It was an interesting experiment, however we wouldn't recommend using bots to write actual code. Although ChatGPT generated valid code, the process was slow and needed frequent corrections. Developers can work more efficiently with IDEs and other tools to catch and correct errors. We spent about 30 minutes to get ~100 LOC (~50 CDK Typescript and the rest Node and CDK configs). This is not very productive in terms of code writing.

It also were making multiple error, couldn't find a good suggestion for subnet type. CDK changed enum for subnets few times, sure. This kind of information is so well encoded and widely availale, that there is no reason for ChatGPT not to improve in the future.

cdk_from_scratch_err_fix1.jpg

It really hung up on other errors, making the same mistake over and over.

cdk_from_scratch_err_fix2.png

What ChatGPT is very useful for is exploratory interactions, such as guiding beginners through creating a bare-bones CDK app step by step. However, learning to use ChatGPT effectively requires mastering prompts, interactions, and other techniques that itself resemble the software development process. There is no way to avoid it. 

Engineers should maximize their efficiency by combining IDE tips and hints, knowledge of CDK compostions and abstrations, and ChatGPT's ability to generate examples and explain nuances. This approach is more effective than relying solely on code generation. Engineers also should use code bootstraping, init commands where possible. 
⁠
⁠Final result - the app.

chatgpt_code.jpg

chatgpt_code2.jpg

Making ChatGPT Write CDK: Slow and Furious

, we discussed how to use ChatGPT to generate a basic CDK application. However, as many developers know, building a functioning CDK app takes more than just generating code snippets. It requires understanding of the language-specific configurations.
⁠
⁠Many guides and examples for building Cloud Development Kit (CDK) applications start at a high level of complexity, which can be overwhelming. Follow instructions and it works, however it's more like magic. Our approach is to provide a clear understanding of the essential elements required to create a functional CDK app.

We cleaned and simplified generated code and published it at 

 repository. It provides a clear, easy-to-follow structure for building a CDK app from scratch. It covers the essential building blocks, how to set up a new CDK app, configure TypeScript, install required npm modules. It is pretty amazing how simple yet powerful and useful CDK app can be!

 is the main entry point of the CDK app. It contains the code that defines the AWS infrastructure resources (we used VPC with 3 subnet types as an example).

  is a configuration file that tells CDK CLI how to run application. It specifies the CDK app entry point, which is language specific. It can also specify command-line options to pass to the CDK CLI, such as the AWS profile or region to use.

 contains the TypeScript compiler options for the project. It tells the tsc compiler how to compile the TypeScript code into JavaScript and what configuration settings to use.

 contains the metadata and dependencies for the project. It tells the Node Package Manager (NPM) what packages and modules are needed to run the application.

ChatGPT can be an excellent tool for generating CDK code snippets. By starting from scratch and building upon the basics, you can gain a solid foundation for developing robust and scalable cloud infrastructure. Despite its barebones structure, this simple app generates functional Cloudformation template 450 lines long!