This exists because terraform DSL, yaml, and Json are not programming languages and are thus tragic choices for making declarative statements
I have been saying this for a while now -- as much as I respect Hashicorp and admire their engineering, HCL is the biggest problem with the tools they have that it's used in -- biggest of which being Terraform.
Don't use a DSL where a full programming language is what you need. I can see that they probably wanted to be able to skip writing 2/3 language CDKs out of the box, and DSLs probably hit on the need for simplicity really early, but infrastructure code is somewhere you really want to be able to drop into the full expressive power of a programming language.
I use Pulumi[0] for my projects, and while I doubt people will be able to get it as much adoption as I think it should have in corporate environments, Terraform has introduced the CDK[1] which is a similar approach. Have a talk with your engineers about complexity (and how not to strangle the rest of the team with it) and stop using DSLs in places you need more expressive power.
I think a good indication of when you need to stop using DSLs or when you should maybe realize a DSL wasn't the right way to go is when you start doing things like writing control flow. You could can get away with DSLs if you were using s-expressions, but that is an exception that proves the rule (because there happens to a family of languages that treat s-expressions as syntax).
[1]: https://www.hashicorp.com/blog/cdk-for-terraform-enabling-py...
The moment you need control flow to define your resources, I'd argue that you're verging away from the realm of declarative infrastructure.
I'm using Terraform to manage 10^4 machines in combination with sane CI/CD, Bash/JQ (for dealing with Terraform outputs), Packer and Ansible. Everytime I see somebody reaching to a full programming language to define their infrastructure, they seem to be doing too much with one tool.
Terraform should merely provision things and in that role I find it fine as is. Preferred, even.
> The moment you need control flow to define your resources, I'd argue that you're verging away from the realm of declarative infrastructure.
Declarative infrastructure shouldn't be pursued for it's own sake -- what I want is efficient, and simple to manage infrastructure automation. The declarative nature is awesome, but once you start doing plumbing of variables and complexity from one static script from another, the cognitive load of keeping this all in line is better managed with a programming language in my opinion, you're just choosing bash/jq/awk/etc instead of a different language.
I think "the way the declarations are made must be static files" is dogmatic or at least limiting for me. Yes it is absolutely the simplest way to view what's present, but the problem is when someone goes into change any of this they will be dealing with your bolted-together complexity (even if it's not very complex).
> I'm using Terraform to manage 10^4 machines in combination with sane CI/CD, Bash/JQ (for dealing with Terraform outputs), Packer and Ansible. Everytime I see somebody reaching to a full programming language to define their infrastructure, they seem to be doing too much with one tool.
> Terraform should merely provision things and in that role I find it fine as is. Preferred, even.
I can't argue with the efficiency and efficacy of your setup, but I don't think much of this has to do with what we were discussing -- Pulumi does not seek to do the jobs of those other tools -- it's not going to build your VM images or do provisioning (unless you use it that way like with terraform[0]).
Here's a concrete example of a benefit I got form using Pulumi over terraform recently, in some code working with SES:
import * as fs from "fs";
// ... more imports and other lines
// Email Access key
const emailAccessKey = new aws.iam.AccessKey(
`${stack}-ses-access-key`,
{user: emailUser.name}
);
export const emailUserSMTPPassword = emailAccessKey.sesSmtpPasswordV4;
export const emailUserSecret = emailAccessKey.encryptedSecret;
// Write the smtp username and password out to a local secret file
const apiSecretsDir = path.join(__dirname, "secrets", "api", stack);
const smtpUsernameFilePath = path.resolve(path.join(apiSecretsDir, "SES_USERNAME.secret"));
const smtpPasswordFilePath = path.resolve(path.join(apiSecretsDir, "SES_PASSWORD.secret"));
emailAccessKey.sesSmtpPasswordV4.apply(password => {
console.log(`Writing SES SMTP username to [${smtpUsernameFilePath}]`);
fs.writeFileSync(smtpUsernameFilePath, emailUsername);
console.log(`Writing SES SMTP password to [${smtpPasswordFilePath}]`);
fs.writeFileSync(smtpPasswordFilePath, password);
});
I can't say that this is perfect Pulumi code (ex. I could have defined a custom Resource to do this for me), but I have saved myself having to do the plumbing with bash scripts and terraform output awk-ing, and the information goes just where I want it (NOTE: the secrets folder is encrypted with git-crypt[1]). When someone comes to this file (ses.ts), they're going to be able to easily trace where these values where generated -- similar with bash scripts, but now they don't have to be a bash/awk/jq master to manipulate information. There are definitely some gotchas to using Pulumi (like the `.apply` there), but in the end, I'd prefer to make changes like this in a consistent language I like (Typescript).
My toolkit looks very similar to you, except I basically only use make + kubectl + pulumi + ansible (rarely, because of the kind of servers I rent).
You can just write information out to files in Terraform with no stress.
In terraform resources this is what that looks like:
resource aws_iam_user email {
name = "email"
}
resource aws_iam_access_key email {
user = aws_iam_user.email.name
}
resource local_file smtp_password {
content = aws_iam_access_key.email.ses_smtp_password_v4
filename = "SES_PASSWORD.secret"
}
(I left out username, because I don't see where you're setting emailUsername)
In my pipelines I don't bother writing out to files things that are in terraform state. I just create an output for that state (potentially set to sensitive) and then use that output in my CI/CD. Remote state stays encrypted and without wide access and I don't have to worry about secrets being in files anywhere.
That's where the bash scripts do things with outputs. It could by python or whatever, it doesn't matter really. But with bash I can easily just set variables to `terraform output -json | jq `.
Mainly all I do is write terraform outputs to vault (i have simple bash automation to do all of this) and then I can use the Vault secrets in other CI/CD pipelines.
> You can just write information out to files in Terraform with no stress.
This wasn't the point -- it was that I wanted to do something that I know how to do in a fully featured programming language, and I can "just do it". Writing local files is a very simple example -- unless you're arguing that terraform's capabilities amount to an entire language's ecosystem, this is just the tip of the iceberg.
> So what's the plumbing that you would have to do? Under the hood, Pulumi is using the Terraform providers...
I think I didn't explain it well enough. Pulumi and Terraform are almost the same tool, but the difference is how the pieces are plumbed together. I prefer plumbing with a programming language more than shell scripts, utilities and/or some other programming languages.
Also, Pulumi's system of custom resources which are just pieces of code sitting in your codebase is fantastic and novel (terraform has custom providers but this feels significantly more heavy weight).
> In my pipelines I don't bother writing out to files things that are in terraform state. I just create an output for that state (potentially set to sensitive) and then use that output in my CI/CD. Remote state stays encrypted and without wide access and I don't have to worry about secrets being in files anywhere.
If you're really adjusted to Terraform, and it works great for you, then awesome -- I'm not out to change how you do things. It sounds like you've fully bought in to the terraform way of doing things, and it's working for you, and that's great.
> That's where the bash scripts do things with outputs. It could by python or whatever, it doesn't matter really. But with bash I can easily just set variables to `terraform output -json | jq `.
Here it is again... My point is that the plumbing matters, and a fully baked programming language offers the possibility of better plumbing. I didn't touch on it much, but just having access to custom resources with Pulumi might be able to cut down the external plumbing to zero, and enable creating more reusable pieces.
> Also, Pulumi's system of custom resources which are just pieces of code sitting in your codebase is fantastic and novel (terraform has custom providers but this feels significantly more heavy weight).
Pulumi literally uses Terraform's custom providers as its dependencies under the hood to make this work.
Moreover, you're entrusting your entire production stack to an ultra-aggressive, hypergrowth, early stage startup...
Right but you can see that the interface is easier as a custom resource though right? The Pulumi docs on making a custom resource are just extending a class. Making a good one I'm sure is fraught with peril but it's so much easier than trying to extend terraform as far as I can see, I'm glad that Pulumi has done this for me.
> Moreover, you're entrusting your entire production stack to an ultra-aggressive, hypergrowth, early stage startup...
Source code is Apache 2.0 and available[0]... I'm not against them trying to profit, but they've created a useful thing that is licensed very permissively (they could have gone with BSL or whatever else)...