published 2020-07-28
★★★★☆ Pulumi's awesomeness is somewhat mitigated by its opinions.
I decided to try building the Azure deployment of my current project with Pulumi. Pulumi defines infrastructure in code using what appears to be a terse and well-designed domain-specific language. Defining deployment infrastructure in code makes deployments repeatable and transferrable.
The app I'm working on is designed to be deployed by a lot of different people in a lot of different places. The main app codebase should contain the deployment definitions as part of the app code, but those definitions will need to be able to be used in all kinds of contexts.
Pulumi is well-suited for these requirements, but some adjustments will need to be made in how we use Pulumi as compared with the defaults.
First, Pulumi puts some of the environment-specific variables as constants in the (Python) code (I'm using the Python variant, but the same things could be said for all of them) — for example, the sizes of various resources. These constants will have to be transferred to environment variables and replaced in the code with variables loaded from the environment (e.g., using Python's os.getenv('NAME') or Node's process.env.NAME).
So it's then just a matter of having a mechanism for injecting the environment-specific configuration into the working environment before running Pulumi.
for local development, I've been using direnv for this: You define your environment in an .envrc file, and the direnv program auto-loads the variables into the working environment when you switch into that directory. Very convenient. The .envrc file should not be committed to version control, but a template of it could be, which will help other users know which variables they need to define.
for deployment, you put your environment variables per-environment into your CI system or into your cloud provider, and then you inject them into the working environment when you are going to run the infrastructure / deployment code.
Second, replacing constants with environment variables is fine for the {Python, JS, ...} code. But what about the YAML? There we have things like
# Pulumi.ENV.yaml
config:
azure:location: EastUS
The Azure location really shouldn't be hard-coded into the YAML that is stored in the archive — that, too, should be an environment variable.
It turns out that this sort of configuration is stored in the environment-specific Pulumi.ENV.yaml files. Pulumi makes a distinction here:
Pulumi.yaml file and in the {Python, JS, ...} codePulumi.ENV.yaml files.For our purposes, it seems that the best approach is to ignore these Pulumi.ENV.yaml files completely, .gitignore them to keep them out of version control, and to instead define all deployment-environment-specific configuration as environment variables as discussed above.
And it looks like Pulumi will aid and abet us in this endeavor:
The Pulumi documentation suggests that you can ignore (and .gitignore) these Pulumi.ENV.yaml files just as I'm suggesting (see https://www.pulumi.com/docs/intro/concepts/config/#config-stack).
All of the configuration that the environment-specific Pulumi.ENV.yaml files contain can be defined in the {Python, JS, ...} code instead using pulumi.Config (as it's called in Python).
So you could truly ignore the Pulumi.ENV.yaml files, .gitignore them from version control, and just define all your configuration variables in the {python,...} code based on environment variables. That approach makes sense and is most useful for distributable apps. (Then you just have to instruct people to inject all the variables from their environment. Yeah, it's a little more work to create a distributable app that other people can use.)
A third problem with the default Pulumi setup, for the purposes of a distributable app, is that when you initialize a Pulumi project in Python, Pulumi creates a virtualenv in the venv folder under the working directory, populated from a requirements.txt file, and points to the venv folder from the Pulumi.yaml file.
This is not really a great example of setting up a Python environment, and it creates problems for distributing the code. From my perspective, it's much better just to add the Pulumi requirements (in my case, pulumi and pulumi-azure) to the project's deploy requirements (which might include the build and testing requirements, if build and test are being done in the same CI environment), and then do the local pulumi work in the project virtualenv. One codebase, one set of requirements.
As with the other issues, this one has a fairly easy solve: Just delete the following lines from the Pulumi.yaml:
options:
virtualenv: venv
When the virtualenv isn't defined there, Pulumi just looks in the current Python context, which is exactly what we would want.
The fourth and final issue I've run into so far is more of a quibble than a problem: In order to deploy your stacks, Pulumi wants you to
pulumi login
to pulumi.com, where they offer to track all of your deployments for you. Eh, no thanks. Well, thankfully, this is really easy to solve — instead, just type
pulumi login --local
and forget about pulumi.com (see https://www.pulumi.com/docs/troubleshooting/faq/#can-i-use-pulumi-without-depending-on-pulumi-com). The nice thing is that, if other people who use your project want to track their stacks for the project on pulumi.com under their account, they can do so just as easily as can be.
As I said, it's just a quibble. Pulumi is open source, but their tool wants to push you onto their infrastructure. That seems to me like a commerce smell. Maybe they are just trying to make it easy for you to version your deployments. Maybe. I also think they are trying to get you onto their infrastructure. After all, they need to make a living, and this is how they are choosing to try to get paying customers. I don't really like this sort of nudging built into the tools, in part because it is a signal that they might use even darker patterns in the future if their efforts-to-date haven't yielded the living to which they would like to be accustomed. But, despite my grumbling, it really isn't hard to not use pulumi.com, so whatever. It's nice that pulumi itself is open source so we can fork 'em if they really misbehave.
There are a lot of good ideas in Pulumi, and so far I'm impressed with how easy it looks to make deploying and maintaining infrastructure. For the sorts of apps I'm building, it's important to me to make deployment as easy as possible, and both repeatable and transferrable. And I love the idea of defining infrastructure in Python or Javascript or whatever I'm using rather than some hashi-dsl-thing. Pulumi looks like it fits the bill very, very well. Pulumi is pretty opinionated, and I will just have to grumble a little about some of their opinions — but only a little, because they make it pretty easy to replace those opinions with my own.