I Cloned Twelve Ops Engineers with Nix

whoami

What Will We Talk About?

We'll talk about specific technologies_(whoops!) but in general:

What are some high-leverage ways to scale operational engineering?

How do we approach novel tools and evaluate benefits and drawbacks?

Background

Marketing-free Product Overview

Story Time

"There's a catch-all solution for this"

…maybe. But we’re going to think smaller

12 Problems

☁️ 9 Different platforms

➕

⚙️ Configuration management

➕

📈 Developer experience

➕

🧪 Testing

Technology and Cautionary Tales

• How to scale yourself when risk isn’t "we missed an OKR", but "the startup is out of money?"
• Buy vs. build to bridge the gap? Are we more strapped for cash or time?
• Prioritization is good but can only take you so far

some secret, third thing

USE WEIRD, FRINGE SOFTWARE

☁ 9 Platforms

• Create a network appliance that supports these platforms
  • Cloud, on-premise, and containers
• Lifecycle management
  • Getting updates to self-hosted environments?

Constraints

• Wide support: we need to support users where they are
• Automated as possible: single-digit engineer employee headcount!
• Testable: can't scale without assurances at release time
• Config-manageable: need to define and enforce system profile

The Hashicorp in the Room

…maybe

Hol Up

What if we tried something really different

Nix

Why Should I Care?

Explicit Dependencies

Impossible to ship built artifacts to targets with mismatched dependencies:

$ ./exe
libstdc++.so.6: cannot open shared object file: No such file or directory

Not only shared libraries, but all kinds of dependencies: in-$PATH programs, toolchain versions, etc.

Pure Outputs

Changes to real-world time, downloads, and build environment (like environment variables) can't impact build viability

Every input to your build is hashed based upon its contents

Buried lede: nix has total knowledge of how to build software

Abstract Syntax Trees?

Buried lede: nix has total knowledge of how to build software

Buried lede: nix has total knowledge of how to build software

What if we took the intermediate form of our system configuration – the abstract form – and then constructed different end results?

(this is sort of analogous to a Puppet catalog or a CloudFormation JSON if you squint)

☁ 9 Platforms (again)

What if we took the intermediate form of our system configuration – the abstract form – and then constructed different end results?

What if we took the intermediate form of our system configuration – the abstract form – and then constructed different end results?

What does it look like?

{
  imports = [
    nixos-generators.nixosModules.all-formats
  ];

  services = {
    caddy.enable = true;
    minio.enable = true;
  };
}

nixos-generators

Compile a system into arbitrary forms:

$ nix build .#qcow 
$ file result/nixos.qcow2
result/nixos.qcow2: QEMU QCOW Image (v3)
$ nix build .#vmware 
$ file result/nixos-vmware-x86_64-linux.vmdk
result/nixos-vmware-x86_64-linux.vmdk: VMware4 disk image

Key Benefits

• Sidecar APIs aren't necessary
  • We know the necessary format and have the bits, just create it
  • 100% local, but it does mean implementing another 10% yourself to push to AWS/Azure
• Easy support for arbitrary formats
  • Adding a new platform takes a one-line change

End Result

⚙ Configuration Management

Why or Why Not?

Caveats

Benefits

📈 Developer UX

Past, Present, and Future

What is a devshell?

{
  devshells.default = mkShell {
    packages = [
      rustc
      python311
      nodejs_20
    ];
  };
}

{
  devshells.default = mkShell {





  };
}

{
  devshells.default = mkShell {
    inputsFrom = [
      backend
      restapi
      frontend
    ];
  };
}

Development Environment Setup Steps

$ cd src/project

(full disclosure: direnv makes this possible with an .envrc)

Results: pretty good ✅

🧪 Testing

Impressions From a First-time CI Engineer

But Wait, There's More

• Toolchains
  • Python (Django), Node (Vue), Rust
  • Related libraries (for example, libsystemd for Rust)
• Integration tests
  • Networking product → multi-node networked tests

Testing Considerations

You Knew This Part was Coming

nixosTest {
  name = "network-appliance";
  nodes.appliance = { lib, pkgs, ... }: {
    services.nginx.enable = true;
    networking.firewall.allowedTCPPorts = [ 80 ];
  };
  testScript = /* python */ ''
    appliance.wait_for_unit("nginx.service")
    appliance.succeed("curl http://localhost")
  '';
}

Key Takeaways

• 📖 Internal Documentation
• 🫂 Developer support
• 💪 Confidence to build

📖 Documentation

• Foundational
  • What, how, and why
  • Pre-empting questions and workflows
• As an itch-scratching beacon
  • "Why am I writing docs when I could automate it?"

🫂 Support

• Leading with examples is important
  • Building initial CI/CD pipelines is a helpful illustrative case
• The purpose of a system is what it does
  • If things are broken, there’s no point at all

💪 Confidence to Build

This is the Last Slide