A hand-cranked pump in a rice field

The Well in the Field

When I was a child, I lived with my parents in the woods. We had an old-fashioned hand-dug well, the kind from storybooks that looks like a big hole in the ground lined with stacked stones. In summer, if there was a drought, the well would sometimes run dry.

Read More

Hiking near the Meramec River

SIGAVDI #82: Broccoli Rabe Edition

Hello friends,

It’s been another couple weeks of hunkering down with Jess & family in St. Louis. It turns out the way I deal with social distancing is to buckle down and work really hard. When all this is over I may need to impose periodic “productivity quarantines” on myself to match this level of output.

Read More

Reproducibly Setting Production Environment Variables for AWS Lambda Apps

The problem with infinitely flexible building blocks is that there are always a half-dozen well-documented ways to do things wrong.

I wanted my AWS Lambda app (I refuse to call it a “function” because it isn’t) to get the third-party API keys it needs from environment variables. Not from hardcoded strings or config files.

The most immediately accessible way to set up env vars for a Lambda is via the AWS console. But that’s not a reproducible, versionable workflow I can capture in my project.

Searching for info on setting up environment variables in a SAM-based AWS Lambda app turns up a lot of results. It seems you can set variables in the SAM config in template.yaml.

Resources:
HelloWorldFunction:
  Type: AWS::Serverless::Function
  Properties:
    CodeUri: hello_world/
    Handler: app.lambda_handler
    Runtime: python3.7
    Environment:
      Variables:
        API_KEY: 12345...

More usefully, you can define env vars that apply to all of your functions by putting them in the globals section that normally lives at the top of template.yaml

Globals:
Function:
# ...
  Environment:
    Variables:
      API_KEY: 12345...

Which is all great, except that I explicitly don’t want these variable values to apply universally, and I don’t want them to be checked-in to my repo. I just want to set them for my production deployment.

My searching next took me down the rabbit hole of AWS Systems Manager Parameter Store. But that’s way too heavyweight a solution for what I want to do.

Finally, I realized that I could probably set env vars via the aws CLI (not sam). Like everything to do with AWS, the command isn’t particularly intuitive, but here’s what I cam up with:

aws --region us-east-1 lambda  update-function-configuration \
   --function-name myapp-MyFunction-1234ABCD \
   --environment "Variables={API_KEY=12345}"

Some points to note:

  1. Nothing worked until I explicitly set the –region
  2. Environment variables are part of a “function configuration”
  3. The function name for a SAM-managed app is the full name that SAM invents, including the app name, the function name, and some ID garbage at the end.
  4. You can’t just set environment variables directly; that would be too easy. You need to pass Variables={...}, where the contents of the curly braces are key=value pairs delimited by commas.

This gives me a documentable, reproducible template for setting API key environment variables in just the production deployment. Which is what I wanted.

Oh no, Sam the Firefly was in a jar!

AWS SAM in a Docker Dev Container

When you look up AWS Lambda tutorials they usually walk you through using the web console to roll out a function-as-a-service. This is a nice parlor trick, but it doesn’t help you do real work on Lambda because it’s not reproducible. Until Github Actions can include “click 150 different things on console.aws.amazon.com”, reproducible builds means writing code on a developer machine, and making use of code-as-config and command-line tools to build and deploy.

So yesterday I took another crack at learning AWS SAM (Serverless Application Model), which is a command-line-and-config-files abstraction layer on top of AWS CloudFormation.

Read More

SIGAVDI #81: Mustard Greens Edition

Hello friends,

It’s been a weird week for all of us, I think. I don’t have a lot more to say about that, so I’m just going to proceed forward as usual.

I'll Trade Ya!

Hey there! Archived SIGAVDI letters are for newsletter subscribers only. All it costs to join (and unlock this post) is an email address! I'll write to you weekly-ish with a few interesting links, some updates, and some reflections on the intersection of software and life. And I'll respond to your replies! Whattya say?

A photo of Avdi Grimm

What’s Good

What’s New

Where to find me

Toss a coin to your SIGer

And now a word from our sponsor: me! Do you like SIGAVDI? Do you enjoy my other publications like that video with Eric Normand? Consider tossing a buck or two my way on Patreon. Thanks! 🙏


What to write of my life in these strange days? As a programmer, my life is your life is our life now: sequestered in our houses, adjusting to a world in which work-from-home is no longer a perk but a mandate.

One minor-but-real risk we confront almost immediately is the mushing-together of time: what is to differentiate the days when our surroundings are the same 24/7? A first line of defense: a new appreciation for, and dedication to, the seasonal holidays. Sláinte, friends.


In The Closed World, author Paul Edwards says that there are “two major genres of historiography” when it comes to the computing revolution, but that:

…the tropes and plotlines of both genres impose requirements that lead authors to ignore or downplay phenomena outside the laboratory in the mind of the scientist. Both versions of the story explain developments in a given field solely from the perspective of actors within it. As Mahoney puts it, the authors of this “insider history… take as givens… what a more critical, outside viewer might see as choices”. There is little place in such accounts for the influence of ideologies, intersections with popular culture, or political power. Stories based on the tropes of progress and revolution are often incompatible with these more contingent forms of history.

This puts me in mind of The Californian Ideology:

Because these core workers are both a privileged part of the labour force and heirs of the radical ideas of the community media activists, the Californian Ideology simultaneously reflects the disciplines of market economics and the freedoms of hippie artisanship. This bizarre hybrid is only made possible through a nearly universal belief in technological determinism.

(Emphasis mine).

The common theme here is the belief that technological progression—and the resulting business, economic, societal, and political upheavals stemming from it—flow out more or less inevitably and mechanically from technical advances. But what if we have more agency in this process than we imagine? More importantly: who benefits from us, the technologists, adhering to this deterministic view of blind, inevitable change?


What’s next

Priority one is still finding a new gig. I’m also taking advantage of the enforced down-time to up my rate of publishing, do more collaborations, and tackle a bunch of unshaven yaks on my websites.

Last week(s):

  • ✔ Find someone for whom to do some short-term billable work. The next round of funding for a video project I’ve been consulting on came through, so that’s going to be a priority this week.
  • ❌ Do some actual billable work. But soon…
  • ✔ Get through one more (ugh) job search hoop.
  • ✔ Narrow down a longer term consulting client on the cautious assumption that the preceding item won’t go anywhere.
  • ✔ Catch up on mail etc.
  • ✔ Make at least one quickie RubyTapas episode.
  • ✔ Come up with next tasks for the guest episodes that are now stalled because I have less organizational help.
  • ✔ Stream something educational. Not technically streamed, but I’m counting that Eric Normand interview.

Bonus points:

  • ✔ Kick off a new remote work newsletter collaboration.
  • ✔ Revamp the Wide Teams website in support of the above.
  • ✔ Update subscription forms etc. to reflect the BRUNCH/SIGAVDI merge.
  • ✔ Study Dark a bit more.
  • ✔ Free up a RubyTapas episode
  • ✔ Send out a new “RubyTapas from the Freezer” email after a long hiatus

This week:

  • Work on $corporate_video_project
  • Spend a couple hours on one of my WIP courses.
  • Stream with Jess
  • Write for The Wide Teams Almanac
  • RubyTapas

Threat board:

  • GOTO talk prep – gotta get ready for a very different format.
  • Becoming revenue-positive.
  • WIP.
  • Fixing my house.

WIP:

No movement this week.

  • Client Video Project: ~85% .
  • Robust Ruby: 25%.
  • MOOM: 91%
  • The Rake Field Manual: 10%.
  • AsyncJS Course: ~50%.
  • Patreon improvements.

That’s it for this week. Thanks for reading, and as always, feel free to write back!

Cheers,

Avdi

A single red 20-sided die on white | © Michaeleisenhut | Dreamstime Stock Photos | © Michaeleisenhut | Dreamstime Stock Photos

A Tool for Choosing a Tech Stack

OK so I understand you think teams should stick to the tech stack they have the most familiarity with.

Yes, under most circumstances.

Riddle me this though: I have a freshly-assembled team. One dev is fresh off a Rails bootcamp. One is a veteran Java developer. One has spent most of their time with PHP and Laravel. And the last one is a Node/Express fan.

Read More

Cat stalking his feather toy

New project, new tech stack?

What is the right language and framework for this new project?

What language and framework is your team most familiar with?

I guess… most of us have worked on Java/Spring projects…

Then those are the right technologies to use.

What about Ruby and Rails, though? Aren’t you a big fan of those?

Is your team a big fan of them?

Well, no…

Then it’s not the right choice for this project.

Wait, are you saying that we should always just use what we are familiar with?

Almost always. 

What’s an exception?

Some domains have strongly converged on a particular tech stack. If there is an overwhelmingly better ecosystem for the kind of work you are doing, the it may be worth it to take the productivity hit of getting up to speed on a new stack and its accompanying idiosyncrasies and idioms. 

Can you give me an example?

Well, a lot of scientific work is done in Python. So the ecosystem of science tools and libraries is particularly mature, and there are a lot of people who can help you if you get stuck. Also, a lot of scientists have used some Python, so if they are your users, choosing Python may give you a leg up working with them. 

What about scaling issues, though? Wouldn’t we be better off choosing Go or Elixir?

I don’t know. Would you? Do you have enough information right now to be certain that the types of scaling issues you are going to run into are going to be the type that Go or Elixir would make a crucial difference to your project’s success?

I don’t know… 

Do you have confidence that your team will design a Go or Elixir system in a way that it will benefit from those language’s strengths when the time comes? What did your team’s first Java/Spring system look like, architecturally speaking?

Oof, I see your point. But… it’s just that some of the team members have been playing with Elixir lately, and they’ve been really impressed with it.

Ah! I understand now. Your team is experiencing Tech Fatigue.

What?

Your developers have been using Java and Spring for years, and they are bored. They are watching people give talks about other languages and frameworks and different approaches to problems, and they are discontent.

So you’re saying we’re just chasing a new shiny object?

Yep. And you’re hoping I’ll give you a solid technical justification to pursue that shininess.

Ouch. So should we just suck it up and use what we know?

No!
Programmers are creatives. Creatives need new media and new constraints to get their juices flowing. New languages and tools can help developers break out of their creative ruts, and regain excitement for their work.

Wait… so should we go ahead with new tech after all?

Yeah! I say go for it. But! Be honest with yourselves. You’re not doing this because you know it’s the right choice architecturally. You’re not addressing a technical deficiency. You’re addressing an enthusiasm gap.
And for developers, enthusiasm is crucial! We don’t do good work when we’re bored.

But what if it doesn’t work out after all? What if we run into unforeseen issues with these unfamiliar technologies?

There’s a good chance you will. That’s why you should time-box your experiment with new languages or frameworks. Schedule regular check-ins and retrospectives to ask: are we “getting this”? 
Are we feeling competent and confident? Or are we spinning our wheels on questions about the “right” way to structure things in this unfamiliar stack? 
Are we delivering stories? Or do we keep re-writing the same ones over and over?
When we get stuck, are we finding the help we need?
Are the developers on the team who were dubious of this technology warming to it? Or are they increasingly frustrated?
Set a decision date somewhere between three and six months out. Get together and decide as a team whether to move forward with this new stack, or whether to chalk it up to research and use something you understand better.

It seems like it would be pretty demoralizing to throw all that work away.

It could be. But you’ll probably find that even if you do, the break and the fresh perspectives you’ve explored will give your team some new energy that they can apply to their old familiar stack. And they’ll have a newfound appreciation for just how productive they are with the tools they are comfortable with.