Enable nginx debug logs on an IP

The other day, I had a weird situation on an nginx server.

I read a little, thought a little, but I didn’t figure it out. I thought about turning up logging, but I was concerned about the increase in disk IO changing the issue. If only there were a way to turn on debug logging just for an IP…

There is, and it’s called debug_connection! (I was able to figure the issue out in just a few minutes.)


Follow changes in a file on Github

Sometimes, I want to watch for changes in a file on Github.

A CODEOWNERS file works OK to alert someone of pull requests (upcoming changes), but it isn’t really an option for folks who aren’t part of the development team.

I found out recently that you can do this via Github’s RSS/Atom feeds!

To watch the file README.rst on the main branch of, use the RSS feed:

Thanks, Jake!

I would love for a way to extend this and be alerted to public pull requests that affect that file. Let me know if you have any ideas!


Search and parse a field with Awk

I frequently need to do extremely simple parsing. Often I resort to grep and cut, or a regular expression I send through Python or perl. Today, I learned how to use awk for this.

awk '-F: ' '/Configuration backup archive complete/{print $2}' will look for a line with “Configuration backup archive complete”, split the line up by colons, and print the second field.

Tech Thinking hard, or hardly thinking?

Two-Hand Philosophy

Monday morning! Time to get to work!

This morning, I put on my morning playlist to find my inspirational message from The Rock himself, Mr. Dwayne Johnson, was missing.

I notice that the five tracks from “Dwayne Johnson Workout” were missing from my playlist on my computer. These are 30-second clips of Dwayne Johnson telling you how hard work is worth the effort. It’s 2020, folks. I listen to these multiple times every single day.

I pulled these from a bigger playlist, “Dwayne Johnson’s Progress Workout Playlist“, with his motivational spoken tracks and some POWERFUL ROCK MUSIC TO KEEP YOU GOING. I click through, and they are all greyed out and unplayable in the original playlist too. Harrumph.

I knew these worked yesterday! My stomach drops. I had a feeling I had to act quickly.

I grab my phone, put it in in Airplane Mode instantly, trying to catch it before it realizes the tracks have been removed. As silly as it seems, to be faster than a computer, I have a chance here. Android pauses nearly all apps when it thinks your phone is sitting on a desk, not plugged in. (They call it Doze.) I had also marked the whole playlist for offline playing. Spotify says they give you ~30 days in offline mode before you have to go back online to reauthorize things. Maybe I can finagle a way to keep these tracks, so The Rock can help me get my work done and tell me I’m doing a great job.

After confirming I was in Airplane Mode, I open up Spotify. 4 of the 5 tracks are playable. The first, however, was greyed out and unavailable. :/ 4 outta 5 ain’t bad, folks. We can do this!

I tweet about the tracks going missing, and do a little research. They came out about a month ago, as part of a promotion Dwayne Johnson is doing. They’re on Spotify under “Seven Bucks Productions“, which was founded by (and still CEO’ed by) Dwayne Johnson. It doesn’t look like they’re available anywhere else, and I can’t buy them.

Before I dig in, I should confirm that there isn’t a weird glitch with my account or something. I jump into Spotify support chat. They quickly confirm that these tracks were no longer available in any region, so I rule out VPNing to a different country as a helpful component of my recovery strategy. The support person won’t tell me any reason why the tracks were no longer available, if they were always going to be pulled today, or if they were coming back.

OK, so let’s say there’s some sort of glitch and Spotify thinks the tracks violate some policy, or some intern somewhere marked them as a violation and another company automatically pulled them, or some other wishy-washy excuse. How can I get these tracks back?!

The first thing most folks will think of is copying the files. I knew that the Spotify cache was encrypted. This means they had gone out of their way to protect it. I am not a full-time security engineer, but I have done more than my fair share of security work. While there are likely issues that someone who could really devote some time could dig into, I’m not particularly interested in trying the front door, if I can see a bunch of bars and locks and guards from across the street.

This is voice, not a huge orchestral recording. I should be able to get the audio in a roundabout way and still have a decent recording. I look at the Android API docs to see if apps can record other app’s audio. They can, as of Android 10! Apps can block this in their manifest, but let’s give it a shot. Screen Recorder is free, has no ads, and uses these APIs. I install it, turn up the audio quality settings, and set it to record internal audio. I start the recording, open Spotify, and play one of the aforementioned tracks. I stop the recording, play it back, and… bupkis. No audio. Looks like Spotify blocks the recording in its manifest.

Android 11 comes with a screen recorder, now! Maybe because it’s part of the system, it has different permissions and can record this audio! I add the Quick Setting tile, set it to record audio from internal sources, and… nothing. Harrumph.

OK, what about analog? Some fellow nerds call this “the a(nalog) hole“. The Pixel 2 doesn’t have a headphone jack, but I have a USB-C headphone jack adapter. I could run a 3.5mm patch cable to my laptop, grab the audio, and have *something!*

After connecting it up, I jump into Audio settings on my Macbook to switch the headphone jack to an input. Hmm, where’d the “use this jack for audio input” box go? Oh, now you can only get audio input using TRRS cables? I don’t have any way to make the audio from my phone output that way. :/ I have some TRRS adapters at my workshop (and USB audio dongles that have mic inputs) but it’s rapidly approaching Monday afternoon and I haven’t yet gotten started with my “work”.

What about decompiling the APK, modifying the manifest to enable recording, signing it, and putting it back on my phone? I’ve done deeper APK modifications in the past with some of my AOSP work. I’m not a pro-level Android APK reverse engineer, but this shouldn’t be a long job. As I think about it, however, I realize that I can’t figure out any way to do this while preserving the offline file. As far as I know, I’d have to remove the current APK before loading my modified APK onto it.

What about something even more esoteric? The avenues are limited. My phone isn’t rooted and I don’t have this song on any Spotify installations on any emulators. I take a quick look, anyway, and folks report that Spotify has a fair amount of anti-piracy/anti-reverse engineering/anti-root stuff built-in. Fine, fine.

What about through a system-level debug utility? I do know my way around AOSP documentation and I’ve set that stuff up before… I quickly find:

“For privacy the tee sink is disabled by default, at both compile-time and run-time. To use the tee sink, you will need to enable it by re-compiling, and also by setting a property. Be sure to disable this feature after you are done debugging; the tee sink should not be left enabled in production builds.”

AOSP Documentation, Audio Debugging

Alright. I take stock of my options. I think my best bet is to figure out how to get the audio out of my phone on a 3.5mm cable and record it. Many of my computers, my Macbook, even the ones running Linux, (WHO NEEDS FREEDOM IF IT CAN’T HELP THE ROCK TELL YOU THAT HE APPRECIATES YOUR HARD WORK?!?!) have combined headphone/mic jacks. After some reading, it seems that many of those have the ADC/DAC TRRS muxing done automatically by some intermediate circuitry, rather than under the direction of the operating system. I do have a computer that has a split headphone/mic jack, but my 6YO uses it all day for “distance learning”.

Rather than interrupt my son’s education (for this increasingly lengthy endeavor) for twenty minutes as I get levels set and these tracks backed up, I decide to put this on pause. It’s REDACTED o’clock on Monday morning, and I haven’t billed a single minute yet. I’ll get to work, two-hand philosophy, without Dwayne Johnson in my ears, and after my oldest is done with school, after I get some work done, I’ll make a copy of these tracks. My phone’s in Airplane Mode; I’ve got nothing to worry about. If this all fails, I have other ways of recording 3.5mm audio at my workshop, like USB audio dongles, Teensy Audio Shields, a variety of dev boards…

Before I really get to to work, I post some of this to a few places, to see if any smart folks had any other ideas. Some recommended keeping this phone in Airplane Mode forever, turning it into a Dwayne Johnson shrine. (Spotify gives you a 30-day timer when you go into offline mode, so once they learned about that, it morphed into requiring a software-defined cellular base station without internet access, with a modified clock.) Other folks recommended setting up an offline wifi network, joining my phone to it along with another computer that I’m also logged into in Spotify, seeing if I could play the track through the computer from my phone, and then using some fancy audio routing software to record it on the computer. I think that’s an amazing idea! I would want to try that *after* all else had failed, because I guess it’s possible Spotify could share “media no longer available” lists even offline, and I didn’t have any computers logged into my Spotify account that didn’t already know that those tracks shouldn’t be available.

Alright, end of the day! Let’s do this! I take my phone, still in Airplane Mode, over to my son’s computer, connect up the USB-C to 3.5mm adapter and the 3.5mm patch cable. I don’t think I’m exactly supposed to blast headphone-driving outputs into this mic input, electrically speaking, so I play a random track (the Phineas and Ferb theme, actually) while looking at the levels to get the volumes right. l use arecord to grab some audio, then play it back. My brain knows that it’s a photocopy of a photocopy of a photocopy, but it sounds fine to my Old Man Ears (and I’m trying hard to be optimistic). IT’S GO TIME, BABY!

I switch over to the list of the five Dwayne Johnson motivational tracks, and… What? My face falls. They’re all marked unavailable. How? They worked REDACTED hours ago, and it’s been in Airplane Mode since the morning!

Did my phone take itself out of Airplane Mode? Did Spotify see I was trying to get at its music and go into lockdown? Did the tracks have some sort of deadline metadata built into them? I know that, in general, you can put Spotify in offline mode and play music for days and days. It says up to a month–I’ve never tested that, but I’ve used it over long weekends without internet access. This Dwayne Johnson catastrophe happened in the span of a day, so I truly have no idea what happened.

One favorite–my mom.

Attack Surface by Cory Doctorow

Attack Surface, Cory Doctorow’s newest novel in the Little Brotherverse, is excellent. Like the previous novels, it has nerds fighting for human rights, it has protests and activism, it has open-source and soldering irons. In comparison, however, it now has expense accounts, workplace intrigue, and added nuance. It shows us while technology can help organize political power to fight oppression, it can be quickly adopted by the oppressors themselves–a marked growth of this series’ evolving thesis. If you enjoyed Little Brother or Homeland, you don’t really need to read any further–you’ll love it, and appreciate the increased insight that comes with another decade of Doctorow’s focused thinking on the intersection of technology, freedom, and politics. I was unable to put it down, and read it in a single sitting, until the early hours of the morning. If you haven’t read the previous books or thought Marcus seemed naive, caught up in his own technonavel, give Attack Surface a shot.

While the previous two books were from the point of view of Marcus, Attack Surface is about Masha. She signed up with the Department of Homeland Security to help fight terrorism, after the terror attack from the start of Little Brother. Something like a decade later, she’s bounced around a few places. The start of Attack Surface has her jet setting around the globe, working for a cybersecurity company, installing software for authoritarian, totalitarian, and fascist governments by day, and helping the very activists she’s helping target at night. This is not a stable situation. Before long, she’s looking for a new job. She ends up back in the Bay and stumbles upon another government operation violating civil rights.

I loved Masha as the viewpoint character. Masha is certainly an adult, and, while she has some personal blind spots, she’s full of self-insight. I enjoyed getting to walk through her past, and I liked seeing old favorites from previous books. Although no longer nemeses, Masha still serves as a way to see Marcus from a different angle, enriching the previous works.

Doctorow still delights in didactic description–for instance, you’ll read about how governments analyze social networks, what to avoid and what to strive for in a protest, and I hope you come out of it realizing that you definitely still can’t trust your pocket camera microphone with its unauditable baseband processor.

When thinking about technology from a political angle, it is very easy to get wrapped up in the tech. After learning how to use some math and engineering, it’s easy to confuse playing super spy with these things as fighting for freedom, as making the world a better place. It’s equally as easy to get overwhelmed and apathetic because you can dream up a scenario where any particular thing could be broken, so why try to do anything at all? Attack Surface does an excellent job of helping the reader keep this balance.

Honestly, I don’t think you need to read Little Brother and Homeland to enjoy Attack Surface, but if you haven’t read them and are looking to read them spoiler-free, stop reading here. SPOILERS FOR LITTLE BROTHER AND HOMELAND FOLLOW.

Little Brother (2008): Marcus, the main character, and a few of his friends are arrested and detained by the Department of Homeland Security after a terrorist attack in San Francisco. After witnessing some government abuses of power, Marcus and his friends use commodity hardware, strong encryption, open-source software, and The Power of Friendship to “fight back against the surveillance state”. It ends after Marcus convinces a journalist, who writes about some shady things the DHS is doing, and the State Patrol comes in to shut the DHS’s operation down.

Even though Doctorow brings the reader to a boil showing that “it could happen here”, Little Brother is a relatively fun read. There’s teenage romance, of course. Doctorow drips with delights in his didactic sections, instructions showcasing various technologies and techniques that can be used to increase “opsec” or to bypass surveillance and tactics used by oppressive regimes.

Homeland (2013): Set a few years later, California’s economy is in rough shape. Marcus is working as a web developer for an idealistic politician. Marcus gets a thumb drive from Masha, a character we know from Little Brother. It’s full of secret documents detailing government abuses from around the world, and Masha tells Marcus to leak the documents if she goes missing. Sure enough, she does, and Marcus has a dilemma. If he releases the documents publicly, his boss won’t be elected, but there’s no guarantee that just dropping the documents on the internet will actually help anything.

Between Little Brother and Homeland was Occupy Wall Street, that campus police officer walking down the line of students spraying them with military pepper spray from a foot away, the Arab Spring…

Homeland expands on the idea that fancy tools and techniques can be used to organize mass movements to hold the government accountable while adding a bunch of “things will get complicated real fast, and many things won’t seem black-and-white.”

Attack Surface is a welcome addition to the Little Brotherverse. It’s upbeat, without being naively optimistic. It avoids cynicism and apathy while acknowledging these as common responses to “the current political realities”. Experiencing an adventure from Masha’s perspective is an enlightening change from riding along with Marcus. The added depth and perspective expand not only this work but also the previous ones. I definitely recommend it!

(I received an advanced copy of this book in exchange for an honest review. A friendly person over at the publisher actually asked if I would review it, which is the first time that’s ever happened!)


Milling Planetary Position Jewelry

For Valentine’s Day 2020, I made my wife a necklace with three CNC-milled charms, each with a representation of the positions of the planets on specific important days: the day we started dating, and the birthdays of our two children.

Generating Artwork

First, I developed some Python code to generate the images I needed. Since it was graphical and I knew I’d have to play around, I created a Jupyter notebook. (You can play around with my code, too! Change the date and make your own with Binder or Colab!)

You can click through on the Python, but I use sunpy to get coordinates of the planets at a specific time.

After I played around with the data, I created a ray for each planet, starting from the sun, aimed at the planet.

I then saved an SVG for each of the dates for import into my CAD/CAM tool.

CAD/CAM; or “transforming the artwork to something millable”

Once I had the SVG, I created a Fusion 360 project. I made a sketch with three circles, imported the SVGS, and added little holes for the jump rings. An extrusion later, and tada!

The charms in Fusion 360

In the Fusion 360 CAM view, I set up a trace operation with an engraving bit, and then a 2D Pocket and 2D Contour operation with a 1/16″ flat end mill.


I loaded up an anodized aluminum wallet card (which also lasers really really well, by the way!) into my Othermill V2, and a few minutes later, the charms were ready.

The mill right before it engraved the line art into the anodized aluminum card.
I took a video of the mill cutting the charms out.

After loading the first set onto jump rings and then onto a necklace chain, the charms seemed too big, so I went back to Fusion 360 and shrunk the charms down, recut, and a few minutes later, I was finished.

The larger charms, before having their edges buffed.


I think this project turned out quite well. Using a Jupyter Notebook was a huge win, although PyCharm in late 2019 tends to freeze when using it. I really like that using Colab or Binder, folks can open a webpage and use my code–without me having to think about it being a web service when rigging code together. I love laser cutters and 3D printers, but sometimes milling is such a perfect fit for a project.

(This project (1, 2) is also on Instagram.)


Laser-cut “Jigsaw” Puzzle

My son had a Kindergarten assignment–to bring in 100 of something, for a celebration of 100 days of school.

The finished puzzle
“100th day 2-20-20” (I guess I’ll have to teach him about ISO 8601.)

He decided he’d make a puzzle. I took him to my workshop to laser-cut a 100 piece puzzle out of 1/8th inch Baltic birch.

He was incredibly patient and simply delighted by how awesome laser cutting is.

We used an online jigsaw puzzle generator to make the design. It worked well enough, but the pieces weren’t really distinct enough with the settings that I used.

It was really fast!

Next time, I would tape the puzzle together after it had been cut. Solving a blank two-sided puzzle of “wood grain” was not fast!

After we solved it, my son colored it with markers. It turned out great!

(This (1, 2) is on Instagram, too, if that’s your thing.)

Projects Thinking hard, or hardly thinking?

“Art in Space”

On a cold, dark, Minnesota November evening, after what was, for me, a long week, and for the students, months of work, we shipped Blue Origin two payloads: art, for space.

In November 2019, I was contacted by the kind folks at Playful Learning Lab.

They had conducted a contest with OK Go where middle schoolers and high schoolers could dream up some art that could only happen in space. Two teams were selected, one focused on paint splatters and music, and the other focused on suspended magnets and whirling particle vortexes. The undergraduate team at the Playful Learning Lab worked with the winning student teams and also the engineers at Blue Origin.

Once these payloads were created, they’d be launched on Blue Origin’s New Shepard for an 11-minute suborbital flight into space. Actually, that’s not true! As you may suspect, putting things into space tends to have strict deadlines (and mass requirements!), and the team needed some help getting everything ready in time.

They knew I was handy with microcontrollers and electronics and making solid one-off interactives and worked well under pressure, and wanted to know if I was interested.

WAS I INTERESTED?! Rewind a few years to an interview of mine by Mark Fraunfelder.

Mark asked me what sorts of things I dream of making, and I spouted off a few things, and ended with "I'd love to build something that goes into space."

Long story short, it was like a heist movie and Apollo 13 combined. There was the “gathering the team” intro. I brought in folks I’ve spent a decade or more working with, like Matthew Beckler (with whom I have a set of informal Hardy-Littlewood rules) and some of the folks from Kidzibits (who are great at thinking creatively while creating robust, rock-solid fabrications).

We spent a week or so of long days and nights working with a truly extraordinary team of engineering undergraduate students, covering massive whiteboards with diagrams and frequently dumping everything we had onto a table and saying “We gotta remove off 100 grams from this subassembly.” (100 grams here, 100 grams there, when you only have 500 grams per payload, it really adds up!)

474 grams, or 26 grams under the limit! (Photo credit: Maria Baklund)

There were two projects. Cosmic Song had a paint chamber with sticky paper and bright powder pigment, and what I can’t describe any other way than “a space guitar”, with three strings strung across some aluminum, struck by little solenoids. They were driven in a random way, driven by a procedural combination of the telemetry data from the rocket and a little chamber with a proximity sensor and a ball in it. Once it reached space, the powder chambers opened, the powder vibrated out of the boxes and onto the sticky paper, while the space guitar’s strings get tapped by solenoids. Dark Origin had a series of magnets suspended in the middle of the payload, various types of magnetic debris in chambers, and powerful fans blowing a whirlwind. When it reached space, the chambers opened, the fans blew, and the particles swirled and interacted and coalesced upon the strong suspended magnets.

Art in space.

The “space guitar” (Photo credit: Maria Baklund)

(This showed up on Instagram, but I didn’t want to spill any beans.)


“Radicalized” by Cory Doctorow

Radicalized, by Cory Doctorow, is a collection of four independent stories set just a moment in the future.  If you want four Black Mirror episodes filtered through Boing Boing, you’ll absolutely love Radicalized.

There’s an intensity, a purity in these stories, that really keeps you reading.   They are moral, they’re cutting, they’re biting.  They’re about otherness and othering.  These are stories written by an immigrant, the son of an asylum-seeker, and someone who has spent his life fighting for our digital rights. They’re not going to make you feel good.

The first story, Unauthorized Bread, is about a refugee who gets a spot on one of the “poor floors” of a posh high-rise, living in a Silicon Valley dream Internet of Crap dystopia, who learns to jailbreak her things.

There’s a story, Model Minority, about a Superman-alike who witnesses some senseless police brutality, steps in, and has to deal with the consequences.  Superman realizes Black Lives Matter but then struggles because he can’t punch police brutality in the face, and learns how quickly otherness can happen.  I loved the interplay between the Superman-alike and the Batman-alike.

The third story, Radicalized, was quite difficult for me to read.  It’s about a man who joins an online support group for folks dealing with terminal cancer in their loved ones, and their health insurances refuse to authorize their treatments.  The support group becomes more and more extreme, and he just can’t tear himself away.  It’s easy, too easy, to dismiss angry alt-right 20 somethings in chat rooms, but what’s it look like when that same rage is focused on health insurers?

The fourth story, Masque of the Red Death, is from the point of view of a rich financial trader, Martin, who has created his own “Fort Doom” and picked thirty lucky folks who will shelter out the apocalypse with him.  Doctorow has talked a few times about the choices we have when “it hits the fan”.  When your neighbor comes over for help, do you work together, or do you point a gun at him?  We’ve seen what working together looks like in Walkaway.  Doctorow shows his increasing skill in this story with point of view and word choice. Every moment we see the world from outside of Martin’s POV is simultaneously great and heartbreaking.

I know there will be a lot of people turned off by this book, who will get a whiff of it, feel feelings for things that they don’t want to have, and dismiss it as “propaganda”.  Midway through the first story, Unauthorized Bread, I had a different worry—are these all going to be Electronic Frontier Foundation fiction think pieces? I *love* the EFF, don’t get me wrong, but I do not need to read a novel yelling at me about the evils of DMCA and DRM and the Internet of Crap.  However, these worries were unfounded. Radicalized quickly digresses into dark glimpses of the world we’re creating for ourselves.

I received this book at no cost from NetGalley in exchange for an honest review.


How Not to Hate Your Husband After Kids by Jancee Dunn


I really enjoyed this book, and I’m putting much of it to good use. It’s a relatively quick read. In each section, she introduces a problem, tells some stories, and then talks to different experts before breaking it down into advice.

I’ve been recommending to basically every other dad I talk to.