#DecemberAdventure
Low key hackin'
(Inspo 👉 eli.li/december-adventure 💌)
I'm still nose to the grindstone on #452. I'm enjoying it as I do have hope this migration is actually going to work out. I'm not doing much at all today though. I'm mostly busy with my new cat sitting duties and re-watching Star Trek TNG 😌
A few people have chimed in on my fedi posts about Incus. I'm glad people are reading along and taking an interest in Incus. It's a very interesting tool and community which shows a lot of promise. I just found out today that someone is building a "docker-compose for Incus" which is fantastic. I believe this could be a viable migration path off of Docker swarm for Co-op Cloud in the future 🤯 One less corporate part of the stack!
I also took the Incus web UI for a spin. It is super useful and usable already. It's very "fine grained", meaning, you'd need to know Incus before making sense of it. However, for a system adminstrator, it is perfect. It only makes me believe there is less software to be written and more organising to be done, i.e. docs, training, conceptualising, vocabulary, etc. for a community hosting project which proposes an alternative to corporate VPS platforms. It really does seem that a considerable amount of the ingredients are already contained in Incus.
It's a bit of a spoiler but we're ordering a subnet for Varia from our new ISP. The idea is that we'll have an additional 8 IPv4 addresses pointed to the space and under one roof. This allows us to really expand collective server operations with friends, allies and co-conspirators. We've been playing around with a lot of ideas but we do have a name for this project. We're framing it as a "union", taking inspiration from the Dutch (libertarian) socialist tradition, both old and new. We're calling it Vakbond Precaire Netwerken (VPN) 🔥 This naming borrows from the BPW, a tenant union that I've written about here quite a bit which has been using Varia as a home base for several years. VPN is intended as a continuation and expansion of the HBSC "current" within self-hosting practices. More Coming Soon ™
All credit to comrade crunk for sharing this image with me. Source unknown, please lemme know and I'll update it!
Day highlights: Guiness stew, Grand Theft Auto V and the start of Cat Sitting ™ Snuck in a few keypresses for #452.
I'm continuing my work on the same
abra PR in between some house chores, tying up some end of year loose ends and doing the christmas eve shopping for tomorrows cooking. I can never really predict when I'll have
abra hacking spoons but they seem to be in abundance now. I just need to get over the hump of the current morale destroying maintenance problems and then it should be clean sailing.
Somehow, I have spoons to keep working on this abra PR and it seems to be paying off. I'm seeing some promising behaviour from the experimental changes that might fix this bug once and for all. It wouldn't be the first time I've been dissapointed on this issue, so I've asked for some help with testing. The TLDR; is that we have some tricky CLI design on some of the abra commands and not every CLI library is able to handle it 😬 I'll leave that sit until I hear back from the rest. I'm now off to do some house chores and then rest (I have a bit of a nasty flu).
I finally got the courage to look at the abra issue tracker and it ain't pretty. I had previously carried out a large migration assuming it would resolve an age old bug but this was sadly not the case. I have a new (old) idea for the fix and am giving it another shot. It's a great 'ol project but maintenance is really tough these days when the codebase is so large and the problems require large-scale refactors.
I was hacking on the new Varia website for a short while in the morning with other Variaz. We've been working on it for some time and I can say that it is starting to come together. However, the last 10% is always the worst and we're still trying to think through a few things in terms of structure. There's a lot of information that has historically never been on the Varia website (aka "the members list") and we need to see where all of this new information can be laid out, in a simple way. More to come in the new year.
It would appear that the age of random streaming sites (aka fmovies.to and ancestors) is crashing to an end. I don't know why I was reluctant to get back into torrents but I'm finally diving in again tonight. I was pretty quickly able to wire up a torrent terminal client and a Samba share for my local network! Sometimes, just sometimes, *nix Just Works ™ 🤓
Good news also 😌 The comrades without papers won in the courts and have bought themselves more time. We had a good meeting today to plan further mobilisations. The powers that be have not seen the last of the red and black flag in Rotterdam!
I spent some time this evening learning the huh API and what it can actually do. It's not as "plug and play" as I had hoped. I found myself needing to intervene into the settings and configuration of the input and form itself for most use cases. It is pretty fantastic that you can combine it with wish for "SSH apps".
In general, the Charm tools are very shiny and cool but I feel like the code required to get something actually usable working and to maintain it is a pretty large investment. I'm not sure I'm convinced after trying to use these libraries for quite some time. It could just be that building any sort of user interface beyond a command-line is just a huge job in of itself. I'm not sure.
I took another polishing pass on my Caddy plugin. It seems to be working well now for simple use cases and I'm quite happy with it. I've been thinking about local-first pubnix communication possibilities and decided to start (again) where it all began with the one, the only, the mail command 🥁
A bunch of new shiny fixes and features in Cerca landed! I've done some local testing and things are running smoothly. The upgrade and (mini-) migration on the PMC server was over in a flash. It's been really great to see how easy it is to maintain this software, even when it's still in the "bleeding edge" development stage 🙃
I'm happy to report I made it further with my community-hosting galaxy brain. I managed to wire up a new VPS from a $corporate_provider as an experimental testing ground. I was able to build a custom Caddy with my plugin and a DNS provider baked in. Then I configured a wildcard certificate to automagically terminate TLS and route to my Incus instances. I documented some steps on-the-fly on this pad as a sort of "trail of bread crumbs" for how gnarly the setup is. Not bad?
I am again stuck in a bit of a mental block as to what to actually work on next for this project. I'm starting to think the TUI is a dead end in terms of the maintenance it will require. A command-line might be nice but then Incus already provides this? $corporate_providers are specifically set up to obscure the actual material resources and underlying hardware and I want to do the opposite. So, what level of abstraction is appropriate? I'm just gonna try to do stuff manually right now and see where the rough edges are. So far, my setup could already work as a nice little pubnix with lightweight virtualisation for its users.
I was mostly squashing life-admin in the morning and then was visting a friend in the hospital in the evening. I spent a large chunk of the late evening drinking a duvel beer, listening to de Internationale (slowly learning the Dutch lyrics) and inventing names for a hopefully soon-to-be-real system administration union.
It was a really wild universe day and nothing went to plan. I hid at home. No-op.
I was in Amsterdam for the day, helping out with the Varia bookstall at a Rijksacademie event. I did zip off for my ritual visit to Het Fort van Sjakoo for some print anarchism. It's almost like a religious pilgrimage at this point. I cannot visit this city without going there. It was nice to see a lot of familiar faces today and I enjoyed the collective publishing discussion at the end of the day. Yes, you guessed it, an adventure no-op day. This log is taking on the character of a life diary more than a code diary 🙃
I helped with a facilitation training at Varia after lunch. Everyone decided to organise their end of year social events this Friday, so I was busy making gluehwein and hopping between two gatherings in the neighbourhood. So, another beautiful adventure no-op. Happy 1312 folx!
I finally have a calm day 😌 I wanted to dive into finishing this Caddy/Incus plugin and after a few hours, the first prototype is done 🎉 I got a lot of satisfaction from this and learnt a lot about Caddy internals and Incus API bindings. I haven't tested it extensively but please give it a shot if you're feeling brave. The source code is here. I made sure to write some docs. I also posted on the Incus discussion forum.
I think my next step would be to test it with On-Demand TLS on a server on the public internet. I'm going to revist my community-hosting plans now that I have this new piece of the puzzle. The idea is that once users create new instances and configure domains for them, they will just automagically get HTTPS and routing without any further intervention. If I can simplify the user-facing TUI setup and add some more functionality, I think I could launch a little YOLO offering on an invite-only basis to see how things go.
The main doubt I have now is having the TUI accessed from the server and not from localhost. I am going to look into what authentication Incus can offer and how that might work out in from a system administrator perspective. I am also not sure if the whole fullscreen TUI experience is required.
I am back on the horse. Kind of. I managed to take a flying pass on the invite work of @cblgh on Cerca. It's really shaping up to be an excellent change set which will make adding new users much easier to manage. I am hoping to deploy these changes soon once things get merged.
No-op++ 🙃 I had hoped for a calmer week but it just isn't. I visited a potential new studio today and hosted a housing union training in the neighbourhood. I'm realising for the 4th year in a row that winter is just housing union go time. Landlords and politicians LOVE to take advantage of this period.
Moar glorious adventure no-op. I had to prepare and facilitate a large-ish meeting in the evening, so I spent most of the day preparing for that after taking care of various life-admin stuff. The meeting went well. We are building a coalition with several other groups in the city to fight awful policy changes which will put asylum seekers who get their application rejected (unjustly, most of the time) out on the street during winter.
The assad regime has fallen! A historic day for all Syrians. Besides watching everyday people pull down statues and celebrating in the streets, I did manage to write a few lines of code. This did help me get back up to speed on what is going down in Syria. I think I've mentally got into the model of writing a Caddy plugin. I've managed to translate the Docker-isms to Incus-isms (labels 👉 config settings). The code is starting to come together 👀 There are still a lot of unknowns up in the air but I think they'll iron themselves out once I get manually testing. I think I should be able to get something up and running in the next days 🙏
Another adventure no-op. A day a rest has been observed.
I spent most of the day today Varia infra hacking with crunk. We did a thing. Happy to see this week through 😌 Two galaxy brains from today: 1) a class struggle aware variant of SQL, playing around with the language keywords. For example, instead of "CREATE", "GRASSROOTS" and no "DELETE" only "OVERTHROW" 2) Purchasing a residential subnet range from our future ISP for some anti-data center revolutionary horizons. Hoping for a few hours this weekend to dip further into the adventure. We've been blasting intergalactic.fm for the evening 🖤 Here's a sneak preview of the new Varia SSO login theme 🙃
I don't have much energy tonight. Started off with gardening a couple of issues on the Co-op Cloud tracker. This was a little follow-up to the Kite flying hour that I sadly missed (pretty good excuse tho, mobilising with the housing union at the courthouse!). The weekly kite flying is starting up again, so I'm aiming to be there next week. I did manage to make a start on the Incus automagic Caddy router thing here (mentioned here). I think I'll try to ride this one out as I'm quite excited to learn more about the internals of how Caddy works. I feel like there is a lot to learn from this project in terms of engineering ideas but also how to get pretty "advanced" stuff done with Go.
Today was an adventure no-op. It's just a wild end of year week but things calm down next week. Looking forward to some hackin' while listening to Moondog (back on a big Moondog buzz since earlier today). Amongst other nice things, we had our final member meeting at Varia and celebrated the new funding for the upcoming 2 year programme.
Full schedule again today and didn't have much time to adventure. I did have a great housing union meeting though ❤️🔥 As a follow up to the 01.12.24 entry, I was glad to see some movement on the issue about warawara not being able to connect to the lurk.org XMPP server using a TLS config. Much thanks for @SamWhited for picking this up, recommending a work-around and getting a patch out, super cool! The fix on our side was pretty simple. I've managed to build another static binary and let it rip on the server. I'll check in again on it in a few days 😌
No rest for the wicked, as they say. A serious amount of life-admin and comms back & forth really took me out today, but I still had a few cycles to think about what I want to focus on this month.
I should “probably” be working on abra which hasn’t seen a release in a while and is more or less stalled in development pending more spoons. It just feels a little bit too much like “work” (although at the moment, that would be unpaid work until we figure out the budget again 🙃). So, yeh, not picking that up just yet.
I was keen to revisit a galaxy brain I had about a Capsul inspired system for community-hosting. I had done some prototyping around this a few months back and it was starting to take shape.
The idea in short is to find a “3rd way” between with specialised hardware in data centers (inaccessible and expensive) and self-hosting (too much “self”). On the one hand, people who know how to interact with the bureaucracy of a data center, get a suitable server with virtualisation support, configure it and foot the bill etc. are quite rare. On the other, people who can self-host are less rare but are unlikely to let others they don’t trust a lot have shell access to their homebrew server.
Since I’d rather get out of the data centers, I’m wondering about all those single board computers, old laptops and 3rd hand machines people have set up. So many people are still renting VPSs from Big Tech and meanwhile, there’s a lot of unused compute at home. Can we have some of the good aspects of the "VPS culture": private virtual spaces and simple graphical interfaces for maintenance? Can we transform aspects of self-hosting into community-hosting: non-specialised hardware of all kinds, (invite-only?) multi-tenant and low maintenance?
I first thought “Docker”. However, there is too much sharing of the host system and the container. Breaking out of containers is sometimes trivial. I did discover Incus which is actually pretty fantastic. It has the concept of "system containers" which provide more isolation but are still lightweight.
I don’t have it all worked out yet, but I do have a mini-demo. It shows an initial prototype of a “self-service” TUI where you can create a system container and access it like a VPS (running on my homebrew machine). It can mutualise the same machine and searches for a free port to proxy from the host to the container. It isolates this container from other users on the same server using an isolated project. It also takes care to put the SSH key into the container via cloud-init. The host system uses the command=... <ssh-key> trick in the ~/.ssh/authorized_keys file to limit SSH access to run the TUI.
Incus has pretty excellent security considerations, putting measures in place to limit the damage of container break outs and using sockets and user permissions in a way that reduce the blast radius of malicious users. I’ve been quite surprised by how much care is taken by the maintainer. I’ve even got a patch into the core of Incus! The maintainer seems really cool 🎉
The only missing piece for my prototype is a way to serve HTTP/S to these containers. I can imagine a scenario where people use the “self-service” TUI to create and register domain names against the containers. How will the system pick up these new containers and route requests to them?
Since my prototype already has 3 moving parts (Incus, the TUI, the intermediate service (incus-admin proxy to avoid users having root access)), I’m not very keen on adding another one. However, it seems unavoidable. I believe I have a good idea though: I can write a Caddy plugin to automatically read the labels of Incus containers and use On-Demand TLS.
So, tonight I was reading up about Caddy plugins and I believe I just need to adjust and re-work this one to work with Incus labels. The code doesn’t look too gnarly. I could then manually test firing up a customised caddy + plugin alongside a few Incus containers running stuff on port 80 and hopefully see requests get automagically routed with proxy terminated HTTPS.
There’s a lot still up in the air but I think this could be useful at some point. It doesn’t cover every community-hosting use case. It can cover some simple ones though. I’m not really sure about the setup I have now and would like to see many things simplified.
We can imagine groups/communities who just want some “on-demand” VPS-like thing to learn, host static files, run some basic web services etc. and one of them has a SBC and is willing put it to good use. If the system can help reduce the maintenance tasks of the homebrew sysadmin and supports collective approaches to maintenance, it could be nice to see where this goes.
It's been a hectic few days! All good things, but still a lot. Today was mostly spent talking to neighbours, comrades and accomplices at Solidarity Network(ing). Great chats 💖
Once home and rested (aka "face down on couch"), the urge to kick off this adventure soon arrived! With the refactor in cerca (mentioned below) rounded off with help from cblgh (🤘) and a large piece of work underway to enable invites, I'm temporarily dropping tools on cerca hacking. There's plenty of new things to test out and I don't want to disrupt the invite hacking flow 🌪
One of the cerca-adjacent things that I wanted to take a look at is our cerca XMPP notification bot, warawara (written by ugrnm (⚡)). warawara seems to be losing its connection to the server when up and running for a while. Also, I discovered that it seems to do some extra notifications on restart. So, I sprinkled some commits and will check in on the logs tomorrow.
It's not December yet, of course. However, I was at Calm Coding this evening and I feel like the idea of December Adventure has much overlap. So, here's a little sprinkle of pre-december-adventure hype, let's say. With the permacomputing.net gang, we've recently started to test out an instance of cerca and I've been hacking on various parts of it in response to discussions and needs. It's been quite a nice dynamic so far, bringing those discussions to the cerca issue tracker and then balancing that with upstream concerns. One of the things we ran into on deployment was how the sysadmin functionality was split into separate binaries and this was a bit inconvenient when setting things up. So, here's the changeset to merge all the things together.
Back