History All the Way Down

Transcript

Martin Wimpress 0:00
A few episodes ago, I talked about micro being a modern Unix text editor. But I made a bit of a faux pas because I started my modern Unix adventure in the middle of the alphabet. So today in modern Unix, A is for action, I think is how it’s pronounced.

Mark Johnson 0:21
How’d you spell that? Like the title out of Discworld?

Martin Wimpress 0:24
Is it? So it’s spelled at u i n? Yeah. This means something to you. Does it work

Mark Johnson 0:29
in the Discworld series of books? That world is a flat disc that rise on a turtle called Great achoo in?

Martin Wimpress 0:36
Ah,

well, okay, then this is enlightening. So the action I am referring to is in another sequence of segments, a magical shell history.

Alan Pope 0:50
What are you using at the moment? I know in the past, you’ve used different shells, the US Fish and all kinds of stuff. Is that what you’re using at the moment?

Martin Wimpress 0:58
I have been using fish for about 10 or 12 years now? Yeah, so I’m still using fish.

Alan Pope 1:03
So that’s the letter F sorted.

Martin Wimpress 1:07
So yes, so actually in is a magical shell history supercharger. And it supports bash, and fish, and Zed sh. And for the cool kids in town, it supports new shell as well. Now action has a number of cool features. And I’m just going to run down a few of those that I have found particularly useful. The most useful, in my opinion, is the fact that action can be installed on all of your workstations. And it can backup and synchronise your shell history into one place. So now your shell history is the same across terminals, sessions, and your machines. And you can query your shell history from Forever.

Alan Pope 2:04
I’m not sure I like this. This sounds like voodoo and I don’t like it.

Mark Johnson 2:08
Does it perform any sort of segregation of that history? So you can say, Oh, I know I did that on this machine. And I just want to look through the history of this machine. Or, you know, I know I did feel that machine, always it just all of it, whatever order it happened in is in a timeline.

Martin Wimpress 2:23
So yes, it stores lots of additional context around the commands that you execute. So each time you run a command, it stores the exit code, the current working directory, the hostname of the machine, it was on the session it was in, and how long that command took to execute. All of those things are now available to you in the search filters. So not only can you bring it up a bit like your shell history, I’ll explain more about how this works. And sort of scroll through it and type a query. And you can actually run queries from the command line to say, give me everything on this host where I ran make yesterday. So you just see those make commands, because you know, you did something that you’re trying to recall. So it’s got very rich sort of query language for how you can inspect your history.

Alan Pope 3:15
Okay, I’m sure it’s wonderful. I have a mental model where I know where things will run. And so the thing that freaks me out is the fact that it’s aggregating. Obviously, this is optional, but it’s aggregating the shell history from lots of machines. I know, this particular machine, I do this thing on, and one of them is a Mac. And so some of the command like things that I do, there will definitely not work on anything else.

Martin Wimpress 3:38
Right. So I’m delighted to tell you, you won’t have a problem with this. Because I felt the same way about this. So it doesn’t interfere with your existing shell history. So the way that I actually use action is like this, the up arrow for my shell continues to invoke the shell history from my shell, in the usual way that either bash or fish or whatever does it but Ctrl R, I bind to action. And that gives me the full screen search. So when I’m sat here at home, and I know yesterday, I ran something on my laptop or my other workstation. And I want to recall that I can now control that and search my history of all of the things and recall that item. And as you type your query, it narrows down your search and it puts one to nine next to the current line. So it’s a bit like launching a hotkey. So anything that’s numbered you can alt number in order to launch that without actually having to scroll all the way up to to hit it is absolutely fabulous.

Mark Johnson 4:50
And how does this actually work? Is it is there like some demon set in the background monitoring all of your shells or is your shell doing something after it command to tell achoo in record this,

Martin Wimpress 5:03
you can do both. I believe I’ve got it running as a daemon and I’m running an actual wind server. So the project provides a server that you can use a public Lea available server, you sign up for an account ng generate an encryption key, and your stuff goes up there, or you can run your own. So actually, it has its own service. So I’m, I’m running a server here. And I also have some of my history on their public servers, because I use that to sort of get started with it all. And one of the things you can do is set an interval for how often the synchronisation happens. So automatically in the background, my command history synchronises, and it’s a two way sync every hour. And you can import your existing histories. So the first time you come to actually when you can seed it with whatever history you have available. So when I did this, because I to a data hoarder, I have a shell history that goes back 15 years. And I do prune it from time to time I attended like a beautiful garden. So I gave it another round of cleanup, and I imported that. So I think I have around 40,000 distinct commands in my history, which are now query of all by attribute and where ever I may go.

Alan Pope 6:27
So there’s a popular thing people do, where they run history, and then pipe it through a bunch of machinery in order to find out what they’re like top 10 Most run commands. And for everyone, it seems, if you’re a software developer I often get is like one of them. But for most normal people, it seems like CD and Ls are the two most popular commands that anyone ever runs on a on a Linux system is does it have a way of exposing those kind of like popular commands that

Martin Wimpress 6:54
you use, it has a built in statistics generation. And so yes, you can just run stats and get a breakdown of everything imaginable, including a Git style sort of heat map of which days, your most distinctive new commands were executed. Like this. So it’s a bit of fun, but it’s extremely useful. And now I no longer backup my shell history. In my dot file managers, I have this database, which is always accessible to all of my machines, and then a discreet history on each machine. Because the beauty is, is that if I start with an empty history, but have just synced from action, I can query actually and run the command. And that’s know my local history as well.

Alan Pope 7:43
That’s pretty cool maps give that a go. I’m one of those tedious people who has an Eevee. And as such, I am compelled to talk about it all the time. Yeah. And some questions come up quite a lot. When people who don’t have an Eevee are interacting with someone who does have one. And I’ve had mine, it’ll be two years in December of this year. So I’m a relative newbie, if you think of all the people who have Nissan LEAFs, og, Tesla and stuff. But I wanted to have answers to some of the questions people often ask. And I’d be hand waved some of it. You know, people always ask about range and charging, and you know, what the charging infrastructure is like, and all that kind of stuff. And often I hand wave some of these questions away with, like, really answers. You know, it depends. And you know, what’s charging? Like? It depends. Because it does, it depends on where you are, and you know, how you’re charging and all that kind of stuff. And I wanted to be more data driven. And I wanted to get data from my car to be able to answer some of these questions with facts rather than hand wavy answers. So what I did was I got some data, and I put it into my employers platform. axiom is my employer. And I wanted to put the data in there because it’s got tools for building dashboards and a great query language for rummaging around in that data. So that was, that was what I did. I got the data and I put it into axiom.

Martin Wimpress 9:17
Where did this data come from? Did you w get my.com.com? Was?

Alan Pope 9:23
I’m glad you asked. There’s two things here. There’s the ongoing data. And then there’s the historical data. And I wanted to answer the question based on the historical data, I didn’t just want to start capturing data now and then be able to answer the question in two years time. I wanted to get the historical data for what I’ve done with my car. And it’s a mini manufactured by BMW and BMW have a personalised website where I can go in and turn on and off options to do with the car but one of the options is to download my data archive. And so it’s a bit like have downloaded your Twitter archive or your Google Takeout or something like that you press a button, verify your password. And then in a couple of days, you get an email, click through the email and you get a zip file. And the reason why I was more interested in the history is to be able to query the 18 months worth of data about my car. And the thing I’m most interested in is the charging history, which is something that is provided, you end up with a zip file that contains an XML file, which has entries for all the different features that the car has, how many doors it has, what model it is, every single feature that the car has options and everything. And there’s a base 64 encoded png of the car in that XML as well quite amusingly. And there’s a JSON file that comes with it one big JSON file, and it has an entry for every time the car was charged. It’s the full charging history of the car. And it’s quite fascinating because it has information in there about the location it was that the car was charged, the state of charge before it was charged, and the state of charge after it was charged as in battery percentage, the time of day in Unix format, of course, and also the street address of where it thinks the car was when it was being charged. And so that allowed me once I got that data, I mean, it’s just a JSON file. And in the 18 months, I’ve had this car, I’ve charged the car according to that JSON file, like 530 times or something like that. And I uploaded it into axiom. And then I started playing around with the query language and the query language is it’s a bit like Cousteau, which is a Microsoft query language. And I was able to build queries, and say, it’s a bit like SQL here, you select from this table where address is not equal to and then my street address, because then I can only look at data that is external public charges, or I could say, show data that’s only related to my home charger. And so I could split those two datasets and and look at how does the car behave differently when it’s at home when it’s away, or some of the interesting things I found. One interesting thing was that I charge away from home. So little compared to how much I charge at home. And I was able to build a little table that showed my favourite charging locations. So my favourite, it’s the ones I visited the most right? And it gives the street address of each one. And there’s one that’s at the top of the list and the street address was just be four, one to two or something like that. And I was like, where’s that? What is that? And it turns out it’s a charger on the be for one to two or wherever that wherever the road is in a place called chippenham, which is about halfway between my house and my son’s girlfriend’s house. I have charged at the chippenham services a few times when I’ve been taking my son over to see his girlfriend, He now uses public transport. But at the time, I found that was interesting that that was the top charger, because I charged on the way there and on the way back because it’s too far for the car to make it on its own on one charge. I can also aggregate like how many kilowatts I’ve got from each charger as well. And how much draw there was from the grid. And so you can see the difference between the amount of energy you’ve pulled from the grid and the amount that actually went into the battery of the car itself, which is quite interesting, I think. So some stats them for you. I’ve charged in the last 533 charges, 464 of them have been at home. So that shows the proportion of how much I charge at home, and how much I charge away. In fact, almost every day I charge at home. But sometimes I go for days without charging at all. And sometimes I charge multiple times a day. So it’s not once per day, every every day. It’s it’s it varies because sometimes I do a big journey and have to recharge and then go out again. And more often than not only trickle charges a little bit overnight. There are sometimes errors like people complain about the charging stations like being off or broken or unable to charge your car. And I have had 16 of what’s counted in the JSON as business errors. In the business errors are basically something failed, unplug it, plug it back in again. And I’ve had 16 of those failures. And interestingly, half of them were at home and half of them were public chargers so actually I’ve only had eight failures to charge away from home. Now often you find the company up and they reboot it or you just unplug and plug it back in again. So of the 500 some charges, eight of them failed, which I don’t think is too bad. It’s not great because I’ve never had a petrol car failed to fuel though, you know, so yeah, I created a little dashboard. So now I’ve got this dashboard, it doesn’t dynamically update because the data is a one shot that I downloaded from BMW. And there is an option for me to sign up to get API access. And I signed up and it takes ages to get approved, you have to wait days for BMW to approve your access, and actually got rejected. I think because I’m not a business making an app or a website. I’m just an individual. And so maybe there’s another way I can get that data. But I only got the rejection email today. And it was over a week ago, I applied for API access. So I might have to try another route to get that access so that I can get the ongoing data. Or I could just keep doing a download once in a while, have my whole data and just re upload it each time if I want to. Because it only creates a record every time you charge the car. And I’m only charging about once a day. So it’s not like it’s like hyper updating this file on a regular basis. But it was fun. And it was cool to make charts. I actually did this as part of a hackathon we had internally in the company, we have three days last week where you can work on anything you want, that either builds on the platform or improves the platform in some way. And all the developers did super cool stuff. And here’s me with my little dashboard, Advil, graphs of when my car was charged, and all that kind of stuff. But it was fun. It was fun. And I’ll write a blog post that goes along with this that gets all the details. And so you can see the actual charts and that kind of stuff.

Mark Johnson 16:28
I’m interested in what you said about having stats for how much is drawn from the grid and how much goes into your car? Have you been able to sort of notice any difference in that, especially compared to your home charger in public chargers, or AC chargers and DC chargers? In terms of efficiency?

Alan Pope 16:43
I haven’t queried that. But I could, all I’ve done is look at what the difference is. And it’s like 10%. At most, right, I think there were a couple of spikes on the graph, I did plot that on a graph and there’s a couple of spikes, I think that’s where the data is mangled when the charge didn’t work or something, you know, where it pulled a kilowatt from the grid, but didn’t actually charge because the charging failed. And then you end up with a massive disparity because the zero went into the car. So those are anomalies. Most of the time, the graph just looks normal as it charges every day at home. But it was super fascinating to me to back up my assertions that generally charges work fine. And generally, if you charge heroin, everything’s fine. But when you go onto the public charging network, there can be problems. But I also found it interesting that as a human behaviour, there are certain charges that I favour more than others, I’ve used lots of charges. But of those charges, there’s only like 12 that I’ve used more than once. There’s a lot that I’ve used once and never gone back to. The other thing to note actually is the data is needs a little bit of a cleanup. When I first got the car, my own address was wrong. It had the wrong house number in the data. And so that shows us two different locations because there’s a house down the street with a different number that is supposedly my house. So the data needs a little bit of cleanup, and I’ll probably get to that and it might make it look a little bit more reasonable. But other than that, it’s it’s been good fun. But yeah, I’ll write it all up in a blog post.

Mark Johnson 18:17
The next matters is part of the late night Linux family. If you enjoy the show, please consider supporting us on the rest of the late night Linux team using the PayPal or Patreon links at Linux matters.sh/support. For $5 a month on Patreon. You can enjoy an ad free feed of our show, or for $10 get access to all the late night Linux shows ad free. You can get in touch with us via email show at Linux matters.sh or chat with other listeners in our telegram group. All the details are at Linux matters.sh/contact. I got a fully Googled android phone Oh,

Alan Pope 18:54
I misread that I thought it was a fully D Googled Android phone. I know what it is. And I thought Wait, what how?

Mark Johnson 19:02
So my old phone, which was a one plus five t which I’d had for three or four years now was failing me because the USB port was knackered for the second time I’d already opened it up and replaced the USB port when it stopped charging. And the replacement USB port has also failed. It was still charging most of the time. But it wouldn’t do a data connection which was quite important because my new car supports Android Auto and I couldn’t plug it into my new car right and given the state of the built in infotainment system. I wasn’t going to put up with that. So I went looking for a replacement phone, which was about the right price and sported Android Auto. And ideally these days I buy my phones used because there’s enough phones floating around they’re in decent condition that I don’t need to generate more e waste by buying a new phone. So I went Looking around, and I found a site called Mozillian, which has nothing to do with Mozilla, despite the name, how weird but it’s a essentially an online marketplace for used phones, whereby people can list the phone they have, you then offered to buy it for the agreed price, they then send the phone to Mozillian, who do the checks to make sure it’s in the condition they said it was, and everything works. And then they send it on to you.

Martin Wimpress 20:26
That sounds like a very good service. Yes. And it

Mark Johnson 20:29
worked very well for me, because in this case, it was listed as a pristine phone, and they received the phone. And they decided it wasn’t pristine. So they called me and they called the seller and agreed slightly discounted price, sent me pictures of what they saw as the damage to it, and made sure that I was happy with that before they then sent it on. To me, it was just a few sort of minor cosmetic things, it was basically in perfect working order as far as I was concerned. So yes, Full marks for them. The phone itself is similar size to a one plus five t you don’t get phones that are much smaller than sort of almost six inches, diagonally these days. It’s slightly slimmer and taller rather than shorter and fatter, which fits my hand better. So that was quite good, I did often find myself sort of juggling my one plus five t in one hand, sort of having to rest it on the corner and wave my thumb across. So this fits me a lot better, which is good. So this is the pixel six A, which is the sort of budget version of the Google Pixel. It’s a slightly cheaper build, it’s got a plastic body, but it still is not cheap plastic, it’s still good and has a solid feel to it. And you know you wouldn’t want to be, you don’t want to have it thrown at you or anything.

Martin Wimpress 21:42
So you’ve used this marketplace to get previously owned phone. What era of pixel is this because I’m not up to date with that anymore.

Mark Johnson 21:51
This is the last generation of pixel. The current version of this is the seven a although you can still buy the six a new, I saw it advertised in Tesco last week, for example. But if you wanted to bang up to date, you’ve got the pixel seven and the pixel seven A. So it’s been around a couple of years now I think probably launched in 2021, if I remember rightly. So this one I imagine this one had been used for about a year before it was sold. So as I said, this is a fully Googled phone, this is running Android 13 With all the Google you could possibly have on a phone, which was interesting, because I’ve not had that since the Nexus four, I think was the last phone I bought from which was a Google phone.

Alan Pope 22:39
When you say all the Google stuff, I’m thinking the usual suspects like Gmail, Google Drive, and that kind of stuff is like a whole load more.

Mark Johnson 22:48
Yes. So quite often, if you buy a phone from someone like Samsung, or let’s say my experience with one plus, what you’ll get is something that’s built on ASP plus Samsung’s App Suite, or open source app suite, plus some bits and bobs or some third party proprietary thing that you’ve never heard of. Whereas this is Google everything. Google Contacts, Google Calendar, Google Calendar, Google, Google, Google Google, all the way down. Which means every time you want to do basically anything at all on a new app, even the calculator, it pops up with Google’s privacy policy, which it wants you to accept before you do anything and possibly link it to your Google account. And I found that it was all a bit much for me. So I went having a bit of a hunt for alternatives for all this stuff to see if there were decent sort of basic utilities that I could instal instead of all of the Google stuff. And I found a suite of apps called simple mobile tools, which is essentially an open source set of core apps for Android. So it has all the things like the photo gallery, and the calendar and the file manager and a note taker in the Contacts app. And they’re all open source. They’re all following a similar style, material design, kind of styling, and UI. And they’re all open source, you can get them from F droid. And I just started going through and seeing how many of these were a decent replacement for the Google version. And I found pretty much all of them, with the exception that I’ve kept google camera, because it has a lot of features which the open source camera tools just don’t have. And there’s enough smarts in there that produces better quality images at the end of the day. And I do also have Google Photos. I mainly use the simple gallery for actually viewing photos but Google Photos has Google lens built in, which is very handy if you want to read a QR code in a photo that you’ve taken or if you want to OCR some text In a photo and be able to copy and paste it, which is something I didn’t have on any phone previously, and is an absolutely excellent feature to have.

Martin Wimpress 25:07
F droid is a third party marketplace.

Mark Johnson 25:11
Yes, it’s a third party app store specifically for open source, right. Android apps.

Martin Wimpress 25:16
Okay. And this collection of open source apps, are they essentially the AOS, P apps, as was re vitalized. And presented? Are they wholecloth? New Project?

Mark Johnson 25:30
I’m not actually sure. I mean, they look and feel similar to the ASP apps that existed before Google started basically making Google versions of everything. But I don’t know if they have any history in that code base, or if they have been developed from scratch.

Alan Pope 25:45
When you’ve installed all these replacement applications, are you able to remove the Google ones? Or does it just remove the icon and keep everything on the phone? It depends.

Mark Johnson 25:53
So I’d say the majority of them could actually be uninstalled. But a few of them could just be disabled.

Martin Wimpress 26:01
So you’ve been able to reduce the clutter on your phone by disabling or removing the bits that you don’t need?

Mark Johnson 26:07
Yes. And migrating. So before I was running android 10, on my one plus five t, which was I think Oxygen OS was their spin of Android. And I’d previously moved from one plus one to a one plus five t using one pluses migration tool. In this case, I actually used a built in Android transfer feature to do a transfer over Wi Fi. So it could just see, I wouldn’t have to instal any other apps, it just found it as another Android phone and did some handshake II stuff, and then did it over Wi Fi. And what I found there was a few Apps Sync to their original settings, most apps needed to be set up again and signed into accounts. But there were also quite a few, especially the open source ones that had a decent export import feature where you could just say, write my settings to a file. And then I used KDE connect to pin that file between the two phones, and then imported it in and it was just like it was running on the old phone. So that was quite a nice experience, sort of three different levels of synchronisation there. But in almost all cases, I was able to keep the configuration from the old phone,

Alan Pope 27:20
did you say you sent the export files directly from one phone to the other via KDE connect or via a PC? Because I’ve never seen that as a use case that anyone’s talked about for KDE Connect. It’s always been connected to your computer and then access things on your phone or exposed features of your phone on your desktop. But never like direct peer to peer transfers, not something I’ve considered.

Mark Johnson 27:43
Yeah, he’s great. Yeah. Katie, can any KDE connect running device can see all of the others and do whatever is supported between them. So yeah, if you have someone else in your household, who you want to be able to ping stuff to, you can do it via KDE Connect. Hmm.

Martin Wimpress 27:58
Nice. Yeah. Or between households if you use zero.

Mark Johnson 28:03
I knew it was coming. Oh, another one other thing that I wanted to mention about Android 13 is so the phone has no hardware buttons, all sorts of soft hardware buttons, it will display optionally display the home back and app switcher. But what it recommends is gesture navigation, right? Which was a brought me screaming back to the Ubuntu Touch days, it uses very similar gestures, you swiping across the side to go back. So I picked up to get the app tray. And I sort of thought, Well, I’ll try it. But I don’t know if I like it. But actually, I find especially with the size of the phone the way it fits my hand. I’m sold.

Alan Pope 28:42
Nice. Yeah,

Martin Wimpress 28:43
I turned those gestures on my one plus, and used it that way for many years. And fortunately the tablet the Android tablet that I have also supports gestures. And I found my path migrating to iPhone, way easier than my wife’s because I’d spent two years of basically using the same gestures to move about the phone that the iPhone uses.

Alan Pope 29:09
Nice. Would you recommend the pixel six ad then?

Mark Johnson 29:11
I think I would. I would definitely recommend Mozillian for buying a phone if you don’t burn it by a pixel six.