Dell Latitude 7410 has a broken ACPI implementation

On 24-Mar-2021 01:36:43:

So last year I got a new Dell Latitude 7410 laptop at work. Recently some other coworkers were getting new laptops and they asked about this one. One specific question that I was asked was whether or not suspend to RAM worked properly. I told them it was working fine.

I was wrong, I was so wrong. Today I discovered that it only appears to work fine. The truth is that it is actually very broken in a non-obvious way.

The default behavior of suspend to RAM on most operating systems is that it automatically enters suspend on lid close and automatically resumes on lid open. If that is how you want suspend to work, then it works just fine on the laptop. Of course if you frequently close the lid on your laptop to walk to another room at work you might chose to disable this and have the laptop enter suspend only if the user explicitly tells it to. You might also set it to not wake up until the power button is explicitly pressed. These are normal user configurable settings. I'm too lazy to reinstall Windows on this laptop but I'm guessing those settings don't work right on Windows either. I'm not even sure its possible to work around this particular ACPI bug from the OS.

Lid close is an unconditional ACPI wakeup event!

Whiskey Tango Foxtrot?!? If you put the laptop in suspend to RAM and then close the lid, the laptop wakes up. I cannot fathom any situation in which this is desirable behavior.

Okay, but you can usually disable individual wakeup events from the OS. On Linux you do it in a file called /proc/acpi/wakeups. If you cat the file you'll see a list of all the possible ACPI wakeup event triggers and whether or not each trigger is enabled. If you echo the name of one of the triggers into the file it will toggle the enable/disable state. The list is kind of big, so here are the ones that show up as enabled when Linux boots on this laptop:

RP05      S4    *enabled   pci:0000:00:1c.0
PXSX      S4    *enabled   pci:0000:01:00.0
RP09      S4    *enabled   pci:0000:00:1d.0
RP13      S4    *enabled   pci:0000:00:1d.4
XHC       S0    *enabled   pci:0000:00:14.0
LID0      S3    *enabled   platform:PNP0C0D:00
PBTN      S3    *enabled   platform:PNP0C0C:00

LID0 is the lid switch. This is the event you disable if you don't want your laptop automatically waking up when you open the lid. On every previous laptop I've ever used this LID0 is only a wakeup event on lid open and never on lid close. After all, why would you ever want the laptop to wake up if you're closing the lid? Okay fine if this laptop wants to be weird we can echo LID0 into the file to disable it. Thats not a big deal, I don't want a wakeup event on lid open anyway.

Except this toggle doesn't actually work! Even with it set to disabled both lid open and lid close remain ACPI wakeup events. My immediate thought was to blame Linux, but I don't think this is a bug in Linux. I went into the BIOS (which is not actually the right name for it anymore, but everyone still calls it that) which has its own options for controlling lid switch behavior. These BIOS settings also do not work which suggests that the problem lies in the firmware and is not something the OS can control.

Another indication that this is not a bug in Linux is that other events do toggle properly. I discovered that unplugging the USB-C power adapter is also an ACPI wakeup event. I traced this behavior to the RP05 event and was able to disable it from Linux. So toggling wakeup events from Linux actually does work, just not for the lid. The ACPI firmware itself must be buggy and ignores any attempt to disable wakeups on lid events.

On top of all of this Dell decided the laptop didn't need a power LED so if the lid is closed there is no clear indication of whether its still in suspend or if it woke up. I had to debug all of this using a USB mouse flipped upside down as a crude power indicator. I also tried upgrading the BIOS from 1.1.1 to 1.5.2 but that didn't help.

I'm not an ACPI expert but I have every reason to believe this is a bug in the ACPI firmware and there is nothing the OS can do about it. If this works correctly on Windows then it is only because Microsoft has some impressive workaround. Most OSes have workarounds for ACPI bugs, but I'm not aware of any way to disable an ACPI wakeup event if the firmware is ignoring the normal interface that the OS would use to do that. The only OS level workaround for bogus wakeup events that I can think of is to just immediately go back into sleep if the OS thinks that the wakeup event was bogus.

So I have no choice but to use the default Windows behavior of having lid close trigger suspend to RAM and lid open trigger resume. This is the only way to avoid the rogue ACPI wakeup event that occurs when the lid is closed. I'm okay with the latter, but I really do not want the former. Or do I... Linux is still in full control of exactly what happens when a lid close event occurs. If I want lid close to mean suspend to RAM but only during a full moon and at any other time it should say "I'm sorry Dave, I'm afraid I can't do that" over the speakers, then I can make it work that way. Or more usefully, I can make it check to see if the screen is locked.

So with my workaround in place the actual act of suspending is triggered by the lid close event, but it only happens if the screen is already locked. This allows me to carry my laptop around with the lid closed without it going to sleep and still have working suspend to RAM.

Click here to comment

ACME Client

On 10-Jun-2018 13:54:05:

I just realized i never posted about it here, but a while back I wanted a simple ACME client for handling my Let's Encrypt SSL certificates. I found a nice and simple one for doing http-01 challenges that was named acme-tiny. I made a fork of the original client which does dns-01 challenges instead. This also means it can handle wildcard certificates! (The original can't do wildcard certs because http-01 auth doesn't support them.) You can get my fork from:

There is a distinct pattern where I mostly work on it every 90ish days when I renew my LE certificates.

Disclaimer: I've never really tested the dynamic DNS support, someone sent me that as a pull request.

Click here to comment


On 29-Jan-2018 23:20:55:

MFW I write a program that might actually be useful enough to put a proper license on.

OpenSSL does not support converting an elliptic curve private key from using explicit parameters to using a named curve. So I wrote a program to do that.

Github repo
Local mirror (cgit > *)

Click here to comment

Offline GPG Primary Keys

On 18-May-2015 01:11:18:

A bunch of years ago I drank a little too much Kool-Aid and went full on paranoid with PGP keys. GPG keys are made up of primary keys and subkeys. One cool thing you can do is keep the primary key offline and generate new subkeys each year. Would I do this again? Probably not, its annoying. But it was interesting at the time and now I have to re-learn how to manage the key each year when generating new subkeys. This post is really just my own documentation.

I haven't done it in a long time, but the procedure for generating a new key to manage this way should be very similar.

The main problem with this is to sign other PGP keys I need the primary key and an even sillier GPG command:
gpg --homedir PATH_TO_FULL_KEY --keyring ~/.gnupg/pubring.gpg --secret-keyring ~/.gnupg/secring.gpg --trustdb-name ~/.gnupg/trustdb.gpg

IIRC this was even more ridiculous when I first did it, but GPG has gained some options which make it easier

Click here to comment


On 27-Oct-2014 01:58:00:

Since none of the other SSL certificate revocation mechanisms actually work, Google invented their own for Chrome, they call it CRLSets. Their conclusion was that if the existing mechanisms didn't work, maybe they could design one that works in a more limited scope. The limited scope they chose was EV certificates, ideally only those revoked for actual security reasons. I think that within that scope they succeeded, although others certainly disagree.

I was curious exactly what was being pushed out to Chrome, so I decided to dig a little deeper. If you don't mind installing Go, then Adam Langley released a tool for pulling the latest set. I certainly don't mind installing Go, but what better way to see what the data is like than porting Adam's tool to say Python? I've never done the kind of crypto work I need to verify the file in Python, might as well go learn it.

What does a CRLSet look like? (I'll skip over the authentication and integrity checking, its interesting to code that sort of thing in python for the first time, but its not that interesting to talk about) Well it has a sequence number (currently 1882) and an expiration date that looks to be about 4 days in the future. From what I've seen so far its generated twice a day right at around noon and midnight in the Bay AreaExclusion Zone. The file contains a set of certificate serial numbers categorized by the SHA-256 hash of the "Subject Public Key Info" field of the issuing certificate autthority, basically the RSA public key (although there are DSA and ECDSA certificate authorities out there). Packed in with the metadata (it was added after the fact) is an explicit list of blocked certificates, against listed by the SHA-256 hash of their SPKI.

So there you have it, thats a CRLSet. You'll notice something though, the limited scope of the CRLSet means that only 54 CAs are included. Given that many of those are likely to be intermediate certificates, just how many CAs do we really have in the file? What if you were setting up an important website and wanted to be *SURE* that you used a CA that was included? Well Google doesn't officially disclose the participating CAs, they are also upfront about the fact that some of the revocation information they crawl is non-public. I don't think they really intend for the list of CAs in the CRLset to be a secret though, they just aren't volunteering the information either.

So now that I'm done figuring out what is in a CRLSet, now it is time to figure out what is in a CRLSet. So there are currently 54 CAs in the file and 8 explicitly blocked certificates. The latter bit is faster to talk about, because with the exception of the 2 well known entries in that list ( and we are probably out of luck finding out what those are. The list of CAs on the other hand is probably knowable. For starters, I think it is pretty unlikely that Google would include a CA that Chrome wouldn't trust, which means we can start by extracting the root certificate store from a few operating systems.

Well my listing of about 200 root CAs only got me 13 entries out of 54. :( Turns out very few CAs directly issue end user certificates off of their root certificate anymore, instead they issue themselves (or subsidiaries) intermediate CA certificates and issue end user certificates from those. So we need a good source of intermediate CA data, which is generally not included with the operating system. Some of the CAs have nice and easy to download bundle files, which helps a litttle, but still didn't get much. Since end users with chained certificates are supposed to be serving up the intermediate certs as part of the SSL connection, I could just scan the internet on port 443, but that would take a while.

Notice how I said supposed to up there? Yeah, some servers are misconfigured and don't share the intermediate certificates which means their end user certificate can't be verified. Apparently Mozilla caches these intermediate CAs indefinitely when it receives them, presumably to help it complete the certificate chains when it finds these bad servers. Even better, we easily can extract this from the internal certificate store! I now have nearly 600 CA certificates which gets me 36 out of 54, not bad for a little scripting work. CA hash mapping. I'll keep updating that as new SPKI hashes show up in the CRLSets and as I collect new intermediate certificates.

The names of the CAs in and of itself isn't terribly interesting, its exactly what Google said it was. The CAs are mostly EV and High-Assurance intermediate CAs. So what was the point? I'll admit, mostly because as far as I can tell no one has done this yet. Dumping the list let's everyone know who the star bellied sneetches are in the eyes of Google. The explicitly blocked SPKIs would be more interesting, but guessing those just isn't possible. I suspect they all correspond to high profile revocations and with some research I might be able to identify one or two others. Perhaps any researchers doing long term monitoring of revocations would be interested.

Click here to comment

Important Message

On 11-Oct-2014 12:38:50:
                Y.                      _
                YiL                   .```.
                Yii;                .; .;;`.
                YY;ii._           .;`.;;;; :
                iiYYYYYYiiiii;;;;i` ;;::;;;;
            _.;YYYYYYiiiiiiYYYii  .;;.   ;;;
         .YYYYYYYYYYiiYYYYYYYYYYYYii;`  ;;;;
      :YYY$!.  TYiiYY$$$$$YYYYYYYiiYYYYiYYii.
      Y$MM$:   :YYYYYY$!"``"4YYYYYiiiYYYYiiYY.
   `. :MM$$b.,dYY$$Yii" :'   :YYYYllYiiYYYiYY
_.._ :`4MM$!YYYYYYYYYii,.__.diii$$YYYYYYYYYYY
.,._ $b`P`     "4$$$$$iiiiiiii$$$$YY$$$$$$YiY;
   `,.`$:       :$$$$$$$$$YYYYY$$$$$$$$$YYiiYYL
    "`;$$.    .;PPb$`.,.``T$$YY$$$$YYYYYYiiiYYU:
    ;$P$;;: ;;;;i$y$"!Y$$$b;$$$Y$YY$$YYYiiiYYiYY
    $Fi$$ .. ``:iii.`-":YYYYY$$YY$$$$$YYYiiYiYYY
    :Y$$rb ````  `_..;;i;YYY$YY$$$$$$$YYYYYYYiYY:
Click here to comment


On 31-May-2014 02:02:52:

As my twitter followers are already aware, I recently realized just how stupid Subversion's "tagging" behavior actually is. As someone who had never been forced to use CVS, I wasn't aware of what I was missing by not having real repository tagging. While goofing around with the various git web frontends, it finally dawned on me just how broken Subversion was. Let's take a little journey through some web frontends as a demonstration.

First off, this is a page most Subversion users will be familiar with. This is the commit log for a file as display by the popular frontend ViewVC. There isn't much more to say other than the top revision shown has a few tags made from it. (it is really hard to find a public repo with just the right amount of activity to make these screenshots)

Here is the git repository for git itself as displayed in the popular web frontend GitList. If you view a project on GitHub you would see a very similar interface. Coming from Subversion you wouldn't see anything wrong with this, GitList is a perfectly cromulent feature match for ViewVC's Subversion viewer. Of course, the reason you don't see anything wrong with this is because you've suffered brain damage from using Subversion too long.

This is a screenshot of CVSweb showing part of OpenBSD's repository. Notice how just by looking at the commit log, we can see where along the line each one of the tags was made? Because CVS has real repository tagging, it can easily connect each revision with the relevant tags. The CVS command line tools are more than happy to display this information as well, its not just a trick in the web frontend. The lack of this information in ViewVC isn't an oversight because ViewVC actually supports both Subversion and CVS and when it is used with CVS it display this information.

GitList on the other hand did simply leave it out, this is the exact same set of commits shown in the GitList image above but in the cgit repository viewer. Like CVS, git has real repository tags and the information is readily available at the command line by passing --decorate. Comparing cgit and GitList is what drew my attention to this. Of course, there is nothing wrong with GitList omitting this information, the git command line tools don't share it by default either. What drew my attention was that as an Subversion user I was completely unaware that this information was so easily available!

Subversion is effectively incapable of sharing information like this because Subversion does not have tags. Instead it simply has copies of directories that a team has agreed have special meanings based upon location. Doing some very basic operations on tags (like say show the commit log between two tags on the same branch) require you to jump through ridiculous hoops. People who have only experienced Subversion probably don't even realize how crippled they are by these "tags".

Subversion's atomic commit operations are definitely an improvement over CVS, but the fake tags are not as harmless as the Subversion manual leads you to believe. If you have only ever used Subversion, be warned that this unique "tag" behavior has likely warped your sense of what a VCS is capable of.

PS: Since inevitably someone will ask about Mercurial, as far as I am aware based on the hg documentation, it (like git and CVS) displays any tags associated with revision when viewing history. Finding a good public repo to use for a Subversion screenshot was hard enough, I didn't want to go try to find one for hg.

PPS: I made it my whole rant without bringing up the fact that Subversion "tags" are not read-only! (shit, nevermind)

Click here to comment

Im Sorry Dude Youre Getting A Dell

On 18-May-2014 22:12:51:

Dell, we've got to have a little talk...

This rant has been 3 months in the making because I was hoping the situation would improve, but its not. You see, I have this little problem where I need a new laptop at work. Which means I need to nagivate the abomination that is your website...

Broken Search Functionality

A search of Lattitude laptops with the following options checked:

Returns exactly 3 unique models (9 total configurations. They are

Why have you omitted the following models:

There used to be more models omitted from the results, it appears to have been cleaned up a bit. The system also used to return results for laptops that were not actually available with an SSD. I'll give you some credit here, its better than it was 3 months ago, but its still pretty broken. I'm sure if I actually played around with it more I could find plenty more examples, this didn't exactly take long to notice.

Tech Specs do not match what you can buy

You can pretty much throw a dartboard at the website and find tech specs that are not actually available for purchase. Likewise, finding things available for purchase which aren't listed at all. Take the Lattitude 14 7000's processor options

It doesn't take much browsing to find tech specs unavailable for purchase, but I am not Dell's unpaid QA department.

So Dell, please fix your website.

Seriously, how hard should it be to find a laptop? I don't exactly have complicated criteria:

Is that really that hard? Even six months ago you had machines that fit this, mostly because you actually allowed customization of machines. These days if the 4 pre-configured machines for each model don't fit the bill you are screwed. If you simply brought back the ability to truely customize machines these criteria would easily be matched by any number of models on your website.

Click here to comment

No Title

On 17-May-2014 17:57:26:
Click here to comment

Hello World

On 17-May-2014 17:53:27:

Got sick of my overly complicated blog, still working out how the old simpler SQL-free stuff worked...

Looks like I have some mediocre category support, will probably remove that... seems to be working otherwise

Needs support for post titles too

Click here to comment

Archived Entries