Wednesday there was an unfortunate message posted from Richard Hughes regarding his firmware service.

As a company, we really try to focus our energies on positive progress in building open hardware. It’s important to us to set a good example for how an open community should collaborate. When this message was posted, we originally took the stance of not refuting the articles’ gross misrepresentations. However, this article has caused some confusion with our customers.

When we reached out to Richard over a year ago, we were enthusiastic about LVFS and interested in whether or not it would work for us. Once we described how we needed to deliver firmware, we were told that it would not work well and would likely not be acceptable to Red Hat Legal.

System76:
Yes, AFUEFI [a firmware flashing tool] is proprietary. We could distribute it in binary format, as part of the cab in LVFS. What we need to flash is for a simple, open source, piece of EFI code to run the extracted AFUEFI executable with the correct parameters.

The EC would also be flashed with a similar tool. If desired, we can package it all up into a single EFI executable to be run by fwupd on reboot.

The method we are currently using is to bless a new EFI binary (currently a UEFI shell) and reboot to it, then it runs a script that does the flashing, and then removes itself as a boot option.

Richard Hughes:
I don’t think this will work well; for a few reasons:

* there’s no way to reference a flashing tool in the XML, or to sign a
executable on the LVFS
* I don’t think Red Hat legal will like the idea of shipping the
flashing program, we’ve only ever talked about the firmware files
themselves (although I concede the images in the .cap are probably
firmware executable code wrapped up in layers).
* I know the Red Hat security team will do more than blink when we
tell them we want to ship a untrusted nonfree binary which would get
run as root on RHEL on customer machines.

There is no other way to interpret this email other than LVFS won’t work well for us. UpdateCapsule is not supported by over a decade of machines in the field and could not be added without a firmware update. With UpdateCapsule, the vendor’s proprietary update software is still present (with the same security considerations) but is integrated into the firmware blob. This is less modular than our approach, and does not allow the reverse engineering and open-sourcing of firmware update components.

So we can’t use LVFS for BIOS. Maybe the EC? We experimented with flashing the EC from user space. The keyboard is connected to the same device and would freeze during the update process. The experience was suboptimal and not up to our standards.

Delivering automatic updates to customers is important for our customer’s security and our ability to constantly improve computers. We continued to experiment. In about a week or two we had a working proof of concept that delivered the BIOS and EC reliably and securely. Over the next months we improved the tool, added unique blockchain security infrastructure, moved it into laptop production and then out to customers.

The decision making is simple. We are told we can’t distribute the tool we need in LVFS. Flashing the EC in user space is suboptimal. And in any case managing separate repos for two components that work together is error prone. And we have a working prototype. Soon thereafter we were shipping firmware built on secure infrastructure out to customers. Thanks to this work we were able to quickly and automatically patch computers when the Intel ME vulnerability arrived, and we have a battle-tested system in place for the future.

LVFS Misrepresentations

The statement that claims our firmware tool only works on Pop!_OS is not true. System76 computers ship with Pop!_OS and Ubuntu, and both include automatic firmware updates. All Ubuntu based distributions are supported. We’ve had customers with Arch use our firmware update tool. We’ll officially support more distributions as time permits.

Our conversations were completely misrepresented and distorted in a way that was designed to make us appear as though we made the decision not to use LVFS rather than the truth - there were several conversations to try to implement this, followed by many roadblocks, and then we were told “I don’t think this will work well.”

Be wary

We had intended to use LVFS in the future, but that is no longer the case; there are too many problems with the project.

If you want to use LVFS, disable the data collection. There’s no need for it. Understand that the first instinct of the project leaders was to unnecessarily opt-in data collection from user installations.

Understand that you’re electing for your distribution to communicate with third party servers. The more servers your distribution communicates with out of the box (especially as root), the more surface area there is for vulnerabilities. Heavily scrutinize any addition of a third-party server for updates.

Understand that if you are a company specializing in Linux-only devices and considering using LVFS, you are handing your private sales data to LVFS. We suggest hosting LVFS on your own servers.

We engaged with the project in good faith. When it was determined that LVFS would not work for our needs, we built an open source tool that would provide the necessary functionality for our customers. Having a collaborative conversation about how to use existing tools, then needing to build an open tool that best fits your needs should never result the way this conversation did.

System76 and GNOME

It’s unfortunate this appeared on the GNOME blog, but it only represents one person—not the GNOME community as a whole, nor our relationship with GNOME. We couldn’t be more excited about our shared future with the GNOME project and working together to advance the free and open source desktop.

The Future of Firmware

LVFS and UpdateCapsule might be okay for companies mostly focused on a proprietary future (Logitech, Dell, etc.). UpdateCapsule is not the technique companies will use in a future of open source firmware—the future we’re working toward.

Liberating firmware is going to be a long and challenging process. Much like Free Software has replaced proprietary software over time, we must chip away at the proprietary firmware pieces within the hardware supply chain. Manufacturing is the first step. This year we’ll manufacture open source desktop designs in our Denver plant. The CAD will be free to download, change, and produce.

There will be a separate, open source electrical design and open source firmware daughter board to control functions within the desktop. On a mainboard there is the BIOS chip and one or more embedded controllers that manage fans, keyboard, LEDs, hotkeys and other critical functions. It’s all proprietary. Our strategy is to move this functionality from the proprietary mainboard to the open source daughter board. Then anyone can create a PCB with basic computer functionality, understand how it works, and improve upon the work. One could have this PCB made at Osh Park, install it in their desktop, tune it, and replace a bunch of proprietary firmware instantly. We’ll grow from there.

Slowly we’ll chip away at more and more of the mainboard functions until what’s left is Intel and AMD bits. Then there’s the challenge of convincing them to go open. There’s room for cautious optimism.

On Open Source and Community

There are thousands of open source projects led by smart and supportive people that are kind to each other and good to work with. Open Source as a community gets a bad rap at times. Projects that attempt to bully others into saluting their flag are the exception. We choose to support those that want to build things and those that encourage others to build things. Go GNOME, go KDE, go i3, go Solus, go elementary OS, go Kdenlive, go OpenShot, go, go, go! Let’s just make things. Don’t worry about saluting one flag over another. Do what is right for you and your project. The premise of “NIH” is antithesis of open source.

Now, we’re off to build a factory to manufacture open source computers. Oh, and when you buy a System76 computer, you support people actually working to liberate the computer.