Switched from Ubuntu-based to Fedora

tl;dr: Fedora’s debugging packages work, Ubuntu’s are out of date.

Linux = Linux = Linux, whether Arch or Slackware or Ubuntu or OpenSUSE or Linux from scratch as I once did (before there were instructions!). Unless and until the kernel forks and someone decides to modify the syscall table, they all use the same basic syscalls, they typically share the same basic libraries and core utilities, etc. They’re all the same.

Why did I use Ubuntu-based distributions? (Note: Not Debian) Because Ubuntu came pre-configured with all the things I did not care to learn or manually configure: ACPI, firmware, X11, a pretty WM theme, etc. I did not particularly care whether I was running Mint, Elementary, Ubuntu MATE or basic Ubuntu (except Unity…nah). As long as it did not do strange things like remove /sbin/ifconfig or have a radically different file structure than I was used to. I felt at home with knowing where the standard file paths were, and knew how to administer my machine. Their package repository was pretty solid. It had almost everything I wanted – and what little was not on it was typically available in Debian-package format. The broader Linux community effectively standardized on this package format. This is crucial. Debian’s apt and FreeBSD’s pkg are in my muscle-memory at this point.

Literally one thing pushed me over: Ubuntu’s SystemTap was broken. Utterly broken!

I got into OS-level programming, specifically, porting a Linux WiFi driver to FreeBSD. I wanted to use SystemTap, Linux’s answer to DTrace, to help understand what is going on during live execution. But SystemTap does not work on Ubuntu – at least currently.

But wait, I thought Linux = Linux = Linux and programs from 20 years ago will still work. Why does SystemTap fail?

SystemTap works by producing C code for a kernel module, compiling it and loading it into memory. Sometime ago, the kernel team changed the get_user_pages() kernel API call. This meant that any code compiled against the old function definition failed. I encountered this in the professional space when the VMWare kernel modules failed to build and I hacked it until it worked. (They think I’m a wizard now). I was on Kernel 4.10 but the version of systemtap Ubuntu used was nearly 2 years old. This meant no one from the Ubuntu team was using it.

I submitted a bug report and installed Fedora 26.

SystemTap was developed by Red Hat and was trivial to get working under Fedora. And while not every single package is available (Bitcoin, Steam thus far), there is enough that moving over was trivial. Also, they come in Cinnamon, which I prefer, with a pretty theme. And it provides a clean terminal out-of-the-box. Which I need. (I would rather use stock XFCE if their terminal was clean than fully-loaded CentOS with an ugly terminal)

dnf took a little getting used to, but a hop-over from apt. So whatever on that front….

I would be willing to try OpenSUSE again, but the latest time I did, they got rid of /sbin/ifconfig for /sbin/ip, which is unacceptable. Silly, perhaps…Does it come in Cinammon? What does it offer? Are the packages as clean and up to date? I may never know, unless another business-need arises. I do not care to run any of these “hardware” distributions, like Arch. I paid my Linux dues around kernel 2.2 on Slackware and its time to move on from that.


But look, if you’re 99% of the Linux world, any specific distribution is trivial. Pick one and go with it. Unless you’re doing very specific tasks like me, it really does not matter what you use. So stop Distro Hoping!


Custom Kernel Modules for Chromebook

Note: I wrote this about a year and a half ago, but I refer to it all the time. Hopefully the instructions have not changed too much! Enjoy!

I recently purchased a Chromebook. It’s great, it symbolizes the direction the PC market should head – inexpensive, low-powered ARM processor, defense in depth resistance to malware and simple for non-technical users. And with crouton, it functions quite cleanly as a Debian-based workstation.

With its simplicity and low price, there are certain key features that are lacking in the stripped down Linux kernel that can make it frustrating for a power-user. Unfortunately, Chromium addons have not or cannot satisfy some tasks that require kernel-level functionality. Even in crouton, you may find your ability limited to the user-space. Those looking for casual additions, recompiling the kernel may seem like daunting over-kill. Instead, compiling and inserting a single module may serve as an apt alternative. In this guide, I will explain how to compile a custom kernel module to add additional functionality to your Chromebook and how to circumvent the built-in security mechanisms that prevent you from adding into the kernel-space.. This guide is specifically written for an ARM-based CPU using kernel 3.10.18 for the CIFS (SMB) module, but can be trivially ported to any other architecture, kernel and module.

Compiling the Kernel Module

As mentioned, Chromium OS is a stripped down version of Linux. Therefore, you should be able to compile and dynamically link kernel modules from the stock kernel into Chromium.

Per Google’s documentation, you must compile the kernel and modules on an x86_64 CPU, even if you will be compiling an ARM or 32-bit x86 module. This is possible thanks to GNU C Compiler’s cross-platform capability. The documentation also specifies using Ubuntu, but it worked just fine on my Debian 8 workstation.

If you have not already done so, install git, subversion and perform the basic configurations:

sudo apt-get install git subversion
git config --global user.email “name@domain.tld”
git config --global user.name "Your Name"

Google manages its various git repositories with wrapper depot_tools, a custom git wrapper. You can clone the associated git repository and set your PATH environmental variable to include the wrapper scripts as follows.

git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git

export PATH=`pwd`/depot_tools:"$PATH"

Next, make a directory where your Chromium OS build will reside, download the Chromium source, and synchronize it to the latest updates. This take around 30 minutes to complete.

mkdir chromiumos && cd chromiumos
repo init -u https://chromium.googlesource.com/chromiumos/manifest.git
repo sync

Once completed, you will need to download the cross-platform SDK environment, build the dependencies and enter a chroot(1) environment. This will take another 30 minutes.


Now that you are inside the chroot(1) environment, you need to specify the hardware configuration for your Chromebook device, either x86-generic, amd64-generic or arm-generic. You can determine your architecture by running uname -m on your Chromebook. For my ARM-based CPU, I did the following:

export BOARD=arm-generic

Now you must prepare the core packages associated with your board.

./setup_board --board=${BOARD}
./build_packages --board=${BOARD}

Change directory to ~/trunk/src/third_party/kernel/ and then to whichever subdirectory is associated with your kernel (ie, v3.10 for 3.10.18). You can determine your kernel version by running uname -r on your Chromebook.

Next, we will need to tell the kernel which hardware platform you are on and start with the base configuration of the kernel. A list the options of base configurations by running find ./chromeos/config. In my case I am using NVIDIA’s Tegra motherboard, which is ./chromeos/config/armel/chromeos-tegra.flavour.config, so I use chromeos-tegra as follows:

./chromeos/scripts/prepareconfig chromeos-tegra

If you are compiling for a non x86_64 CPU, set the architecture and compiler settings as follows:

export ARCH=arm
export CROSS_COMPILE=armv7a-cros-linux-gnueabi-

This next portion is the same as compiling any other kernel module. Configure the kernel by running make menuconfig

Select whichever controls you would like to install and save. Once completed you will have a .config file that corresponds to your hardware. Since we are only compiling the kernel modules, you can either run make modules to compile all kernel modules, or make fs/cifs/cifs.ko to build only a specific module. I prefer the former because your module may require other dependencies in other modules, such as with crypto/md4.ko for cifs. You can verify that the file was built for the right architecture by running file fs/cifs/cifs.ko. Great! On to inserting the module!

ChromiumOS’s Security Mechanisms

ChromeOS is the official signed release of ChromiumOS, which is what you run in developer mode. Even in developer mode, Google implemented multiple defensive mechanisms to slow down a would-be attacker from gaining access the underlying system. To protect the kernel, Google utilized the Linux Security Module (LSM), which validates files from the root partition against a list of cryptographic hash values stored in the kernel, thereby preventing an attacker from loading a malicious kernel modules. In effect, the only way to insert a kernel module is to have it stored on the root partition. But by default, the root partition is set to read-only, so you cannot simply move a file to the root partition and load it.

Therefore, we must disable the root partition verification running the following script.

sudo /usr/share/vboot/bin/make_dev_ssd.sh --remove_rootfs_verification --partitions 4

Now, reboot the machine and from ChromiumOS remount the root partition to be read-writeable, as follows:

sudo mount -o remount,rw /

From here, you should be able to simply insert the kernel module with insmod. Now, you can install

Draw this shape without picking up your pen

For many years, while in a meeting or in a moment of free time, I have tried to draw this shape without picking up my pen or drawing over the same two points twice.


At best I would get 1 line away, but never completed the shape.

I wanted to know if it was even possible. So I wrote some python code to try every possible combination.

But, the code is below.

#!/usr/bin/env python3

import copy
import sys

lines = {

def check(cstate):
    for offset in lines:
        if sorted(lines[offset]) != sorted(cstate[offset]):

def iteration(clocation, cstate):

    if len(cstate) == 8:

    for ilocation in lines[clocation]:
        nstate = copy.deepcopy(cstate)
        y = nstate.get(clocation, [])
        x = nstate.get(ilocation, [])

        if ilocation in y:

        y = y + [ilocation]
        x = x + [clocation]

        nstate[clocation] = y
        nstate[ilocation] = x
        iteration(ilocation, nstate)

iteration(1, {})
iteration(2, {})

The lines list is an abstraction of the possible points in the shape and where they can connect to. Point 1 is the top point, 2 and 3 are the top corners of the square, 4 and 5 are the far left and right points of the triangle, etc.

Starting at points 1 and 2. Point 1 is functionally the same as points 4, 5 and 8, while point 2 is the same as 2, 3, 6 and 7. No need for unnecessary iterations. Give its current location, the code recursively builds lines to all possible connection points. If no points are available, it just returns.

It breaks when all possible links are met, as seen by the check function. This is done by checking if every point is touched at least one, and then iterating through all points to see if that point is connected to every possible other line.

Turns out it is not possible.


FreeBSD and Linux Remote Dual Booting

The following is a quick and dirty guide on how to setup remote dual booting for FreeBSD (12.0-CURRENT) and Linux (Ubuntu 16.04). Granted, this method is slightly a hack, but it works and suits my needs.

Why remote dual-booting? I am currently developing a FreeBSD kernel module for a PCIe card. The device is supported on Linux and I am using the Linux implementation as documentation. As such, I find myself frequently rebooting into Linux to look printk() outputs, or booting into FreeBSD to test kernel code. This device is located at my house, and I typically work on it during my downtime at work.

Why not use Grub? I would have preferred Grub! But for whatever reason, Grub failed to install on FreeBSD. I do not know why, but even a very minimalistic attempt gave a non-descriptive error message.

efibootmgr? Any change I made with efibootmgr failed to survive a reboot. This is apparently a known problem. Also, this tool only exists on Linux, as FreeBSD does not seem to have an efibootmgr equivalent.

Ugh, so what do I do???

The solution I came up with was to manually swap EFI files on the EFI partition no an as-needed basis.

First, I went into the BIOS and disabled legacy BIOS booting, enabled EFI booting, and disabled secure booting.

Then, I installed Ubuntu. I had to manually create the partition tables, since by default the installer would consume the entire disk. However, this does not automatically create the EFI partition. So, you must manually create one. I set mine to 200MBs as the first partition. After installation, I booted up, mounted the /dev/sda1. I found that ubuntu had created /EFI/ubuntu/grubx64.efi and other related files. Great!

Next, I installed FreeBSD and while manually setting up the partition tables, FreeBSD auto-created an EFI partition. One already exists, so I safely deleted it, and proceeded with the rest of the install. Right before rebooting, I mounted /dev/ada0p1 (sda1 on Linux) as /boot.local/ and /dev/da0p1 as /boot.installer/. I then copied /boot.installer/EFI/BOOT/BOOTX64.EFI too /boot.local/EFI/BOOT/EFIBOOT/BOOTX64.EFI (I think I had to re-create EFI/BOOT, I’m forgetting off-hand). Then I rebooted.

When I rebooted the machine, Ubuntu still came up. This is because Ubuntu edits the EFI boot order and places ubuntu as the first partition. Ordinarily you should be able to use efibootmgr here to boot into FreeBSD and use the non-existent FreeBSD equivalent to boot back, but with the lack of that option, I mounted the EFI partition (/dev/sda1) as /boot/efi, and when I wanted boot into FreeBSD, I renamed /boot/efi/EFI/ubuntu/grubx64.efi to ubuntu.efi and then copied /boot/efi/EFI/BOOT/BOOTX64.EFI to /boot/efi/EFI/ubuntu/grubx64.efi. When I rebooted, FreeBSD came back up! Then on the FreeBSD side, I mounted /dev/sda1 to /boot/efi and did copied /boot/efi/EFI/ubuntu/ubuntu.efi to /boot/efi/EFI/ubuntu/grubx64.efi.

And that’s it! I can now remotely boot back and forth between the two systems.

Ugly? Yes. But it does the job.

Linux could fix this problem by debugging their efibootmgr utility and FreeBSD could fix this by having an efibootmgr equivalent at all.


My python3 Programming Environment

UPDATE: I have since started using a very good vimrc. I recommend it over mine listed below. My only modification is that I removed all line numbers, eww.

I ssh into a FreeBSD jail with everything setup.

The Jail runs on code.mydomainname.com, which has an internet-routable IPv6 address – and IPv4 behind a NAT, (boo!)

I have a virtualenv already built-out. (more about my pip list later)

The set my ~/.bashrc to execute source enter-env.sh (even though I run ksh)

My REPL is ptpython, which just requires touch ~/.ptpython/config.py.

I use gitlab, since they offer free repositories, and then periodically manually backup my code at other locations. If there are automatic ways of doing this, I would be interested.

My project’s gitlab wiki has copy-paste instructions to install all necessary packages, both on FreeBSD and Debian (well….Ubuntu) and subsequent python3 packages that you install with pip.

My default browser is vim, and I set ~/.vimrc to: set ts=4 / set expandtab. I used to set syn on, but that does not seem necessary anymore.

My project requires a PostgreSQL database, so I included the very simple instructions on installation and configuration in the gitlab wiki.

Finally, though I typically code off of a FreeBSD Jail, everything is configured to run on Debian. The main reason it works on Debian is because my personal computer (before my Chromebook took over) is was Mint, but I intend to run this code on a FreeBSD server, primarily for ZFS. I used to code on a Raspberry Pi, but it was too slow.

It takes me about 5 minutes to rebuild this environment, in the event that it goes down (which it never does).


Two Types of Penetration Testers

There are two types of penetration testers in the industry.

Those who identify risk and vulnerabilities beyond a simple Nexpose/Nessus/Qualys scan. And those who want to “win”.

The job of the “winner” is to get DA on their client’s network. Great! But once they’ve gotten it, they show off. Look how much information I can get with the DA account! I can get access to these databases and these spreadsheets. Sensitive Information! Be afraid! I pwned you noobs!!!

The other type of penetration tester also seeks DA. He finds it. Great! Now he moves on to another vulnerability. And then another. Can I get DA another way? Maybe? Okay, what else is exploitable here. Along the way if he finds sensitive information, he presents it to the client. But his job is not focused on presenting information, its on finding avenues to find information.

If the “winner” focuses on one issue. He writes his report and presents it to the client. The client is enamored, impressed, even afraid. But if another “winner” penetration tester were to come in tomorrow, because the first only focused on one issue, the second would just find another avenue. A third might find a third issue. Until they are merely playing wack-a-mole.

The second type reports the issues he identified, an array of vulnerabilities, ranked by severity. He may briefly present information he was able to access, but then moves on. His job is to work towards resolution, not playing hacker. Overall security is enhanced.

Thoughts? No? I didn’t think so, no one reads this blog.

Easytether running on ARMv7l Chromebook

Easytether is a proprietary application, so you can’t just apt-get it on a chroot’ed crouton environment. Additionally, I am not running an x86 or x64 CPU, so most of these packages will not work.

Crouton mostly comes for Debian, so I downloaded the Ubuntu package. I installed the package ‘alien’. Then unzipped the deb file as follows:

alien easytether_0.8.5-1_armhf.deb

This will 3 tar.xz files, mostly importantly data.tar.xz. Unzip it with the following command:

tar xvfJ data.tar.xz

Now go into the newly created usr/bin directory and execute the easytether-usb command as root and it should work.