Linux kernel code vs FreeBSD kernel code


Linux driver code contains some serious garbage. I heard this refrain, but I did not realize how bad it was until I looked at it myself. Here is just one example.

Device drivers typically read static memory, typically known as EEPROM or ROM, from the chip to identify version, hard-coded information, device capabilities, etc. These values are used throughout execution of the driver. The reading process is among the first things when the device is attached and powered on.

In the case of FreeBSD, after the kernel reads the ROM, it uses a struct pointer with all the variables pre-populated, and points it at the ROM blob data stored in memory. For example:

struct r88e_rom {
	uint8_t		reserved1[16];
	uint8_t		cck_tx_pwr[R88E_GROUP_2G];
	uint8_t		ht40_tx_pwr[R88E_GROUP_2G - 1];
	uint8_t		tx_pwr_diff;
	uint8_t		reserved2[156];
	uint8_t		channel_plan;
	uint8_t		crystalcap;
#define R88E_ROM_CRYSTALCAP_DEF		0x20

	uint8_t		thermal_meter;
	uint8_t		reserved3[6];
	uint8_t		rf_board_opt;
	uint8_t		rf_feature_opt;
	uint8_t		rf_bt_opt;
	uint8_t		version;
	uint8_t		customer_id;
	uint8_t		reserved4[3];
	uint8_t		rf_ant_opt;
	uint8_t		reserved5[6];
	uint16_t	vid;
	uint16_t	pid;
	uint8_t		usb_opt;
	uint8_t		reserved6[2];
	uint8_t		macaddr[IEEE80211_ADDR_LEN];
	uint8_t		reserved7[2];
	uint8_t		string[33];	/* "realtek 802.11n NIC" */
	uint8_t		reserved8[256];
} __packed;

_Static_assert(sizeof(struct r88e_rom) == R88E_EFUSE_MAP_LEN,
    "R88E_EFUSE_MAP_LEN must be equal to sizeof(struct r88e_rom)!");

Notice the assertion at the bottom, which ensures that the ROM struct’s size equals a pre-defined length. The code will fail to compile if this assertion is not valid. Later, the kernel will instantiate a struct pointer and point it to the ROM, stored in the variable buf, as follows:

struct r88e_rom *rom = (struct r88e_rom *)buf;

Now, rom->channel_plan is set to the correct value. Simple.

Unfortunately, this is not how the same code is written on Linux. As mentioned, the Linux driver also begins by reading the ROM blob and storing it in a value called hwinfo. But rather than creating an equivalent struct pointer, the Linux code uses offset values of the ROM on an as-needed basis. For example, the driver reads the channel_plan as follows:

rtlefuse->eeprom_version = *(u16 *)&hwinfo[params[7]];

In this example, params[7] comes from a list of ROM offsets values set in the previous calling function. (That alone made tracing difficult.) The rtlefuse->eeprom_version is now the same as FreeBSD’s rom->version. This manual process repeats for every variable in the ROM.

While that may be just annoying and require a negligible bit more CPU power, this is not be a problem if it was done all in one place. But instead, the driver reads from the hwinfo blob on a seemingly as-needed during execution. And because these as-needed instances are during normal execution, the driver reads-in the same static value from hwinfo every a simple WiFi function occurs, such as changing the channel.

Okay, but even that might not be too difficult…right? Here’s the real kicker.

Sometimes, the driver works by using incrementing offsets from the ROM blob. For example, consider at read_power_value_fromprom (in drivers/net/wireless/realtek/rtlwifi/hw.c). It initializes eeaddr as a u32 (uint32_t), then assigns it with the offset value EEPROM_TX_PWR_INX. So far so good. But then, rather than using new offsets for every successive value, it increments the eeaddr value in multiple doubly-nested for-loops. Here is a simplified version of the code:

for (rfpath = 0 ; rfpath < MAX_RF_PATH ; rfpath++) {
		/*2.4G default value*/
		for (group = 0 ; group < MAX_CHNL_GROUP_24G; group++) { pwrinfo24g->index_cck_base[rfpath][group] =
			  hwinfo[eeaddr++];
			if (pwrinfo24g->index_cck_base[rfpath][group] == 0xFF)
				pwrinfo24g->index_cck_base[rfpath][group] =
				  0x2D;
		}
}

Notice the line hwinfo[eeaddr++]! Merely reading in that variable changes the offset. Its the Heisenberg Uncertainty Principle equivalent of code. This is a cleaned-up version of the 188-line function. The actual function has 6 nested for-loops, some with if-statements, each incrementing the eeaddr parameter as they go along.

Why would anyone do it this way? You are needlessly using up the CPU, making the code difficult to follow, repeatedly reading in static values and making any minor modifications and re-ordering or re-structuring will essentially break the entire function.

And perhaps the worst offender is when 20 functions deep you are not even working with hwinfo anymore. You are working to a pointer to hwinfo that has been incremented God-knows where, with their own offsets that are near impossible to track down.

In my efforts to port this driver to FreeBSD, I literally resorted to printing out the entire ROM, manually finding the memory, and backing into the equivalent offset. Other bizarre code: I have seen if-conditions that are impossible to reach, misplaced code that should go in the previous function, code that does bits of a tasks, while another function does the entire task – so repeat code, unnecessarily repeated code, etc.

How does this make it into the Linux Kernel?

To be fair, this does not appear to be the fault of Larry Finger, who maintains this driver. This is the fault of Realtek, for vomiting this terrible driver in the first place, providing absolutely zero documentation and refusing to respond to any contact attempts.

I hope my FreeBSD port is cleaner and more performant!

Advertisements

Switched from Ubuntu-based to Fedora


tl;dr: Fedora’s debugging packages work, Ubuntu’s are out of date.

Linux = Linux = Linux, whether Arch or Slackware or Ubuntu or OpenSUSE or Linux from scratch as I once did (before there were instructions!). Unless and until the kernel forks and someone decides to modify the syscall table, they all use the same basic syscalls, they typically share the same basic libraries and core utilities, etc. They’re all the same.

Why did I use Ubuntu-based distributions? (Note: Not Debian) Because Ubuntu came pre-configured with all the things I did not care to learn or manually configure: ACPI, firmware, X11, a pretty WM theme, etc. I did not particularly care whether I was running Mint, Elementary, Ubuntu MATE or basic Ubuntu (except Unity…nah). As long as it did not do strange things like remove /sbin/ifconfig or have a radically different file structure than I was used to. I felt at home with knowing where the standard file paths were, and knew how to administer my machine. Their package repository was pretty solid. It had almost everything I wanted – and what little was not on it was typically available in Debian-package format. The broader Linux community effectively standardized on this package format. This is crucial. Debian’s apt and FreeBSD’s pkg are in my muscle-memory at this point.

Literally one thing pushed me over: Ubuntu’s SystemTap was broken. Utterly broken!

I got into OS-level programming, specifically, porting a Linux WiFi driver to FreeBSD. I wanted to use SystemTap, Linux’s answer to DTrace, to help understand what is going on during live execution. But SystemTap does not work on Ubuntu – at least currently.

But wait, I thought Linux = Linux = Linux and programs from 20 years ago will still work. Why does SystemTap fail?

SystemTap works by producing C code for a kernel module, compiling it and loading it into memory. Sometime ago, the kernel team changed the get_user_pages() kernel API call. This meant that any code compiled against the old function definition failed. I encountered this in the professional space when the VMWare kernel modules failed to build and I hacked it until it worked. (They think I’m a wizard now). I was on Kernel 4.10 but the version of systemtap Ubuntu used was nearly 2 years old. This meant no one from the Ubuntu team was using it.

I submitted a bug report and installed Fedora 26.

SystemTap was developed by Red Hat and was trivial to get working under Fedora. And while not every single package is available (Bitcoin, Steam thus far), there is enough that moving over was trivial. Also, they come in Cinnamon, which I prefer, with a pretty theme. And it provides a clean terminal out-of-the-box. Which I need. (I would rather use stock XFCE if their terminal was clean than fully-loaded CentOS with an ugly terminal)

dnf took a little getting used to, but a hop-over from apt. So whatever on that front….

I would be willing to try OpenSUSE again, but the latest time I did, they got rid of /sbin/ifconfig for /sbin/ip, which is unacceptable. Silly, perhaps…Does it come in Cinammon? What does it offer? Are the packages as clean and up to date? I may never know, unless another business-need arises. I do not care to run any of these “hardware” distributions, like Arch. I paid my Linux dues around kernel 2.2 on Slackware and its time to move on from that.

Thoughts?

But look, if you’re 99% of the Linux world, any specific distribution is trivial. Pick one and go with it. Unless you’re doing very specific tasks like me, it really does not matter what you use. So stop Distro Hoping!

FreeBSD and Linux Remote Dual Booting


The following is a quick and dirty guide on how to setup remote dual booting for FreeBSD (12.0-CURRENT) and Linux (Ubuntu 16.04). Granted, this method is slightly a hack, but it works and suits my needs.

Why remote dual-booting? I am currently developing a FreeBSD kernel module for a PCIe card. The device is supported on Linux and I am using the Linux implementation as documentation. As such, I find myself frequently rebooting into Linux to look printk() outputs, or booting into FreeBSD to test kernel code. This device is located at my house, and I typically work on it during my downtime at work.

Why not use Grub? I would have preferred Grub! But for whatever reason, Grub failed to install on FreeBSD. I do not know why, but even a very minimalistic attempt gave a non-descriptive error message.

efibootmgr? Any change I made with efibootmgr failed to survive a reboot. This is apparently a known problem. Also, this tool only exists on Linux, as FreeBSD does not seem to have an efibootmgr equivalent.

Ugh, so what do I do???

The solution I came up with was to manually swap EFI files on the EFI partition no an as-needed basis.

First, I went into the BIOS and disabled legacy BIOS booting, enabled EFI booting, and disabled secure booting.

Then, I installed Ubuntu. I had to manually create the partition tables, since by default the installer would consume the entire disk. However, this does not automatically create the EFI partition. So, you must manually create one. I set mine to 200MBs as the first partition. After installation, I booted up, mounted the /dev/sda1. I found that ubuntu had created /EFI/ubuntu/grubx64.efi and other related files. Great!

Next, I installed FreeBSD and while manually setting up the partition tables, FreeBSD auto-created an EFI partition. One already exists, so I safely deleted it, and proceeded with the rest of the install. Right before rebooting, I mounted /dev/ada0p1 (sda1 on Linux) as /boot.local/ and /dev/da0p1 as /boot.installer/. I then copied /boot.installer/EFI/BOOT/BOOTX64.EFI too /boot.local/EFI/BOOT/EFIBOOT/BOOTX64.EFI (I think I had to re-create EFI/BOOT, I’m forgetting off-hand). Then I rebooted.

When I rebooted the machine, Ubuntu still came up. This is because Ubuntu edits the EFI boot order and places ubuntu as the first partition. Ordinarily you should be able to use efibootmgr here to boot into FreeBSD and use the non-existent FreeBSD equivalent to boot back, but with the lack of that option, I mounted the EFI partition (/dev/sda1) as /boot/efi, and when I wanted boot into FreeBSD, I renamed /boot/efi/EFI/ubuntu/grubx64.efi to ubuntu.efi and then copied /boot/efi/EFI/BOOT/BOOTX64.EFI to /boot/efi/EFI/ubuntu/grubx64.efi. When I rebooted, FreeBSD came back up! Then on the FreeBSD side, I mounted /dev/sda1 to /boot/efi and did copied /boot/efi/EFI/ubuntu/ubuntu.efi to /boot/efi/EFI/ubuntu/grubx64.efi.

And that’s it! I can now remotely boot back and forth between the two systems.

Ugly? Yes. But it does the job.

Linux could fix this problem by debugging their efibootmgr utility and FreeBSD could fix this by having an efibootmgr equivalent at all.

Thoughts?

Differences between Mint and Ubuntu


 

I looked into the differences between Mint and Ubuntu to see which was best for me. I watched tons of videos, reviews, comparisons, ran them both for months, etc. Here’s what I learned…

They’re the same damn thing. No really, they are identical. The only differences are what software comes pre-installed and some user interface prettiness. Otherwise, no difference in the  underlying system at all. Literally no difference.

Next question?

BSD? No thanks, I’ll stick with Linux


Warning, this is an unstructured Rant.

I’m going to be setting up FreeNAS to manage my home’s personal NAS server. I spent a bit of time looking up the hardware and learning how FreeNAS works. I also really like that its a FreeBSD system at the back-end, so that I can always configure it exactly how I want. This resparked an interest in me to look into the BSD variants to see if they would be better for me.

TL;DR – No, I’m sticking with Linux.

People talk about BSD being “more mature”. But I have yet to get a solid understanding of what this means to the end-user, where Linux seems pretty damn mature to me. I haven’t had Linux crash on me since some bug in kernel 2.6.

Or is the code more reviewed? The Linux kernel is heavily, heavily reviewed.

Or is BSD an OS whereas Linux is a Kernel? Okay, I use Debian distributions, not just a raw Linux kernel. And besides, there are basically 4 main distributions that are largely differ on trivial matters behind the scenes. But you have like 5 main variants of BSD – OpenBSD, FreeBSD, NetBSD, DragonflyBSD and now MacOS X. These OS’s have, in some cases 20 years of variations between them, whereas at least all Linux’s share a common kernel.

And another thing – why do people treat BSD as some common platform, as if they’re all the same? Do the 20+ years of divergence mean nothing?

NetBSD likes to brag about being able to run on a toaster. Great. But in the year 2015, I don’t need to run on a computer from 1986. I don’t plan on running a Unix/Linux system on a Playstation.

Okay, so OpenBSD has security built-in. Are you talking the security in packages or the security of the OS? If you mean the security of the packages, this can be trivially done on Linux. If you mean the security of the OS, you’re talking about memory protection and related features that can, and are, very easily ported over to Linux.

Or is it that BSD systems are for servers? I currently work for a company that measures its internet presence of IP space in the percentages, and we use Redhat or CentOS – and legacy AIX and other random systems.

okay okay, maybe BSD is great for niche devices. But I don’t see its benefit anywhere over Linux anywhere else. I would have picked the FreeNAS equivalent for Linux if I could — well no I prob wouldn’t, I’d wanna see what BSD can do.

oh, and if you’re going to give me a very technical response to why BSD is better, allow me to preempt your response:

  • If your responses are related to something buried well behind the scenes of the system that are trivial to the end-user, you lose.
  • If your responses are blaming the user of the system for not being responsible by running the hardware or software you choose and running the system as you wish, you lose.
  • If your responses are about how BSD invented stuff that Linux stole, I’ll say you lose because you aren’t using the OS that invented the mouse.

End-rant. Okay, I’m gonna install PC-BSD on my spare beat-up computer.

Syncing my Bluetooth headset on Kubuntu


I had a bit of trouble getting my Linux (Kubuntu) to function as a bluetooth headset for my computer and I wanted to write this guide for anyone else having trouble.

First, I’m running Ubuntu 14.04.1 LTS

Second, get your device to pair. I used Blueman, an applet that came with Kubuntu.

snapshot1Getting it to pair was annoying enough, but once I got that working, I clicked Setup, then paired it as a Headset. I then edited the file /etc/bluetooth/audio.conf and changed the following:

HFP=false

It was True. I then Uncommented

#[A2DP]
SBCSources=1
MPEG12Sources=0

Then, I rebooted pulseaudio by doing a simple:

sudo killall -9 pulseaudio

and then restarted pulseaudio as root. Per the folks on Freenode’s #Linux I could have also started a program that uses pulseaudio, but this worked. Afterwards, audio was playing on both my computer’s regular speakers and the headset.

Only downside: When I play a game like Counter-strike, there is a very slight delay between the action and the audio. I have no idea what the cause of that is.