Introduction
In
this series of articles I describe how you can write a Linux loadable
kernel module (LKM) for an embedded Linux device. This is the second
article in the series — please read “Writing a Linux Kernel Module — Part 1: Introduction”
before moving on to this article, as it explains how to build, load and
unload loadable kernel modules (LKMs). Such description is not repeated
in this article.
Character Device Drivers
A character device
typically transfers data to and from a user application — they behave
like pipes or serial ports, instantly reading or writing the byte data
in a character-by-character stream. They provide the framework for many
typical drivers, such as those that are required for interfacing to
serial communications, video capture, and audio devices. The main
alternative to a character device is a block device. Block
devices behave in a similar fashion to regular files, allowing a
buffered array of cached data to be viewed or manipulated with
operations such as reads, writes, and seeks. Both device types can be
accessed through device files that are attached to the file system tree.
For example, the program code that is presented in this article builds
to become a device /dev/ebbchar, which appears on your Linux system as follows:
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ lsmod
Module Size Used by
ebbchar 2754 0
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ ls -l /dev/ebb*
crw-rw-rwT 1 root root 240, 0 Apr 11 15:34 /dev/ebbchar
This
article describes a straightforward character driver that can be used
to pass information between a Linux user-space program and a loadable
kernel module (LKM), which is running in Linux kernel space. In this
example, a C user-space application sends a string to the LKM. The LKM
then responds with the message that was sent along with the number of
letters that the sent message contains. Later in the article I describe
why we need to solve synchronization problems that arise with this
approach, and I provide a version of the program that uses mutex locks
to provide a solution.
Before
describing the source code for the driver in this article, there are a
few concepts that need to be discussed, such as device driver major and
minor numbers, and the File Operations data structure.
Major and Minor Numbers
Device drivers have an associated major and minor number. For example, /dev/ram0 and /dev/null are associated with a driver with major number 1, and /dev/tty0 and /dev/ttyS0
are associated with a driver with major number 4. The major number is
used by the kernel to identify the correct device driver when the device
is accessed. The role of the minor number is device dependent, and is
handled internally within the driver. You can see the major/minor number
pair for each device if you perform a listing in the /dev directory. For example:
Character devices are identified by a ‘c‘ in the first column of a listing, and block devices are identified by a ‘b‘. The access permissions, owner, and group of the device is provided for each device. Regular user accounts on the BeagleBone are members of some of these groups and therefore have permissions to use the i2c-0 and ttyS0 devices etc. See:
The device that is developed in this article appears as a device (/dev/ebbchar) in the /dev directory.
molloyd@beaglebone:/dev$ ls -l
crw-rw---T 1 root i2c 89, 0 Jan 1 2000 i2c-0
brw-rw---T 1 root disk 1, 0 Mar 1 20:46 ram0
brw-rw---T 1 root floppy 179, 0 Mar 1 20:46 mmcblk0
crw-rw-rw- 1 root root 1, 3 Mar 1 20:46 null
crw------- 1 root root 4, 0 Mar 1 20:46 tty0
crw-rw---T 1 root dialout 4, 64 Mar 1 20:46 ttyS0
…
Character devices are identified by a ‘c‘ in the first column of a listing, and block devices are identified by a ‘b‘. The access permissions, owner, and group of the device is provided for each device. Regular user accounts on the BeagleBone are members of some of these groups and therefore have permissions to use the i2c-0 and ttyS0 devices etc. See:
molloyd@beaglebone:/dev$ groups
molloyd dialout cdrom floppy audio video plugdev users i2c spi
The device that is developed in this article appears as a device (/dev/ebbchar) in the /dev directory.
It is possible to manually create a block or character device file entry and later associate it with your device (e.g.,
sudo mknod /dev/test c 92 1
),
but this approach is prone to problems. One such problem is that you
have to ensure that the number you choose (e.g., 92 in this case) is not
already in use. On the BeagleBone, you could examine the file /usr/src/linux-headers-3.8.13-bone70/include/uapi/linux/major.h
for a list of all system device major numbers. However, a device that
idenfies a “unique” major number using this approach would not be very
portable, as the major number of the device could clash with that of
another device on another Linux SBC or Linux distribution. The code that
is provided in this article automatically identifies an appropriate
major number to use.The File Operations Data Structure
The
file_operations
data structure that is defined in /linux/fs.h
holds pointers to functions (function pointers) within a driver that
allows you to define the behavior of certain file operations. For
example, Listing 1 is a segment of the data structure from /linux/fs.h. The driver in this article provides an implementation for the read
, write
, open
, and release
system call file operations. If you do not provide an implementation
for one of the entries in this data structure then it will simply point
to NULL
, making it inaccessible. Listing 1 is somewhat
intimidating, given the number of operations available. However, to
build the ebbchar LKM we only need to provide an implementation for four
of the entries. Therefore, Listing 1 is provided mainly as a reference
that you can use if you need to provide additional functionality within
the driver framework.file_operations
Data Structure of the /linux/fs.h (Segment)
Get Source Code
Source Code for this Discussion
All
of the code for this discussion is available in the GitHub repository
for the book Exploring BeagleBone. The code can be viewed publicly at:
the ExploringBB GitHub Kernel Project directory, and/or you can clone the repository on your BeagleBone (or other Linux device) by typing:
The /extras/kernel/ebbchar
directory is the most important resource for the next part of this
article. The auto-generated Doxygen documentation for these code
examples is available in HTML format and PDF format.
The Device Driver Source Code
The source code for the ebbchar device driver is provided in Listing 2. Similar to the code in the first article in this series, there is an
init()
function and an exit()
function. However, there are additional file_operations
functions that are required for the character device:dev_open()
: Called each time the device is opened from user space.dev_read()
: Called when data is sent from the device to user space.dev_write()
: Called when data is sent from user space to the device.dev_release()
: Called when the device is closed in user space.
Drivers have a class name and a device name. In Listing 2, ebb (Exploring BeagleBone) is used as the class name, and ebbchar as the device name. This results in the creation of a device that appears on the file system at /sys/class/ebb/ebbchar.
In addition to the points described by the comments in Listing 2, there are some additional points:
- This code has a fixed message size of 256 characters — this will be improved in later articles though the dynamic allocation of memory.
- This code is not multi-process safe — that is addressed later in this article.
- The
ebbchar_init()
function is much longer than the last article. That is because it is now automatically allocating a major number to the device, registering the device class, and registering the device driver. Importantly, you will notice that if anything goes wrong that the code carefully “backs out” of the successful operations. To achieve this I have repeated code (which I always dislike), but the alternative is to usegoto
statements, which is even less palatable (albeit slightly tidier). - The
PTR_ERR()
is a function that is defined inlinux/err.h
that retrieves the error number from the pointer. - The functions
sprintf()
andstrlen()
are available in the kernel through the inclusion of linux/kernel.h and indirectly through linux/string.h respectively. The functions in string.h are architecture dependent.
Building and Testing the LKM
A Makefile is required to build the LKM, as provided in Listing 3. This Makefile is very similar to the Makefile in the first article in the series, with the exception that it also builds a user-space C program that interacts with the LKM.
Listing 4 is a short program that requests a string from the user, and writes it to the /dev/ebbchar device. After a subsequent key press (ENTER) it then reads the response from the device and displays it in the terminal window.
In addition to the points described by the comments in Listing 4, there are some additional points:
- The
%[^\n]%*c
uses the scanset specifiers, which are represented by%[]
to use the^
character to stop reading after the first occurrence of the\n
character. In addition, the%*c
ignores the trailing character, ensuring that the subsequentgetchar()
function works as required. Essentially, thescanf()
code just reads in a sentence. Ifscanf()
was used with a regular%s
call then the string would terminated at the first occurrence of the space character. - The
getchar()
allows the program to pause at that point until the ENTER key is pressed. This is necessary to demonstrate a problem with the current code formulation. - The program then reads the response from the LKM and displays it in the terminal window.
All
going well, the process to build the kernel module should be
straightforward, provided that you have installed the Linux headers as
described in the first article. The steps are as follows:
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ make
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ ls -l *.ko
-rw-r--r-- 1 molloyd molloyd 7075 Apr 8 19:04 ebbchar.ko
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ ls -l test
-rwxr-xr-x 1 molloyd molloyd 6342 Apr 8 19:23 test
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ sudo insmod ebbchar.ko
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ lsmod
Module Size Used by
ebbchar 2521 0
The device is now present in the /dev directory, with the following attributes:
You can see that the major number is 240 — this is automatically assigned by the code in Listing 2.
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ cd /dev
molloyd@beaglebone:/dev$ ls -l ebb*
crw------- 1 root root 240, 0 Apr 8 19:28 ebbchar
You can see that the major number is 240 — this is automatically assigned by the code in Listing 2.
We can then use the testebbchar program (Listing 4) to test that the LKM is working correctly. For the moment, the test program must be executed with root privileges — that issue is addressed shortly.
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ sudo insmod ebbchar.ko
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ sudo ./test
Starting device test code example...
Type in a short string to send to the kernel module:
This is a test of the ebbchar LKM
Writing message to the device [This is a test of the ebbchar LKM].
Press ENTER to read back from the device...
Reading from the device...
The received message is: [This is a test of the ebbchar LKM(33 letters)]
End of the program
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ sudo rmmod ebbchar
The
You can see that 33 characters are sent to the LKM but 45 characters are returned — this is due to the addition of the 12 characters “
printk()
output can be viewed by examining the kernel logs as follows: molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ sudo tail -f /var/log/kern.log
Apr 11 22:24:50 beaglebone kernel: [358664.365942] EBBChar: Initializing the EBBChar LKM
Apr 11 22:24:50 beaglebone kernel: [358664.365980] EBBChar: registered correctly with major number 240
Apr 11 22:24:50 beaglebone kernel: [358664.366061] EBBChar: device class registered correctly
Apr 11 22:24:50 beaglebone kernel: [358664.368383] EBBChar: device class created correctly
Apr 11 22:25:15 beaglebone kernel: [358689.812483] EBBChar: Device has been opened 1 time(s)
Apr 11 22:25:31 beaglebone kernel: [358705.451551] EBBChar: Received 33 characters from the user
Apr 11 22:25:32 beaglebone kernel: [358706.403818] EBBChar: Sent 45 characters to the user
Apr 11 22:25:32 beaglebone kernel: [358706.404207] EBBChar: Device successfully closed
Apr 11 22:25:44 beaglebone kernel: [358718.497000] EBBChar: Goodbye from the LKM!
You can see that 33 characters are sent to the LKM but 45 characters are returned — this is due to the addition of the 12 characters “
(33 letters)
”
to the string data that specifies the length of the string that was
originally sent. This addition is performed as a test in order to be
certain that the code is sending and receiving unique data.
There
are two significant problems with the current LKM. The first is that
the LKM device can only be accessed with superuser permissions, and the
second is that the current LKM is not multi-process safe.
User Access to the Device using Udev Rules
Throughout
this article, the program that interfaces to the LKM device is executed
using sudo. It would be very useful to set up our LKM device so that it
can be accessed by a particular user or group, while still protecting
the file system. To address this issue, you can use an advanced feature
of Linux called udev rules that enables you to customize the
behavior of the udevd service. This service gives you some user-space
control over devices on your Linux system.
For example, to give user-level access to the ebbchar device, the first step is to identify the sysfs entry for the device. You can achieve this by using a simple find:
root@beaglebone:/sys# find . -name "ebbchar"
./devices/virtual/ebb/ebbchar
./class/ebb/ebbchar
./module/ebbchar
We then need to identify the KERNEL and SUBSYSTEM values about which to write the rule. You can use the udevadm command to perform this task:
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ udevadm info -a -p /sys/class/ebb/ebbchar
Udevadm info starts with the device specified by the devpath and then walks up the chain of parent
devices. It prints for every device found, all possible attributes in the udev rules key format. A
rule to match, can be composed by the attributes of the device and the attributes from one single
parent device.
looking at device '/devices/virtual/ebb/ebbchar':
KERNEL=="ebbchar"
SUBSYSTEM=="ebb"
DRIVER==""
The rules are contained in the /etc/udev/rules.d
directory. A new rule can be added as a file using these values, where
the file begins with a priority number. Using a name such as 99-ebbchar.rules
creates a new rule with the lowest priority, so that it does not
interfere with other device rules. The rule can be written as in Listing
5 and placed in the /etc/udev/rules.d directory as follows:
molloyd@beaglebone:/etc/udev/rules.d$ ls
50-hidraw.rules 50-spi.rules 60-omap-tty.rules 70-persistent-net.rules 99-ebbchar.rules
molloyd@beaglebone:/etc/udev/rules.d$ more 99-ebbchar.rules
#Rules file for the ebbchar device driver
KERNEL=="ebbchar", SUBSYSTEM=="ebb", MODE="0666"
Once the rules file is added to your system, you can test it using:
You can see that user and group now have the permissions required to read from and write to this device. Interestingly, the sticky bit is also set. The sticky bit usually means that write permission is not sufficient to delete files. Therefore, in the /tmp directory any user can create files, but no user can delete another user’s files. The sticky bit is represented by a capital T in the final character place. This usually appears as a lower-case t unless the executable (x) bit for others is set; however, when the x bit is not set it appears as a capital T. However, it is not entirely clear why the sticky bit is being set by udev — it appears to be unusual to udev rules under Debian. At this point the test application can be executed without requiring superuser permissions.
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ sudo insmod ebbchar.ko
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ ls -l /dev/ebbchar
crw-rw-rwT 1 root root 240, 0 Apr 9 00:44 /dev/ebbchar
You can see that user and group now have the permissions required to read from and write to this device. Interestingly, the sticky bit is also set. The sticky bit usually means that write permission is not sufficient to delete files. Therefore, in the /tmp directory any user can create files, but no user can delete another user’s files. The sticky bit is represented by a capital T in the final character place. This usually appears as a lower-case t unless the executable (x) bit for others is set; however, when the x bit is not set it appears as a capital T. However, it is not entirely clear why the sticky bit is being set by udev — it appears to be unusual to udev rules under Debian. At this point the test application can be executed without requiring superuser permissions.
The strace Command
The strace
command is a very useful debugging tool that can execute a program in
order to intercept and record the system calls that it performs. The
system call name, the arguments passed, and the resulting return value
are all visible, which makes it a valuable tool for solving runtime
issues. Importantly, you do not need the source code for the executable
in order to view the output of strace. For example, you can utilize
strace on your user-space application in order to view the communication
between the user-space program and the kernel module, which results in
the following for the test application:
The system call output gives us impressive insight into the communication that takes place between the user-space program test and the /dev/ebbchar device driver.
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ sudo apt-get install strace
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ strace -v
usage: strace [-dffhiqrtttTvVxx] [-a column] [-e expr] … [-o file]
[-p pid] … [-s strsize] [-u username] [-E var=val] …
[command [arg …]]
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ sudo strace ./test
execve("./test", ["./test"], [/* 15 vars */]) = 0
…
write(1, "Starting device test code exampl"..., 37Starting de…) = 37
open("/dev/ebbchar", O_RDWR) = 3
write(1, "Writing message to the device [T"..., 60Writing message …) = 60
write(3, "Testing the EBBChar device", 26) = 26
write(1, "Reading from the device...\n", 27Reading from the device…) = 27
read(3, "", 100) = 0
write(1, "The received message is: [Testin"..., 66The received …) = 66
write(1, "End of the program\n", 19End of the program) = 19
exit_group(0) = ?
The system call output gives us impressive insight into the communication that takes place between the user-space program test and the /dev/ebbchar device driver.
LKM Synchronization Problems
There is a serious problem with the LKM that is described in Listing 2. In the first article
in this series I pointed out that LKMs do not execute sequentially and
that they can be interrupted. Those facts have important consequences
for the code that is written in this article, which can be demonstrated
as follows:
Step 1: At the first terminal window shell you can execute the test application, but do not allow it to run to completion (by not pressing ENTER when prompted), as follows:
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ ./test
Starting device test code example...
Type in a short string to send to the kernel module:
This is the message from the first terminal window
Writing message to the device [This is the message from the first terminal window].
Press ENTER to read back from the device...
Step 2: Then at a second terminal window shell you can execute the same test application simultaneously as a second process on the Linux device. For example:
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ ./test
Starting device test code example...
Type in a short string to send to the kernel module:
This is the message from the second terminal window
Writing message to the device [This is the message from the second terminal window].
Press ENTER to read back from the device...
Step 3:
You can then return to the first terminal window shell and press ENTER
to run the program to completion, which results in the following output
(shaded output is the repeated output from Step 1):
You can see that the received message is actually the message that was sent by the test application from Step 2, which is running in the second terminal window shell (not the first as might be expected). This is because the message that was sent in Step 2 overwrote the string message that was being stored by the LKM as a result of Step 1.
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ ./test
Starting device test code example...
Type in a short string to send to the kernel module:
This is the message from the first terminal window
Writing message to the device [This is the message from the first terminal window].
Press ENTER to read back from the device...
Reading from the device...
The received message is: [This is the message from the second terminal window(51 letters)]
End of the program
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$
You can see that the received message is actually the message that was sent by the test application from Step 2, which is running in the second terminal window shell (not the first as might be expected). This is because the message that was sent in Step 2 overwrote the string message that was being stored by the LKM as a result of Step 1.
Step 4:
You can return to the second terminal shell and run it to completion by
pressing ENTER, which results in (the shaded output is the repeated
output from Step 2):
No string is received. That is because the LKM is not storing any messages at that point in time. It has already delivered the stored message to the first terminal window test application and reset the buffer index to 0.
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$ ./test
Starting device test code example...
Type in a short string to send to the kernel module:
This is the message from the second terminal window
Writing message to the device [This is the message from the second terminal window].
Press ENTER to read back from the device...
Reading from the device...
The received message is: []
End of the program
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbchar$
No string is received. That is because the LKM is not storing any messages at that point in time. It has already delivered the stored message to the first terminal window test application and reset the buffer index to 0.
Adding Mutex Locks
The Linux kernel provides a full implementation of semaphores — a data type (
struct semaphore
)
that is used for controlling access by multiple processes to a shared
resource. The easiest way to use this semaphore code within the kernel
is to use mutexes, as there is a full set of helper functions and macros
available.
A simple way to prevent the problems described above is to prevent two processes from using the /dev/ebbchar
device at the same time. A mutex is a lock that can set (put down)
before a process begins using a shared resource. The lock can then be
released (brought up) when the process is finished using the shared
resource. When the lock has been set, no other process can access the
locked code region. Once the mutex lock has been released by the process
that locked it, the shared region of code is once again available to be
accessed by the other process, which in turn locks the resource.
There are only very minor changes required to the code in order to
implement mutex locks. An outline of these changes is provided in
Listing 6 below:
The full source code example is available in the /extras/kernel/ebbcharmutex/ directory. The final LKM is called ebbcharmutex.c.
If you now perform the same test on the code that contains the mutex
locks, you will observe a different behavior. For example:
Step 1: Using the first terminal window shell you can load the module and execute the test application, which results in the following output:
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbcharmutex$ sudo insmod ebbcharmutex.ko
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbcharmutex$ lsmod
Module Size Used by
ebbcharmutex 2754 0
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbcharmutex$ ls -l /dev/ebb*
crw-rw-rwT 1 root root 240, 0 Apr 12 15:26 /dev/ebbchar
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbcharmutex$ ./test
Starting device test code example...
Type in a short string to send to the kernel module:
Testing the ebbchar LKM that has mutex code
Writing message to the device [Testing the ebbchar LKM that has mutex code].
Press ENTER to read back from the device...
Step 2: Then using the second terminal window shell you can attempt to execute the test application to create the second process:
As expected and required, the second process fails to access the device.
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbcharmutex$ ./test
Starting device test code example...
Failed to open the device...: Device or resource busy
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbcharmutex$
As expected and required, the second process fails to access the device.
Step 3: Returning to the first terminal window, the program can be allowed to run to completion by pressing ENTER:
At this point the second terminal window shell can now execute the test program, whereupon it will acquire the mutex lock and run correctly.
molloyd@beaglebone:~/exploringBB/extras/kernel/ebbcharmutex$ ./test
Starting device test code example...
Type in a short string to send to the kernel module:
Testing the ebbchar LKM that has mutex code
Writing message to the device [Testing the ebbchar LKM that has mutex code].
Press ENTER to read back from the device...
Reading from the device...
The received message is: [Testing the ebbchar LKM that has mutex code(43 letters)]
End of the program
At this point the second terminal window shell can now execute the test program, whereupon it will acquire the mutex lock and run correctly.
Conclusion
Click for the HTML and PDF version of the auto-generated Doxygen code documentation
There are several different issues described in this article. The important outcomes of this article are that:
- You can now create your own device such as /dev/ebbchar, which you can write information to and read information from. This is important, as it provides a bridge between the Linux user space and the Linux kernel space. It enables you to develop advanced drivers, such as communication devices, which can be controlled by C code that is running in user space.
- You can appreciate why it is important to be aware of the synchronization issues that can arise with kernel module programming.
- You can use udev rules to alter the properties of a device as it is loaded.
The
next article in this series examines how you can interface to GPIO
devices, such as physical button and LED circuits, enabling you to write
your own custom drivers for embedded Linux applications that interface
to custom hardware. You can read that article here: Writing a Linux
Kernel Module — Part 3: Interfacing to GPIOs (coming soon!). Finally,
Listing 7 and Listing 8 are provided for your reference — they provide a
list of the standard error states that are used in this series of
articles, along with their associated error numbers and descriptions.