In this post we’ll be looking at how Linux handles accesses to memory devices and I/O from userspace, and comparing the default method used for development (/dev/mem) against the one better prepared for deployment (/dev/uio). We’ll also look at how to reserve memory and make it available for DMA use.

Legacy device drivers

The character device driver /dev/mem exists in the kernel to map device memory into user space. This kind of access is a security risk, as it allows any process to access kernel memory, and can be used only by the root user. Moreover, a bug in the user space driver could crash the kernel. Memory access can be disabled in the kernel configuration, using the CONFIG_STRICT_DEVMEM kernel setting. This a great tool for prototyping or testing new hardware but not considered an acceptable production solution for a user space device driver.

The legacy device driver (/dev/mem) requires the user space application to know at build time the physical address location and size of the mapped memory region. The programmable logic can re-map these regions inadvertently while adding a new device, for example, and losing access to it from software. This needs to be tightly controlled with the target software to avoid disconnects.

The userspace software application would map the memory after opening the /dev/mem device as follows:

fd=open("/dev/mem",O_RDWR);
...
ptr=mmap(NULL,page_size,PROT_READ|PROT_WRITE,MAP_SHARED,fd,(addr & ~(page_size-1)));
...
*((unsigned *)(ptr+page_offset))=val;

There is a good example of this in the userspace application shipped with Petalinux: peekpoke.

The biggest drawback from this legacy driver method is its inability to handle interrupts. There’s simply no formal way to have interrupts working from a mapped device using the legacy device driver.

UIO device drivers

The kernel provides a generic character device that requires no extra kernel driver code that provides several advantages over legacy drivers. The device tree is used to mark those mapped device regions and assign them a generic UIO device, by adding the “compatible = uio-generic” setting to the mapped device. A new character device is created under the /sys/class/uio/uioX filesystem, and the corresponding device /dev/uioX.

The name of the device may be added to the device tree entry, and it would be reflected in a sysfs file under /sys/class/uio/uioX/maps/mapY/name. The device address and size are also defined in the device tree, and can be read from sysfs in a similar way.

&gpio {
  compatible = "generic-uio";
  reg = <0x0 0xa0000000 0x0 0x10000>;
};

The bootargs may need to be modified to add support for generic-uio, with the setting uio_pdrv_genirq.of_id=generic-uio. This can be done in the device tree file system-user.dtsi:

/include/ "system-conf.dtsi"
/ {
  chosen {
    bootargs = " earlycon console=ttyPS0,115200 clk_ignore_unused root=/dev/ram0 rw uio_pdrv_genirq.of_id=generic-uio";
    stdout-path = "serial0:115200n8";
  };
};

The mapped memory can be mapped into user space using the mmap() function, without specifying the physical address since it comes transparently from the device tree, and subsequently from the hardware description file.

gpio_ptr = mmap(NULL, gpio_size, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);

Memory accesses to this mapped area are simply accesses to internal offsets, as usual.

// Write
*((volatile unsigned *)(gpio_ptr + offset)) = value;
// Read
value = *((volatile unsigned *)(gpio_ptr + offset));

The biggest advantage of using UIO device drivers is interrupt handling. The generic driver provides one automatic interrupt mechanism completely under user space control. Reading from the character device file descriptor will block until an interrupt is received. Once an interrupt is received, the user space process continues running with interrupts disabled, and after the required processing is completed, the interrupts can be re-enabled by writing to the same file descriptor.

// Block until IRQ
read(fd, (void *)&pending, sizeof(int));

// Read interrupt status
reg = gpio_read(gpio_ptr, GPIO_IRQ_STATUS);

// Re-enable IRQ
write(fd, (void *)&reenable, sizeof(int));

Implementing UIO

The benefits of moving away from legacy device driver /dev/mem to UIO are several, starting with removing hard-coded addresses and sizes to devices, and adding seamless interrupt capability.

The process may be constructed to take a device name instead of an [address, size] hardcoded pair. The new process would “walk” through the /sys/class/uio/uioX/maps/mapY/name files until it finds a name that matches the device name from the device tree. Once found, this would be the device that it would open. The following snippets of working code will show an example implementation.

void get_uio_by_name(char* uio_name, int *uio_dev, int *uio_map, int *uio_size)
{
  char dev_name[128] = "";
  char devNamePath[128] = "";
  char devSizePath[64] = "";
  int uio, map;

  for (uio=0; uio<UIO_MAX_DEVICES; uio++) {

    for (map=0; map<UIO_MAX_MAPS; map++) {

      snprintf(devNamePath, 128, "/sys/class/uio/uio%d/maps/map%d/name", uio, map);
      FILE *nameFile = fopen (devNamePath, "r");
      if (!nameFile) {
        continue;
      }

      fscanf(nameFile, "%s", dev_name);
      if (!strncmp(uio_name, dev_name, 128)) {
        snprintf(devSizePath, 64, "/sys/class/uio/uio%d/maps/map%d/size", uio, map);
        *uio_size = get_uio_size(devSizePath);
        *uio_dev = uio;
        *uio_map = map;
        break;
      }
      fclose(nameFile);
    }
  }
  return;
}

In a similar way, the size can be read from the same directory as /sys/class/uio/uioX/maps/mapY/size.

unsigned int get_uio_size(char* size_file)
{
    int uio_size = 0; // the return value: size of the UIO memory
    FILE* size_fp; // pointer to the file containing memory size

    size_fp = fopen(size_file, "r");

    char string[100] = "";
    fscanf(size_fp, "%s", string);
    sscanf(string, "%x", &uio_size);

    fclose(size_fp);
    return uio_size;
}

Then when opening a named device “axil_periph”, the function get_uio_by_name would be called, returning the device id, map and size. The size will then be used to memory map the device’s memory into userspace.

int uio_dev, uio_map, uio_size;
char devPath[16];

get_uio_by_name("axil_periph", &uio_dev, &uio_map, &uio_size);

snprintf(devPath, 16, "/dev/uio%d", uio_dev);

int uio_fd = open(devPath, O_RDWR);

base_ptr = mmap(NULL, uio_size, PROT_READ|PROT_WRITE, MAP_SHARED, uio_fd, 0);

Userspace memory allocation

The kernel has to reserve special memory to be used by a DMA driver. The device tree uses a special node to accomplish this:

reserved-memory {
  #address-cells = <2>;
  #size-cells = <2>;
  ranges;
  reserved: buffer@0 {
    compatible = "shared-dma-pool";
    no-map;
    reg = <0x5 0x0 0x0 0x20000000>;
    linux,cma-default;
  };
};

It’s important to match the #address-cells and #size-cells to the top level property of the processor, i.e. <2> for ZynqMP, <1> for Zynq-7000, 64- and 32-bit respectively. The reg line contains the address and size formatted as pairs. In the example, the address would be 64-bit 0x500000000, with size 0x20000000 bytes. The reserved node name may be renamed. The buffer@0 is a label and may be renamed at will. The driver using this reserved memory would refer to it as follows:

dma_proxy {
  compatible ="xlnx,dma_proxy";
  ...
  memory-region = <\*&reserved\*>;
};

The memory-region property is used to link the driver with its reserved memory.

U-dma-buf

u-dma-buf is a Linux device driver that allocates contiguous memory blocks in the kernel space as DMA buffers and makes them available from the user space. The source code can be found in Gitlab u-dma-buf. Note there is a built-in module available from kernel 5.x with the name udmabuf that is unrelated to this driver.

In the case of using the u-dma-buf driver, its device tree node would read:

udmabuf@0x00 {
    compatible = "ikwzm,u-dma-buf";
    device-name = "udmabuf0";
    minor-number = <0>;
    size = <0x10000000>;          // 256 MB
    memory-region = <&reserved>;
};

Loading the u-dma-buf driver with no parameters, takes the size & memory region from the device tree node:

# cd /lib/modules/`uname -r`/extras
# insmod u-dma-buf

The u-dma-buf-test kernel module, if enabled, tests the different SYNC modes in the driver:

The test incorrectly reports physical address as 0x0, but the parsing (scanf) of the 64-bit address is wrong in the test code and it should have read 0x5_0000_0000.

Adding the u-dma-buf driver to the Petalinux build is quite simple. Add a new kernel module template with command below, then copy the udmabuf.c code from Github under recipes-modules/udmabuf/files. The created Makefile will handle all the build magic.

# petalinux-create -t modules -n udmabuf --enable 

Conclusion

The advantages to using UIO instead of the legacy /dev/mem driver are substantial. The main one is the UIO ability to handle interrupts from userspace. Others include the ability to handle devices by name set from the device tree, and thus not depending on hard coded address and sizes for the device memory to be mapped.