Inside low-level details of RTEMS/MMU project

Objectives :

This post aims to reveal low-level details of libmm for ARM BSPs. The main objectives are to know how low-level layer works and how to port libmm for a new (ARM) BSP.

ARM/LIBMM interaction :

Since libmm is about managing memory protection, Caches and MMU, libmm relies on ARM Co-processor 15: the primary co-processor which manages memory at the HW level for all modern ARM cores. CP-15 contains some control registers that help us to manage memory both at startup and dynamically.

The main CP-15 control register (C1) contains flags that we set/unset during RTEM initialization and application run-time. Figure-1 shows control register format on ARMv6. Please note that some of CP-15 C1 register flags may exist for a specific ARM architecture and absent for another one, also some flags maybe deprecated in modern architectures but still there not to hurt compatability. You can find each flag use and its meaning at the reference manual for ARM architectures [1] and CPU you develop in.

Figure-1 ARM CP-15 Control Register Format

The most important flags we use are M and C flags. M flag is used to enable/disable MMU, while C flag is used to enable/disable Cache unit.
CP-15 also contains other useful registers like :

• Translation Table Base Register : Points to the First level of page-tables (Sections or Super Sections)
• Domain Access Control : The purpose of the Domain Access Control Register is to hold the access permissions for a maximum of 16 domains.
• Cache Operations .
• TLB Operations.
• Interrupt Status Register.

 ARM CP-15 API in RTEMS

RTEMS has a header file (arm-cp15.h) the contains code needed to do any operation on CP-15 and its registers. This should be the backend API for ARM BSPs that implements low-level libmm API. Another new header file called arm-cp15-start.h is created to help initializing CP-15 at BSP startup. The next section illustrates how ARM CP-15 is initialized.

arm-cp15-start.h initialization 

The main components and areas that involve libmm/MMU initialization are

•  Linker defines : These defines at linker-symbols.h export macros and variables that point to registers like translation table base (page-table directory), object addresses for some sections and other useful linker variables. Figure-2 shows some linker variables quoted from ARM linker-symbols.h

• 

  Figure 2: Example linker symbols from ARM linker-symbols.h

•  mm-config-table is an initialization table which contains entries each of which describes a section of the program object file by its starting address, end address and CPU memory attributes it "should" have. Every BSP should define its own init table which is passed to mminit function. Figure-3 shows the format of such a table.

Figure-3 mm_config_table format

For CPU Memory Attributes field, it simply represents a page-table entry for the CPU libmm is ported for. Figure-4 shows an example of RaspberryPi/ARM1176JZF-S CPU first-level page table entries formats.

Figure-4 Backwards-compatible first-level descriptor format.

An example of mm_config_table is the table for RaspberryPi BSP. It uses some linker defines to set starting addresses and sizes of sections. For the CPU Memory Attributes field, arm-cp15.h macros are used. Figure-5 is a real code for mm_config_table for RaspberryPi BSP.

Figure-5 raspberrypi mm_config_table example.

•   linkcmds : This is the linker script passed to the linker during compilation/linking process. It manages how sections are organized in the final executable file and reserve memory areas for run-time demands. What concerns us about the linker file is to tell the linker where should be the global page-table and its size. Also this file should export the translation table base to other rtems low-level layers (such as libmm). For RaspberryPi BSP on RTEMs, the 128 MB memory is mapped as shown in figure-5.

Figure-6 RTEMS/RaspberryPi BSP memory map

•  mminit.c : This file contains an implementation for bsp_memory_management_initialize which is a part of libmm API. Every BSP that supports libmm should implement this function. mminit.c should contain all variables, macros, routines that initialize memory management HW. For RaspberryPi, the implementation is direct wrapper to arm_cp15_start_setup_translation_table_and_enable_mmu_and_cache included from arm-cp15-start.h. mm_config_table and its size are passed to this function which iterates over the table entries, and set HW page table entries according to mm_config_table entries.

Setting page table entries dynamically 

Some applications may need to set memory attributes for a specific area of memory dynamically (within application phase). Examples of these applications are dynamic linker/loader and POSIX mman.h API. libmm supports dynamic setting of memory attributes through a call to bsp_memory_management_set_attributes routine which is a part of libmm low-level API. Every BSP that supports libmm should implement this function. In RTEMS/ARM, we share mm.c file to support this feature. mm.c should translate high-level HW-independent libmm attributes (at score/mm.h file) to CPU specific attributes, thus, we can share high-level APIs between different BSPs and architectures. Figure-7 is a shared translation macro for different ARM BSPs (quoted from mm.c).

Figure-7 Translating high-level LIBMM attributes to CPU attributes.

At the run-time, the process of dynamically setting page-table attributes involves:

1. Disable interrupts.
2. Disable MMU and Cache unit.
3. Get page table indices for the region to apply attributes on.
4. Set page-table attributes.
5. Enable MMU and Cache unit.
6. Enable interrupts.

How to port libmm for a new BSP ? porting !

To port libmm for a new BSP you should simply follow these steps :

For static initialization part :

• Create mm_config_table according to the memory map of the new BSP. For ARM BSPs that share mminit.c and mm.c this table name should be the same (currently it's called bsp_mm_config_table).
• Define the size of the previous table with a shared name (bsp_mm_config_table_size).
• implement bsp_memory_management_initialize for the new BSP.

For dynamic part : 

• Create a translation attributes that convert high-level HW independent attributes (at score/mm.h) to the CPU of your BSP.
• implement bsp_memory_management_set_attributes

File that contains these implementation should include <libbsp/mm.h> which has prototypes for libmm low-level API.

How to make sure that the new port works ? porting !

To make sure your new port works fine you have to :

• Run sample tests (hello.exe, ticker.exe). If these tests runs as expected that means libmm initialization is done successfully, Otherwise the program should hang and give no output.
• Run mmtest2.exe (at libtests/mmtest2) that tests dynamic part of libmm and triggers some exceptions. Figure-8 is an example of mmtest2 output that runs successfully.

Figure-8 Output of mmtest2.exe

Acronyms : 

CP15    Co-Processor 15
libmm    memory management library
BSP      Board Support Package
MMU    Memory Management Unit
API       Application Programming Interface
TLB      Translation Lookaside Buffer
mminit    memory management initialize
init         initialization

References :

[1] ARM Architecture Reference Manual.
[2] github repo for the project.

[GSoC] Final report of MMU project for RTEMS

Brief :

The goal of this year project was to enhance low-level memory management Hardware/Software initialization and boost the process of merging libmm framework with RTEMS main stream. After working mainly on ARM BSPs with enhancements to avoid MMU code duplications, now many ARM BSPs can share a single implementation for low-level libmm. This way, we can move up to high-level layers (i.e POSIX) that rely on the low-level layer.

Enhancements (Finished Tasks) :

The enhancements  for ARM architectures are :

- Using existing arm-cp15.h code for implementing low-level libmm.
- Using newly add MMU code used by realview.
- Creating new mm.c and mminit.c file representing low-level libmm implementation for ARM.

• mminit.c contains function that initializes MMU at BSP startup.
• mm.c dynamically set new attributes for pages.

- Share mminit.c and mm.c between varios BSPs like (realview, xilinx).

- Use mminit.c and mm.c to create MMU support for RaspberryPi BSP.

- arm920 can share new code although Skyeye1.2.8 do not support all Cache instructions used by the shared code.

- Added new test case for SMP environment (realview BSP).

Figure-1 shows libmm hierachy in RTEMS.

Figure-1. Libmm Hierachy in RTEMS

Experience gained :

The most useful experience I got is when I was trying to share MMU ARM code between many BSPs. I was able to share MMU code between Realview BSP which has ARM Cortex-A9 (ARMv7) and RaspberryPi Board that has Broadcom BCM2835 system on a chip (SoC), which includes an ARM1176JZF-S CPU (ARMv6). To achieve this goal I had to :

• Study ARMv6 and ARMv7 ISAs.
• Analyze compatibility between previous architectures.
• Review new ARMv6 Security Extension.
• Study different page-table formats and make them compatible with each other by enabling/disabling some features on CPUs. (i.e subpages extension in ARM1176JZF processor).

Future goals (TODO) :

• Port POSIX mman.h, shm.h.
• Unify Cache management with libmm.
• Spread out libmm by supporting more BSPs.

The following post reveals some low-level details regarding libmm and how BSPs can support it.

libmm design on RTEMS for ARM architecture

First, if you do not know what libmm is about you should review that introduction post.
In this topic, I will give a brief overall picture of how libmm is designed and some implementation specific details for ARM.

Changes : 

Some changes were made to the API mentioned in the previous post. The major change is that both _Memory_management_Install_entry and _Memory_management_Uninstall_entry interfaces were merged and unified into a new interface called _Memory_management_Set_attributes. The new function takes three arguments :
base : Which is the starting address of the memory region to apply new attributes on.
size : The size of the memory address in bytes.
attributes : Abstract target-independent high-level attributes to be applied for a memory region. This argument describes common memory attributes for memory protection and caches.

The reason of the this re-factoring is that we got rid of control structs at high-level and low-level layers, So there is no "entries" to be installed or uninstalled anymore. This interface is similar to POSIX memory protection interface: mmprotect.

Design :

There are three major layers in RTEMS for libmm: score layer, cpu layer, low-level (implementation) layer.

score layer is the front-end API which the developer should deal with. The functions of the score layer were described in the introduction post and changes are mentioned here too. It also acquires locks to manage libmm implemented for SMP environment (like realview). cpu layer is a shared header file that is considered a link or a bridge between high-level score layer and implementation. All BSPs and targets that implement libmm should include this file. implementation layer is the target-dependent code that interfaces directly with hardware memory management system for a specific target. Figure1 shows the three layers and relations between each of them. Arrows indicate that one layer (on the "from" side of the arrow) depends another layer (on the "to" side of it).

Figre1: Relations between libmm layers.

ARM Implementation :

Shareability

Thanks to the good ARM architecture design and backward compatibility, RTEMS can share code for different architectures that deal with memory management part. This way, we can avoid low-level/libmm code duplication for different ARM BSP on RTEMS. libmm depends mostly at coprocessor15 for ARM targets that have MMU to configure memory management systems.

As an example of code shareability, is that RTEMS can use the same peace of code to initialize MMU and Caches at BSP startup as well as dynamically changing memory attributes at page tables. At the time of writing this post, RaspberryPi RTEMS BSP which has ARM1176JZF-S (ARMv6) processor share the same code with RealView BSP which has a Cortex™-A9 (ARMv7) processor. 

Performance and page sizes

For RTEMS, as being RTOS, the most important aspect to consider when making an implementation oriented decision is the performance and the speed. ARM MMU has two levels of page tables, each of which can describe memory attributes for different region sizes. The first level page-table can describe 16 MB super-sections or 1 MB sections. The second level page-table can describe 64 KB large pages or 4 KB pages. To keep TLB misses as few as possible beside avoiding the overhead of memory references and page table walks, we chose 1 MB sections as a basic unit for memory regions. libmm aligns up any region to 1 MB boundary.

Supported attributes

 Currently the supported attributes at high-level libmm layer are read-only, write-enable and no-access. These attributes can be easily extended to support Cache attributes too.

[GSoC] libmm project updates

* Setup RTEM environment for libmm. This action includes :

• libmm score API.
• libcpu/mm.h Header file to serve as an interface for low-level libmm API.
• Stubs for BSPs that do not implement libmm.
• mmtest1 test case.
•  Changes to all BSPs Makefiles to accommodate former files.

* libmm MPU implementation (not tested) : A simple implementation  that wraps ARMv7 MPU API.

* Change API prototypes and delete MME struct. Now low-level API takes base, size, attributes as args instead of MME struct pointer. 

* Adjust the current ARM MMU code and integrate it into Raspberry PI BSP (built successfully). The code is to be tested/debugged once I get the USB/TTL cable. 

[GSoC 2013] LIBMM for RTEMS

Introduction : 

libmm is memory management library for RTEMS. libmm aims is to centralize different memory management techniques supported by HW targets including MMU, MPU, Caches. Last year I worked on designing and implementing libmm and ported it for PSIM, gumstix BSPs [1] during GSoC 2012 program. This year I am working on enhancing libmm and port it for other targets and BSPs [2].

Till now, libmm has three major functions : 

• _Memory_management_Initialize : used to initialize memory resources (MMU, MPU and/or Caches) and any other required data structures. Also this function creates a static mapping for memory sections like .text, .data, IO pages, and others with suitable memory attributes.  It's invoked at BSP startup.
• _Memory_management_Install_entry : used to install entry into the target HW with supplied memory attributes flags representing cache policies, memory protection flags. It takes the start address for the area of memory to be installed, size, and flags. It can be used to dynamically install entries during application phase as well.
• _Memory_management_Uninstall_entry : When a memory management entry is no longer needed, this function should be invoked to clean up HW/SW resources and revert back to the state before calling  _Memory_management_Install_entry.

 BSPs support for libmm:

Currently libmm is ported for :

• PSIM : PowerPC/mpc6xx BSP (603e core).
• Gumstix : ARM based board (arm920t core).
• Realview_pbx_a9 : running on QEMU (Cortex-a9) new!
• Xilinx_zynq : running on QEMU (Cortex-a9) new!

Raspberry Pi is to be supported soon on real hardware.

Added features for libmm :

To use libmm within SMP environment, smplock calls are added to _Memory_management_Install_entry, _Memory_management_Uninstall_entry. 

For ARM/Realview :

-new extension function is implemented to view more info about address caused data abort exception and which error caused the exception.

- As page tables are shared resource between cores and threads, It's mapped as "SHARED", thus, let the HW ensure caches/memory coherency for that area of memory.

Tests cases :

mmtest1: Simple tests that tries to install memory management entries.

mmtest2: 

•  Install entries with specific memory attributes (e.g read only region) :
•  Check for memory protection violations (writing to read only blocks)
•  Reading from read only blocks.
•  Write/Read to/from unmapped region (error!).
•  Write to a valid entry that was installed and then uninstalled (error!).

mmtest3 (on-going): new!

Tests for libmm behavior on SMP environments.

• Create tasks for each core and start it.
• Check for memory consistency and page tables and memory attributes validity.

Snapshot for mmtest2 running on Realview/QEMU:

References :

[1] https://github.com/heshamelmatary/rtems-gsoc2012
[2] https://github.com/heshamelmatary/rtems-gsoc2013