Saturday, September 28, 2013

Inside low-level details of RTEMS/MMU project

Objectives :


This post aims to reveal low-level details of libmm for ARM BSPs. The main objectives are to know how low-level layer works and how to port libmm for a new (ARM) BSP.

ARM/LIBMM interaction :


Since libmm is about managing memory protection, Caches and MMU, libmm relies on ARM Co-processor 15: the primary co-processor which manages memory at the HW level for all modern ARM cores. CP-15 contains some control registers that help us to manage memory both at startup and dynamically.

The main CP-15 control register (C1) contains flags that we set/unset during RTEM initialization and application run-time. Figure-1 shows control register format on ARMv6. Please note that some of CP-15 C1 register flags may exist for a specific ARM architecture and absent for another one, also some flags maybe deprecated in modern architectures but still there not to hurt compatability. You can find each flag use and its meaning at the reference manual for ARM architectures [1] and CPU you develop in.

Figure-1 ARM CP-15 Control Register Format
The most important flags we use are M and C flags. M flag is used to enable/disable MMU, while C flag is used to enable/disable Cache unit.
CP-15 also contains other useful registers like :

  • Translation Table Base Register : Points to the First level of page-tables (Sections or Super Sections)
  • Domain Access Control : The purpose of the Domain Access Control Register is to hold the access permissions for a maximum of 16 domains.
  • Cache Operations .
  • TLB Operations.
  • Interrupt Status Register.

 ARM CP-15 API in RTEMS


RTEMS has a header file (arm-cp15.h) the contains code needed to do any operation on CP-15 and its registers. This should be the backend API for ARM BSPs that implements low-level libmm API. Another new header file called arm-cp15-start.h is created to help initializing CP-15 at BSP startup. The next section illustrates how ARM CP-15 is initialized.

arm-cp15-start.h initialization 


The main components and areas that involve libmm/MMU initialization are

  •  Linker defines : These defines at linker-symbols.h export macros and variables that point to registers like translation table base (page-table directory), object addresses for some sections and other useful linker variables. Figure-2 shows some linker variables quoted from ARM linker-symbols.h
  • Figure 2: Example linker symbols from ARM linker-symbols.h
  •  mm-config-table is an initialization table which contains entries each of which describes a section of the program object file by its starting address, end address and CPU memory attributes it "should" have. Every BSP should define its own init table which is passed to mminit function. Figure-3 shows the format of such a table.
Figure-3 mm_config_table format


For CPU Memory Attributes field, it simply represents a page-table entry for the CPU libmm is ported for. Figure-4 shows an example of RaspberryPi/ARM1176JZF-S CPU first-level page table entries formats.

Figure-4 Backwards-compatible first-level descriptor format.


An example of mm_config_table is the table for RaspberryPi BSP. It uses some linker defines to set starting addresses and sizes of sections. For the CPU Memory Attributes field, arm-cp15.h macros are used. Figure-5 is a real code for mm_config_table for RaspberryPi BSP.

Figure-5 raspberrypi mm_config_table example.

  •   linkcmds : This is the linker script passed to the linker during compilation/linking process. It manages how sections are organized in the final executable file and reserve memory areas for run-time demands. What concerns us about the linker file is to tell the linker where should be the global page-table and its size. Also this file should export the translation table base to other rtems low-level layers (such as libmm). For RaspberryPi BSP on RTEMs, the 128 MB memory is mapped as shown in figure-5.
Figure-6 RTEMS/RaspberryPi BSP memory map



  •  mminit.c : This file contains an implementation for bsp_memory_management_initialize which is a part of libmm API. Every BSP that supports libmm should implement this function. mminit.c should contain all variables, macros, routines that initialize memory management HW. For RaspberryPi, the implementation is direct wrapper to arm_cp15_start_setup_translation_table_and_enable_mmu_and_cache included from arm-cp15-start.h. mm_config_table and its size are passed to this function which iterates over the table entries, and set HW page table entries according to mm_config_table entries.

Setting page table entries dynamically 


Some applications may need to set memory attributes for a specific area of memory dynamically (within application phase). Examples of these applications are dynamic linker/loader and POSIX mman.h API. libmm supports dynamic setting of memory attributes through a call to bsp_memory_management_set_attributes routine which is a part of libmm low-level API. Every BSP that supports libmm should implement this function. In RTEMS/ARM, we share mm.c file to support this feature. mm.c should translate high-level HW-independent libmm attributes (at score/mm.h file) to CPU specific attributes, thus, we can share high-level APIs between different BSPs and architectures. Figure-7 is a shared translation macro for different ARM BSPs (quoted from mm.c).
Figure-7 Translating high-level LIBMM attributes to CPU attributes.
At the run-time, the process of dynamically setting page-table attributes involves:
  1. Disable interrupts.
  2. Disable MMU and Cache unit.
  3. Get page table indices for the region to apply attributes on.
  4. Set page-table attributes.
  5. Enable MMU and Cache unit.
  6. Enable interrupts.

How to port libmm for a new BSP ? porting !


To port libmm for a new BSP you should simply follow these steps :

For static initialization part :

  • Create mm_config_table according to the memory map of the new BSP. For ARM BSPs that share mminit.c and mm.c this table name should be the same (currently it's called bsp_mm_config_table).
  • Define the size of the previous table with a shared name (bsp_mm_config_table_size).
  • implement bsp_memory_management_initialize for the new BSP.

For dynamic part : 

  • Create a translation attributes that convert high-level HW independent attributes (at score/mm.h) to the CPU of your BSP.
  • implement bsp_memory_management_set_attributes
File that contains these implementation should include <libbsp/mm.h> which has prototypes for libmm low-level API.


How to make sure that the new port works ? porting !


To make sure your new port works fine you have to :
  • Run sample tests (hello.exe, ticker.exe). If these tests runs as expected that means libmm initialization is done successfully, Otherwise the program should hang and give no output.
  • Run mmtest2.exe (at libtests/mmtest2) that tests dynamic part of libmm and triggers some exceptions. Figure-8 is an example of mmtest2 output that runs successfully.
Figure-8 Output of mmtest2.exe

Acronyms : 


CP15    Co-Processor 15
libmm    memory management library
BSP      Board Support Package
MMU    Memory Management Unit
API       Application Programming Interface
TLB      Translation Lookaside Buffer
mminit    memory management initialize
init         initialization

References :


[1] ARM Architecture Reference Manual.
[2] github repo for the project.

Wednesday, September 18, 2013

[GSoC] Final report of MMU project for RTEMS

Brief :


The goal of this year project was to enhance low-level memory management Hardware/Software initialization and boost the process of merging libmm framework with RTEMS main stream. After working mainly on ARM BSPs with enhancements to avoid MMU code duplications, now many ARM BSPs can share a single implementation for low-level libmm. This way, we can move up to high-level layers (i.e POSIX) that rely on the low-level layer.

Enhancements (Finished Tasks) :


The enhancements  for ARM architectures are :

- Using existing arm-cp15.h code for implementing low-level libmm.
- Using newly add MMU code used by realview.
- Creating new mm.c and mminit.c file representing low-level libmm implementation for ARM.
  • mminit.c contains function that initializes MMU at BSP startup.
  • mm.c dynamically set new attributes for pages.
- Share mminit.c and mm.c between varios BSPs like (realview, xilinx).
- Use mminit.c and mm.c to create MMU support for RaspberryPi BSP.
- arm920 can share new code although Skyeye1.2.8 do not support all Cache instructions used by the shared code.
- Added new test case for SMP environment (realview BSP).

Figure-1 shows libmm hierachy in RTEMS.
Figure-1. Libmm Hierachy in RTEMS


Experience gained :


The most useful experience I got is when I was trying to share MMU ARM code between many BSPs. I was able to share MMU code between Realview BSP which has ARM Cortex-A9 (ARMv7) and RaspberryPi Board that has Broadcom BCM2835 system on a chip (SoC), which includes an ARM1176JZF-S CPU (ARMv6). To achieve this goal I had to :

  • Study ARMv6 and ARMv7 ISAs.
  • Analyze compatibility between previous architectures.
  • Review new ARMv6 Security Extension.
  • Study different page-table formats and make them compatible with each other by enabling/disabling some features on CPUs. (i.e subpages extension in ARM1176JZF processor).

Future goals (TODO) :


  • Port POSIX mman.h, shm.h.
  • Unify Cache management with libmm.
  • Spread out libmm by supporting more BSPs.


The following post reveals some low-level details regarding libmm and how BSPs can support it.