[GSoC] Porting RTEMS to OpenRISC - Introduction

This year (2014) I’m participating in GSoC program with RTEMS. The project—Porting RTEMS to OpenRISC— (as its name indicates) aims to port RTEMS to OpenRISC architecture. Porting a well-known open source RTOS software like RTEMS for a big open source hardware architecture such as OpenRISC would be of useful to both HW and SW communities. The new port will target a broad range of users and fields (hobbyists, digital designers, computer architecture, embedded systems, industrial applications, etc).

In order to have a successful long term RTEMS port for OpenRISC that people can use, there are main goals (increments) to achieve during the project:

1- Integrate OpenRISC toolchain into RTEMS

To be able to compile and build RTEMS for OpenRISC architecture, some tools need to be added to RTEMS toolchain first. OpenRISC has its own tool-chain independent of RTEMS. So, the job would involve integrating both OpenRISC and RTEMS tool-chain, and make the necessary modifications/additions that fit for RTEMS. These tools include:

• binutils
• gdb
• newlib 
• gcc
• RSB (RTEMS Source Builder)

2- Porting low-level pieces of RTEMS

This includes writing start assembly, device drivers, interrupt handlers, context save, etc. The main components in RTEMS which this goal have to deal with are:

score/cpu: To define general/common OpenRISC architecture's functions.
libcpu (optional): To contain cache and MMU managers.
libbsp: BSPs code.

3- Writing a BSP for a board that OpenRISC supports

A simple BSP that can run hello world and ticker would be needed to be a front-end interface to OpenRISC and RTEMS users. Two BSPs can implemented:

openrisc_or1ksim: This BSP should be for simulation and debugging purposes only.
openrisc_atlys: I have Atlys board which can be used during the project to be the first OpenRISC BSP on RTEMS. Atlys is supported by OpenRISC and libgloss port and can run linux kernel.

4- Setup simulators and debuggers for OpenRISC.

This task includes writing sim-scripts to use openrisc or1k tools, simulators, debuggers. Software and hardware debugging should supported at this stage.

Environment 

• Platform: Fedora 20 and Ubuntu 12.04 LTS.
• Toolchain: Latest toolchain for OpenRISC.
• Simulators: or1ksim and/or qemu.
• Debuggers: gdb connected to or1ksim (OpenRISC tools). Also I should be debugging at real HW (Atlys board) by gdb via JTAG channel.
• Hardware: Atlys FPGA board (supported by OpenRISC).

If you are interested, I should be posting any updates for the project here in my blog. Also, you may want to have a look at project source code at my github account. Project link at RTEMS also exists.

My experience with GSoC

Back to 2012, I was already familiar with some interesting program called GSoC that Google organizes for students to do open source software projects. Like most of CS and Computer Engineering students, GSoC was a great opportunity: only if I could have the chance to participate in. The expectation of being accepted was almost null for me. But why not giving a try? What will I lose? So, I decided to take a step.

Embedded Systems, Operating Systems, RTOS, C: all the keywords I needed to find an accepted organization which matches my interests, and that's exactly what RTEMS tags include in the melange website: the official host for GSoC program. Immediately, I clicked on the link to the RTEMS website, read what RTEMS is about, and searched for open projects. Not surprisingly, there was an open project that appealed to me. To be able to write down a good proposal, I started to read more documentations. After having some headlines in the proposal, it was gradually increasing in size, details and consistency in parallel with reading more about the project details and grasping more concepts. "Nice, now I think I can submit my proposal," I said to myself. Fortunately, submitting my proposal was early and, later, I knew this was one lucky move; I got a lot of valuable and detailed comments from the folks of RTEMS community. That's how things became serious.

The comments made me feel that what I proposed was something important, and needed by users; a feeling that I never had before. I began to try to understand comments, ask about more details, read more and more; consequently, these comments dramatically re-factored my proposal structure and details. After a few weeks of discussions and modifications, the proposal was ready. Once the deadline of application period hit, I was comfortable with what I did. I thought I could not have done better. Few weeks later, the accepted projects were announced, and guess what? I had been accepted! Absolutely, that was a turning point in my life.

I got to do new things and learn interesting stuff about software engineering, how design phase is very important, the benefits of feedback and discussions, why it’s useful to frequently interact with the community via mailing list and IRC. Moreover, I learned a lot about technical issues and coding. Exclusively to my project with RTEMS, my technical hands-on experience has been enriched in a variety of areas like: C programming, RTOS, gdb, gcc, simulators, hardware, embedded systems, git, source control, documentation, other open source tools, and a lot more. Making connections with some great mentors there was one great aspect of participating in GSoC.

I enjoyed having commitments, deadlines, challenges and troubles. That's what GSoC is partially about (other than encouraging students to work with open source software): a real world work experience. People in RTEMS, and any other organization, want you to complete your project successfully. They are thrilled to help you with every challenge you face. Thanks to my mentor, I was able to pass midterm and final evaluations, and I could brag that I am a former GSoC participant.

Statement of accomplishment - GSoC'2012

Next year (2013), I applied to GSoC with RTEMS (again!) and I have been accepted. Right after GSoC, I participated in GCI--Google Code In--as a mentor, not surprisingly, with the same organization. This year (2014), I have applied to GSoC with RTEMS and I am currently awaiting the announcement of accepted students (wish me luck).

GCI'2013

For me, GSoC made what it aims/claims to achieve:

(a) I learned much about open source software.
(b) Now I am stuck to open source and RTEMS.
(c) Got hands-on experience with real world software process.
(d) Made connections with great developers.
(e) Achieved a glory (and got a nice paycheck).

If you are a student and you're considering applying to GSoC, here is my two cents:

(a) Do not worry about your modest experience, you're ought to learn in the first place. Basics are enough.
(b) Make a proposal and get in touch with the organization community. Do both early!
(c) Your proposal should be well structured and detailed. It should explain how will you work on the project and give a clear and reasonable timeline as well as deliverables. Most organizations provide a template for writing proposals.
(d) Choose a project that 1) appeals to the organization and its users, 2) match your interests. Most likely it will be one of the projects ideas.

Inside low-level details of RTEMS/MMU project

Objectives :

This post aims to reveal low-level details of libmm for ARM BSPs. The main objectives are to know how low-level layer works and how to port libmm for a new (ARM) BSP.

ARM/LIBMM interaction :

Since libmm is about managing memory protection, Caches and MMU, libmm relies on ARM Co-processor 15: the primary co-processor which manages memory at the HW level for all modern ARM cores. CP-15 contains some control registers that help us to manage memory both at startup and dynamically.

The main CP-15 control register (C1) contains flags that we set/unset during RTEM initialization and application run-time. Figure-1 shows control register format on ARMv6. Please note that some of CP-15 C1 register flags may exist for a specific ARM architecture and absent for another one, also some flags maybe deprecated in modern architectures but still there not to hurt compatability. You can find each flag use and its meaning at the reference manual for ARM architectures [1] and CPU you develop in.

Figure-1 ARM CP-15 Control Register Format

The most important flags we use are M and C flags. M flag is used to enable/disable MMU, while C flag is used to enable/disable Cache unit.
CP-15 also contains other useful registers like :

• Translation Table Base Register : Points to the First level of page-tables (Sections or Super Sections)
• Domain Access Control : The purpose of the Domain Access Control Register is to hold the access permissions for a maximum of 16 domains.
• Cache Operations .
• TLB Operations.
• Interrupt Status Register.

 ARM CP-15 API in RTEMS

RTEMS has a header file (arm-cp15.h) the contains code needed to do any operation on CP-15 and its registers. This should be the backend API for ARM BSPs that implements low-level libmm API. Another new header file called arm-cp15-start.h is created to help initializing CP-15 at BSP startup. The next section illustrates how ARM CP-15 is initialized.

arm-cp15-start.h initialization 

The main components and areas that involve libmm/MMU initialization are

•  Linker defines : These defines at linker-symbols.h export macros and variables that point to registers like translation table base (page-table directory), object addresses for some sections and other useful linker variables. Figure-2 shows some linker variables quoted from ARM linker-symbols.h

• 

  Figure 2: Example linker symbols from ARM linker-symbols.h

•  mm-config-table is an initialization table which contains entries each of which describes a section of the program object file by its starting address, end address and CPU memory attributes it "should" have. Every BSP should define its own init table which is passed to mminit function. Figure-3 shows the format of such a table.

Figure-3 mm_config_table format

For CPU Memory Attributes field, it simply represents a page-table entry for the CPU libmm is ported for. Figure-4 shows an example of RaspberryPi/ARM1176JZF-S CPU first-level page table entries formats.

Figure-4 Backwards-compatible first-level descriptor format.

An example of mm_config_table is the table for RaspberryPi BSP. It uses some linker defines to set starting addresses and sizes of sections. For the CPU Memory Attributes field, arm-cp15.h macros are used. Figure-5 is a real code for mm_config_table for RaspberryPi BSP.

Figure-5 raspberrypi mm_config_table example.

•   linkcmds : This is the linker script passed to the linker during compilation/linking process. It manages how sections are organized in the final executable file and reserve memory areas for run-time demands. What concerns us about the linker file is to tell the linker where should be the global page-table and its size. Also this file should export the translation table base to other rtems low-level layers (such as libmm). For RaspberryPi BSP on RTEMs, the 128 MB memory is mapped as shown in figure-5.

Figure-6 RTEMS/RaspberryPi BSP memory map

•  mminit.c : This file contains an implementation for bsp_memory_management_initialize which is a part of libmm API. Every BSP that supports libmm should implement this function. mminit.c should contain all variables, macros, routines that initialize memory management HW. For RaspberryPi, the implementation is direct wrapper to arm_cp15_start_setup_translation_table_and_enable_mmu_and_cache included from arm-cp15-start.h. mm_config_table and its size are passed to this function which iterates over the table entries, and set HW page table entries according to mm_config_table entries.

Setting page table entries dynamically 

Some applications may need to set memory attributes for a specific area of memory dynamically (within application phase). Examples of these applications are dynamic linker/loader and POSIX mman.h API. libmm supports dynamic setting of memory attributes through a call to bsp_memory_management_set_attributes routine which is a part of libmm low-level API. Every BSP that supports libmm should implement this function. In RTEMS/ARM, we share mm.c file to support this feature. mm.c should translate high-level HW-independent libmm attributes (at score/mm.h file) to CPU specific attributes, thus, we can share high-level APIs between different BSPs and architectures. Figure-7 is a shared translation macro for different ARM BSPs (quoted from mm.c).

Figure-7 Translating high-level LIBMM attributes to CPU attributes.

At the run-time, the process of dynamically setting page-table attributes involves:

1. Disable interrupts.
2. Disable MMU and Cache unit.
3. Get page table indices for the region to apply attributes on.
4. Set page-table attributes.
5. Enable MMU and Cache unit.
6. Enable interrupts.

How to port libmm for a new BSP ? porting !

To port libmm for a new BSP you should simply follow these steps :

For static initialization part :

• Create mm_config_table according to the memory map of the new BSP. For ARM BSPs that share mminit.c and mm.c this table name should be the same (currently it's called bsp_mm_config_table).
• Define the size of the previous table with a shared name (bsp_mm_config_table_size).
• implement bsp_memory_management_initialize for the new BSP.

For dynamic part : 

• Create a translation attributes that convert high-level HW independent attributes (at score/mm.h) to the CPU of your BSP.
• implement bsp_memory_management_set_attributes

File that contains these implementation should include <libbsp/mm.h> which has prototypes for libmm low-level API.

How to make sure that the new port works ? porting !

To make sure your new port works fine you have to :

• Run sample tests (hello.exe, ticker.exe). If these tests runs as expected that means libmm initialization is done successfully, Otherwise the program should hang and give no output.
• Run mmtest2.exe (at libtests/mmtest2) that tests dynamic part of libmm and triggers some exceptions. Figure-8 is an example of mmtest2 output that runs successfully.

Figure-8 Output of mmtest2.exe

Acronyms : 

CP15    Co-Processor 15
libmm    memory management library
BSP      Board Support Package
MMU    Memory Management Unit
API       Application Programming Interface
TLB      Translation Lookaside Buffer
mminit    memory management initialize
init         initialization

References :

[1] ARM Architecture Reference Manual.
[2] github repo for the project.

[GSoC] Final report of MMU project for RTEMS

Brief :

The goal of this year project was to enhance low-level memory management Hardware/Software initialization and boost the process of merging libmm framework with RTEMS main stream. After working mainly on ARM BSPs with enhancements to avoid MMU code duplications, now many ARM BSPs can share a single implementation for low-level libmm. This way, we can move up to high-level layers (i.e POSIX) that rely on the low-level layer.

Enhancements (Finished Tasks) :

The enhancements  for ARM architectures are :

- Using existing arm-cp15.h code for implementing low-level libmm.
- Using newly add MMU code used by realview.
- Creating new mm.c and mminit.c file representing low-level libmm implementation for ARM.

• mminit.c contains function that initializes MMU at BSP startup.
• mm.c dynamically set new attributes for pages.

- Share mminit.c and mm.c between varios BSPs like (realview, xilinx).

- Use mminit.c and mm.c to create MMU support for RaspberryPi BSP.

- arm920 can share new code although Skyeye1.2.8 do not support all Cache instructions used by the shared code.

- Added new test case for SMP environment (realview BSP).

Figure-1 shows libmm hierachy in RTEMS.

Figure-1. Libmm Hierachy in RTEMS

Experience gained :

The most useful experience I got is when I was trying to share MMU ARM code between many BSPs. I was able to share MMU code between Realview BSP which has ARM Cortex-A9 (ARMv7) and RaspberryPi Board that has Broadcom BCM2835 system on a chip (SoC), which includes an ARM1176JZF-S CPU (ARMv6). To achieve this goal I had to :

• Study ARMv6 and ARMv7 ISAs.
• Analyze compatibility between previous architectures.
• Review new ARMv6 Security Extension.
• Study different page-table formats and make them compatible with each other by enabling/disabling some features on CPUs. (i.e subpages extension in ARM1176JZF processor).

Future goals (TODO) :

• Port POSIX mman.h, shm.h.
• Unify Cache management with libmm.
• Spread out libmm by supporting more BSPs.

The following post reveals some low-level details regarding libmm and how BSPs can support it.

libmm design on RTEMS for ARM architecture

First, if you do not know what libmm is about you should review that introduction post.
In this topic, I will give a brief overall picture of how libmm is designed and some implementation specific details for ARM.

Changes : 

Some changes were made to the API mentioned in the previous post. The major change is that both _Memory_management_Install_entry and _Memory_management_Uninstall_entry interfaces were merged and unified into a new interface called _Memory_management_Set_attributes. The new function takes three arguments :
base : Which is the starting address of the memory region to apply new attributes on.
size : The size of the memory address in bytes.
attributes : Abstract target-independent high-level attributes to be applied for a memory region. This argument describes common memory attributes for memory protection and caches.

The reason of the this re-factoring is that we got rid of control structs at high-level and low-level layers, So there is no "entries" to be installed or uninstalled anymore. This interface is similar to POSIX memory protection interface: mmprotect.

Design :

There are three major layers in RTEMS for libmm: score layer, cpu layer, low-level (implementation) layer.

score layer is the front-end API which the developer should deal with. The functions of the score layer were described in the introduction post and changes are mentioned here too. It also acquires locks to manage libmm implemented for SMP environment (like realview). cpu layer is a shared header file that is considered a link or a bridge between high-level score layer and implementation. All BSPs and targets that implement libmm should include this file. implementation layer is the target-dependent code that interfaces directly with hardware memory management system for a specific target. Figure1 shows the three layers and relations between each of them. Arrows indicate that one layer (on the "from" side of the arrow) depends another layer (on the "to" side of it).

Figre1: Relations between libmm layers.

ARM Implementation :

Shareability

Thanks to the good ARM architecture design and backward compatibility, RTEMS can share code for different architectures that deal with memory management part. This way, we can avoid low-level/libmm code duplication for different ARM BSP on RTEMS. libmm depends mostly at coprocessor15 for ARM targets that have MMU to configure memory management systems.

As an example of code shareability, is that RTEMS can use the same peace of code to initialize MMU and Caches at BSP startup as well as dynamically changing memory attributes at page tables. At the time of writing this post, RaspberryPi RTEMS BSP which has ARM1176JZF-S (ARMv6) processor share the same code with RealView BSP which has a Cortex™-A9 (ARMv7) processor. 

Performance and page sizes

For RTEMS, as being RTOS, the most important aspect to consider when making an implementation oriented decision is the performance and the speed. ARM MMU has two levels of page tables, each of which can describe memory attributes for different region sizes. The first level page-table can describe 16 MB super-sections or 1 MB sections. The second level page-table can describe 64 KB large pages or 4 KB pages. To keep TLB misses as few as possible beside avoiding the overhead of memory references and page table walks, we chose 1 MB sections as a basic unit for memory regions. libmm aligns up any region to 1 MB boundary.

Supported attributes

 Currently the supported attributes at high-level libmm layer are read-only, write-enable and no-access. These attributes can be easily extended to support Cache attributes too.

[GSoC] libmm project updates

* Setup RTEM environment for libmm. This action includes :

• libmm score API.
• libcpu/mm.h Header file to serve as an interface for low-level libmm API.
• Stubs for BSPs that do not implement libmm.
• mmtest1 test case.
•  Changes to all BSPs Makefiles to accommodate former files.

* libmm MPU implementation (not tested) : A simple implementation  that wraps ARMv7 MPU API.

* Change API prototypes and delete MME struct. Now low-level API takes base, size, attributes as args instead of MME struct pointer. 

* Adjust the current ARM MMU code and integrate it into Raspberry PI BSP (built successfully). The code is to be tested/debugged once I get the USB/TTL cable. 

[GSoC 2013] LIBMM for RTEMS

Introduction : 

libmm is memory management library for RTEMS. libmm aims is to centralize different memory management techniques supported by HW targets including MMU, MPU, Caches. Last year I worked on designing and implementing libmm and ported it for PSIM, gumstix BSPs [1] during GSoC 2012 program. This year I am working on enhancing libmm and port it for other targets and BSPs [2].

Till now, libmm has three major functions : 

• _Memory_management_Initialize : used to initialize memory resources (MMU, MPU and/or Caches) and any other required data structures. Also this function creates a static mapping for memory sections like .text, .data, IO pages, and others with suitable memory attributes.  It's invoked at BSP startup.
• _Memory_management_Install_entry : used to install entry into the target HW with supplied memory attributes flags representing cache policies, memory protection flags. It takes the start address for the area of memory to be installed, size, and flags. It can be used to dynamically install entries during application phase as well.
• _Memory_management_Uninstall_entry : When a memory management entry is no longer needed, this function should be invoked to clean up HW/SW resources and revert back to the state before calling  _Memory_management_Install_entry.

 BSPs support for libmm:

Currently libmm is ported for :

• PSIM : PowerPC/mpc6xx BSP (603e core).
• Gumstix : ARM based board (arm920t core).
• Realview_pbx_a9 : running on QEMU (Cortex-a9) new!
• Xilinx_zynq : running on QEMU (Cortex-a9) new!

Raspberry Pi is to be supported soon on real hardware.

Added features for libmm :

To use libmm within SMP environment, smplock calls are added to _Memory_management_Install_entry, _Memory_management_Uninstall_entry. 

For ARM/Realview :

-new extension function is implemented to view more info about address caused data abort exception and which error caused the exception.

- As page tables are shared resource between cores and threads, It's mapped as "SHARED", thus, let the HW ensure caches/memory coherency for that area of memory.

Tests cases :

mmtest1: Simple tests that tries to install memory management entries.

mmtest2: 

•  Install entries with specific memory attributes (e.g read only region) :
•  Check for memory protection violations (writing to read only blocks)
•  Reading from read only blocks.
•  Write/Read to/from unmapped region (error!).
•  Write to a valid entry that was installed and then uninstalled (error!).

mmtest3 (on-going): new!

Tests for libmm behavior on SMP environments.

• Create tasks for each core and start it.
• Check for memory consistency and page tables and memory attributes validity.

Snapshot for mmtest2 running on Realview/QEMU:

References :

[1] https://github.com/heshamelmatary/rtems-gsoc2012
[2] https://github.com/heshamelmatary/rtems-gsoc2013