GSoC with RTEMS! - 1/n

Extending the features of a debugger, especially GDB, sounded like a fun and interesting project at first glance. I already have quite a bit of experience working with GDB, and use it almost daily while Reverse Engineering or during CTFs.
For those of you that are not familiar, CTF stands for "Capture the Flag". It is a competition wherein the hosting team creates and puts out challenges/services that are intentionally vulnerable - usually in a way that is specific to a single field of Cyber Security - and the point of the challenge is to figure out what the vulnerability is and use it to get a text string from it called a "flag". Capture more flags, gain more points. I play CTFs with team bi0s.

GDB has an awesome Python API that it utilizes to do things it cannot do straight out the box, and one such thing is pretty-printing.

Imagine you have a weird class/structure in your program, and you want to view it in memory, so you pop it open in GDB. Then you print your variable in GDB and it spits out something so awful that you wish you never started debugging in the first place. This is because GDB obviously cannot know about any and every single structure out there, and sometimes it needs a little help - this is where the awesome Python API I mentioned comes in.

Python essentially hooks onto GDB and interprets something GDB might be unable to, then do some Python magic and voila, you have the same structure neatly formatted! Some nice examples and a deeper explanation of what pretty-printing can be found in the documentation

GCC ships pretty-printing scripts for libstdcxx , which is essentially all of the C++ STL Structures, and my job is to get the GDB installed on RTEMS to automatically load these pretty-printing scripts at load-time, to enhance the debugging experience on RTEMS 😃. Next, add pretty-printing support for kernel structures present on RTEMS.

Starting off, I already had a couple of things to get ticked off my checklist. I mainly wanted to set up a proper debugging environment, and also study various commonly used kernel structures - before diving deeper into my project. Since my project itself was to extend the functionality of the debugger, what good is it if I don't have a debugging environment setup in place?

Let's jump right into my debugger setup then.

I am working on a Windows 11 (x64), WSL2 (Arch) system. On top of this, I have the RTEMS tooling installed from the instructions in their documentation. On top of that, they provide a variety of options to build various BSPs (think of BSPs as housing for the OS to be embedded upon). I needed a build setup such that:

• I could compile RTEMS C++ programs at will
• GDB could debug those programs remotely through a remote emulator (either built in, or using Qemu)

First, to build a BSP. I had to make 2 important choices:

• Which architecture will I use?
• Which BSP will I use for said architecture? (Note: A single architecture can support multiple BSPs)

I initially thought x86_64 arch, along with any suitable BSP for that arch would have me good to go, but I was sadly mistaken. Not only did I struggle to find BSPs for x86_64 , but I also found that there were no test configurations (yet) for any of the x86_64 BSPs 😦, so I decided not to proceed with that

Next up, I chose the sparc arch, along with the erc32 BSP, since that was one which seemed to have very good support for emulation and GDB support, and was also the one used in most of the examples mentioned in the docs. So I went ahead and built that.

sparc has an emulator built in in RTEMS, sparc-rtems6-sis ( sis : Sparc Instruction Set Simulator), which can emulate the instructions of the RTEMS executable, and GDB can attach to the process through a TCP port.

However, although compiling RTEMS C++ programs and running them worked perfectly on my sparc/erc32 build, I couldn't debug (i.e, step through) code conveniently. I kept hitting some kind of data access violation exception, due to which RTEMS would have a fatal crash and exit. Sad.

So I finalised on the arm/xilinx_zynq_a9_qemu BSP upon the suggestion of my mentor, which worked perfectly. So, I will be using that for the rest of this blog.

First, a sample C++ program.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

// main.cc 

#include <numeric>
#include <iostream>
#include <vector>

int main() {
    auto v = std::vector<int>(5);
    std::iota(std::begin(v), std::end(v), 0);
    
    for (auto i : v) 
    {
        std::cout << i << std::endl;
    }
    return 0;
}

First off, RTEMS uses waf (an alternative to make ), to build BSPs and applications on the platform. It's pretty versatile and easy to understand. You need 2 main things to build any app like this on RTEMS (apart from the source, of course) - waf (the script doing all the work) & wscript (the waf script containing instructions to be executed). Apart from these 2 files, you would also need some dependency files, and an initialisation script.

This page in the documentation gives a pretty good overview on how one can build an app on RTEMS.

Here is my wscript for all C++ applications like the one above:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

# wscript 
from __future__ import print_function

rtems_version = "6"

try:
    import rtems_waf.rtems as rtems
except:
    print('error: no rtems_waf git submodule')
    import sys
    sys.exit(1)

def init(ctx):
    rtems.init(ctx, version = rtems_version, long_commands = True)

def bsp_configure(conf, arch_bsp):
    # Add BSP specific configuration checks
    pass

def options(opt):
    rtems.options(opt)

def configure(conf):
    rtems.configure(conf, bsp_configure = bsp_configure)

def build(bld):
    rtems.build(bld)

    bld(features = 'cxx cxxprogram',
        target = 'cxx_stdvec.exe',
        cxxflags = '-std=c++11 -g -O2 -lstdc++',
        source = ['main.cc', 'rtems_config.c'])

The main function we need to focus on is build . Note the features (type of program), and cxxflags parameters passed to the bld function.

The application can be built with:

1
2
3

# configure the waf for the BSP you are building for 

./waf configure --prefix=$HOME/quick-start/rtems/6 --rtems-bsps=arm/xilinx_zynq_a9_qemu

1
2
3

# build the app

./waf

1
2
3

# run the executable to ensure it works

rtems-run --rtems-bsps=xilinx_zynq_a9_qemu build/arm-rtems6-xilinx_zynq_a9_qemu/cxx_vectorfail.exe

Once you confirm that the app works, we can move onto debugging it.
First, for emulation, I went with qemu. I already had a qemu-system-arm setup so it seemed like the most logical option.

Program can be emulated with:

1
2
3
4
5
6
7
8
9
10
11
12
13

# machine type: xilinx-zynq-a9
# -m 256: 256 megabytes of memory for emulation
# no-reboot: prevents the machine from automatically rebooting after shutdown
# -serial null: serial port output is piped to /dev/null
# -serial mon:stdio : redirects serial monitor output through stdio 
# -nographic: disables graphics 
# -s: enables GDB to debug 
# -S: starts up Qemu in a paused state, allowing GDB time to connect before execution begins 

qemu-system-arm -M xilinx-zynq-a9 -m \
    256M -no-reboot -serial \
    null -serial mon:stdio -nographic \
    -s -S

On a separate terminal, connect to this emulator (default port: 1234)

1
2
3
4
5
6
7

arm-rtems6-gdb <app_path.exe>

# inside gdb 
(gdb) target extended-remote localhost:1234 
(gdb) load
...
# any gdb command you want

More information on remote debugging can be found here.

Voila! Debugging environment set up!

Now, this is a pretty big process, with a lot of commands. This can be made easier, of course. Shell scripting for the win.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

# ./setup.sh 
# initialise environment to build the application 

echo "[+] Installing waf..."

curl https://waf.io/waf-2.0.19 > waf
chmod +x waf

echo "[+] Initialising repository and adding dependencies..."

git init
git submodule add git://git.rtems.org/rtems_waf.git rtems_waf

echo "[+] Configuring waf..."

./waf configure --rtems=$HOME/quick-start/rtems/6 --rtems-bsp=arm/xilinx_zynq_a9_qemu

echo "[+] Building application..."

./waf

echo "[+] Running application..."

rtems-run --rtems-bsps=xilinx_zynq_a9_qemu build/arm-rtems6-xilinx_zynq_a9_qemu/cxx_vectorfail.exe

On a different terminal:

1
2
3
4

qemu-system-arm -M xilinx-zynq-a9 -m \
    256M -no-reboot -serial \
    null -serial mon:stdio -nographic \
    -s -S

Make a script called init.gdb

1
2

target extended-remote localhost:1234
load

On the first terminal, create a new shell script

1
2

# ./loadgdb.sh
arm-rtems6-gdb -x init.gdb build/arm-rtems6-xilinx_zynq_a9_qemu/cxx_vectorfail.exe

I have all of these configured as commands on my terminal to make life easier, you could do that as well.

In next week's blog, I will be diving straight into the main crux of my project - the issue at hand, and how I plan on fixing it, stay tuned!