Emulating NVRAM in Qemu

Being able to emulate embedded applications in Qemu is incredibly useful, but not without pitfalls. Probably the most common issue that I’ve run into are binaries that try to read configuration data from NVRAM; since the binary is running in Qemu and not on the target device, there is obviously no NVRAM to read from.

Embedded applications typically interface with NVRAM through a shared library. The library in turn interfaces with the MTD partition that contains the device’s current configuration settings. Many programs will fail to run properly without the NVRAM configuration data, requiring us to intercept the NVRAM library calls and return valid data in order to properly execute the application in Qemu.

Here’s a Web server extracted from a firmware update image that refuses to start under Qemu:

It looks like httpd can’t start because it doesn’t know what IP address to bind to. The IP can’t be set via a command line argument, so it must be getting this data from somewhere else. Let’s fire up IDA and get cracking!

A quick look into httpd’s main function reveals the culprit. The httpd server tries to get the IP address and protocol settings via calls to nvram_get. If these calls fail, it prints the error message we saw above:

The nvram_get function is imported from a shared library, which will make it easy to intercept using LD_PRELOAD:

But before we can start intercepting function calls, we need to know more about nvram_get. It appears to only take one argument, which is a string (specifically, “lan_ipaddr” and “lan_proto” above), but what type of data does it return?

As can be seen in the previous disassembly, the return value from nvram_get(“lan_proto”) is saved in register R5. Later, the data pointed to from R5 is compared against the strings “static” and “dhcp” using strcmp:

Likewise, the return value from nvram(“lan_ipaddr”) is saved in register R6, which is later passed as the first argument to inet_aton:

So nvram_get takes a configuration key string and returns the corresponding configuration value string. We can easily simulate this function, along with some dummy configuration data, with the following code:

#include <stdio.h>
#include <string.h>

char *nvram_get(char *key)
{
        char *value = NULL;

        if(strcmp(key, "lan_ipaddr") == 0)
        {
                value = strdup("127.0.0.1");
        }

        if(strcmp(key, "lan_proto") == 0)
        {
                value = strdup("static");
        }

        printf("nvram_get(%s) == %s\n", key, value);
        return value;
}

We’ll need to cross-compile this code as a shared library and copy it into the squashfs-root directory that we’re running Qemu from:

eve@eve:~$ arm-linux-gcc -shared nvram.c -o nvram.so
eve@eve:~$ cp nvram.so squashfs-root/nvram.so

Now we’ll try running httpd inside Qemu again, this time specifying the path to our nvram.so file in the LD_PRELOAD environment variable:

eve@eve:~/squashfs-root$ sudo chroot . ./qemu-arm -E LD_PRELOAD="/nvram.so" usr/sbin/httpd 
usr/sbin/httpd: relocation error: /nvram.so: symbol __register_frame_info, version GLIBC_2.0 not defined in file libgcc_s.so.1 with link time reference
eve@eve:~/squashfs-root$

Boo! It looks like the nvram.so file is expecting a __register_frame_info symbol which doesn’t exist in the target system’s libgcc_s.so library. This happens because the tool chain we used to build nvram.so is not the same tool chain the vendor used to build the firmware for the target system. Ours expects __register_frame_info to be present, while theirs does not.

Since the vendor didn’t release GPL code for their system, we can’t simply re-build nvram.so using their tool chain. We could initiate a GPL request with the company, but there is a simpler (and faster!) way. We’ll just add place holder definitions for the __register_frame_info symbol inside nvram.c:

#include <stdio.h>
#include <string.h>

void __register_frame_info(void) { }
void __deregister_frame_info(void) { }
void __unregister_frame_info(void) { }

char *nvram_get(char *key)
{
        char *value = NULL;

        if(strcmp(key, "lan_ipaddr") == 0)
        {
                value = strdup("127.0.0.1");
        }

        if(strcmp(key, "lan_proto") == 0)
        {
                value = strdup("static");
        }

        printf("nvram_get(%s) == %s\n", key, value);
        return value;
}

Note that we’ve also defined __deregister_frame_info and __unregister_frame_info, which the nvram.so will also be looking for (if you didn’t know this off hand, you would get subsequent errors for these missing symbols indicating that you need to add their definitions to nvram.c as well).

We’ll re-build nvram.so:

eve@eve:~$ arm-linux-gcc -shared nvram.c -o nvram.so
eve@eve:~$ cp nvram.so squashfs-root/nvram.so

And try running httpd again:

eve@eve:~/squashfs-root$ sudo chroot . ./qemu-arm -E LD_PRELOAD="/nvram.so" usr/sbin/httpd 
httpd server started at port 80 (delay 0 second) 
nvram_get(lan_ipaddr) == 127.0.0.1
nvram_get(lan_proto) == static

We are successfully intercepting nvram_get calls, the httpd error messages have disappeared, and the server appears to be running. Let’s check:

Success! httpd is now ready for a little one-on-one with IDA’s debugger.

Bookmark the permalink.

7 Responses to Emulating NVRAM in Qemu

  1. Denis says:

    Amazing post ! Thank you.

  2. igorsk says:

    You could just put breakpoint on nvram_get and enter some IDC code into the condition field that would fill in the result and set PC=LR.

    • Craig says:

      Good point, I hadn’t even thought about that when I was working on this. Always more than one way to skin a cat. :)

      Also a related post here on handling conditional breakpoints with IDAPython.

  3. Eyal says:

    great post :-) i still need to understand ARM assembly’ i’m trying to figure out MIPS assembly after some of your other posts :-)

    Got any recomandation for good IDA tutorial and how to start remote debugging and reversing, even on regualr linux system, no need for embedded one right now.

    cheers!

    • Craig says:

      IDA’s debugger is pretty easy to use, but you should check out the IDA online help documents.

      See MIPS Run is a great book for learning MIPS, I highly recommend it. I’ve always found one of the best ways to learn is to just jump in and start reversing stuff and Google the things that you don’t understand. :)

  4. haohaolee says:

    One question. If you statically link the nvram.so to glibc, would the symbol problem be solved?

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>