or, how I found multiple vulnerabilities on a lazy Sunday afternoon⌗

Earlier this year the NSA released Ghidra, a reverse engineering suite with support for a large number of CPU/MCU instruction sets. While I have some experience with Hopper and radare2 I wanted to play with Ghidra to poke around the firmware for my Zyxel GS1900-8 switch which runs on a 32-bit MIPS CPU. All in all this has turned out to be an interesting exploration of both Ghidra and the GS1900-8-2.40(AAHH.2)C0.bix firmware image.

Initially I wanted to write about poking around the firmware image and showing how one can use Ghidra to explore unknown binaries, but whilst looking around some libraries that are used by this switch I realised there is actually an interesting vulnerability to write about.

TL;DR⌗

“Unprivileged” users have full administrative privileges through SSH which also allows for obtaining encrypted credentials, which can then be trivially decrypted.

Secondly, there are two undocumented and password protected interfaces. One is a password recovery menu only reachable via serial console and the other is diagnostic menu which is available via SSH. The passwords for these hidden menus are hardcoded in the firmware.

Extracting the binaries and libraries⌗

The firmware image I used was downloaded from the Zyxel support site or you can grab it from their FTP site.

Binwalk had a hard time figuring out what was in the bix file however it did get a number of binaries extracted that were of enough interest to pursue further exploration of this image. A quick search revealed the gs1900fw project which was used to extract the firmware:

% python gs1900fw.py -w GS1900-8-2.40\(AAHH.2\)C0.bix -e
Loading: GS1900-8-2.40(AAHH.2)C0.bix
Checking file magic: Expected 0x83800000, found 0x83800000
Examining part: 0
  Writing to: GS1900-8-2.40(AAHH.2)C0.bix-part-0.gz
  Decompressing to: GS1900-8-2.40(AAHH.2)C0.bix-part-0-vmlinux_org.bin
  Writing kernel to: GS1900-8-2.40(AAHH.2)C0.bix-part-0-vmlinux_org.bin-kernel
  Writing initramfs to: GS1900-8-2.40(AAHH.2)C0.bix-part-0-vmlinux_org.bin-initramfs.gz

Next I extracted the initramfs.gz o Linux with GNU cpio as it required the --no-absolute-filenames flag:

$ gunzip GS1900-8-2.40\(AAHH.2\)C0.bix-part-0-vmlinux_org.bin-initramfs.gz
$ cpio -vid --no-absolute-filenames < GS1900-8-2.40\(AAHH.2\)C0.bix-part-0-vmlinux_org.bin-initramfs

This resulted in a few files which appear to be Busybox related, nothing that would indicate this being a switch firmware yet, except for an sqfs.img file:

$ binwalk sqfs.img

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             Squashfs filesystem, big endian, lzma signature, version 3.1, size: 4219154 bytes, 549 inodes, blocksize: 131072 bytes, created: 2019-06-05 03:49:3

This contained the real binaries used by the device:

$ ls -lh bin
total 5.3M
-rwxr-xr-x 1 kali kali  36K Jun  5 05:49 arp
-rwxr-xr-x 1 kali kali 223K Jun  5 05:49 boa
-rwxr-xr-x 1 kali kali 426K Jun  5 05:49 cli
-rwxr-xr-x 1 kali kali 191K Jun  5 05:49 dhcp6c
-rwxr-xr-x 1 kali kali 157K Jun  5 05:49 dhcpcd
-rwxr-xr-x 1 kali kali 1.9M Jun  5 05:49 diag
-rwxr-xr-x 1 kali kali 115K Jun  5 05:49 handler
-rwxr-xr-x 1 kali kali  14K Jun  5 05:49 inetd
-rwxr-xr-x 1 kali kali 110K Jun  5 05:49 initd
-rwxr-xr-x 1 kali kali 115K Jun  5 05:49 ksid
-rwxr-xr-x 1 kali kali  63K Jun  5 05:49 msntp
-rwxr-xr-x 1 kali kali 687K Jun  5 05:49 openssl
-rwxr-xr-x 1 kali kali 119K Jun  5 05:49 polld
-rwxr-xr-x 1 kali kali  31K Jun  5 05:49 snmpd
-rwxr-xr-x 1 kali kali  19K Jun  5 05:49 snmptrap
-rwxr-xr-x 1 kali kali 631K Jun  5 05:49 sshd
-rwxr-xr-x 1 kali kali 249K Jun  5 05:49 ssh-keygen
-rwxr-xr-x 1 kali kali  28K Jun  5 05:49 telnetd
-rwxr-xr-x 1 kali kali 106K Jun  5 05:49 timed
-rwxr-xr-x 1 kali kali 174K Jun  5 05:49 zon
$

Initial analysis⌗

Note: all C-like code was copied from the Decompile view of Ghidra, in general this was a very good representation of the assembly so that made it a lot easier to comb to the code.

After some initial poking around various binaries and libraries I focussed my attention to cli which is the entrypoint application that gets spawned when a user connects via telnet or SSH:

gs1900-telnet-start_login.png

Going through main() of cli you’ll eventually hit a call to vtysh_readline_init:

gs1900-cli-readline_init

This is where it gets interesting, because aside from the known keybindings such as 0x3f/? for a help-like text, there is this call:

rl_bind_keyseq("\\C-\\M-t",vtysh_diagDebug);

That’s CTRL-ALT-t to trigger vtysh_diagDebug() which I wasn’t aware of it existed and judging by the key combination needed to trigger it, it’s not meant for general consumption (cat owners might trigger this combination more often than others though..)

gs1900-cli-diagdebug.png

What this function does is it checks if remote debugging is enabled (remote being not connected to serial, so basically via a network connection) by calling fds_sys_remoteDebugEnable_ret() from libfds.so.0.0. Disassembling that function I expected it to check a configuration flag or a switch or anything. But no, it simply returned 1 indicating that remote debugging is enabled. Odd. Also note that cmd_textline_enable_get() is simply returns 0, so this branch is always taken.

Next it prints Diagnostics: and waits for input which, judging by the code, is supposed to be a password of sorts. But there is nothing that indicates this, again, odd. Renaming a few variables (with hindsight) we get:

printf("\nDiagnostics: ");
memset(user_input, 0, 0x20);
input_str(user_input, 0x1f, 0x2a);
memset(encrypted_input, 0, 0x41);
sal_util_str_encrypt(user_input, encrypted_input);
debug_password = (char *)fds_sys_passDebugPasswd_ret();
rc = strcmp(encrypted_input, debug_password);
if (rc == 0) {
	puts("Press ENTER to continue");
	diagdbg_auth = 1;
	diagdbg_flag = 1;
}

So it reads the user input, encrypts it and compares it against what fds_sys_passDebugPasswd_ret() returned. This function returns a hardcoded string:

char * fds_sys_passDebugPasswd_ret(void)
{
  return "jjvoKbG3ShgF1fdV3CxHWA==";
}

Looking around libfds.so.0.0 I also found this function, more on that later.

char * fds_sys_passRecoveryPasswd_ret(void)
{
  return "UAZX5JDMex9wNUNjmOHSRSX8NUrI+Xb1dWsghaV2JEo=";
}

This looked like a base64 encoded string of sorts, however I couldn’t make sense of the decoded output.

sal_util_str_encrypt() imported from libsal.so.0.0 provided a valuable clue:

gs1900-sal-str_encrypt.png

So some encryption and base64 encoding going on and aside from standard C and OpenSSL function calls there is a call to what Ghidra named FUN_000410b0:

gs1900-sal-fun004.png

Bingo! Looking at the OpenSSL API documentation for EVP_BytesToKey() we can rename the variables to paint a more complete picture:

memcpy(salt,"1A3BB2F78D6EC7D8",0x11);
memcpy(iv,"2268BA68768B58C3687D4F205923A741",0x20);
memcpy(key_data,"EC14D4F5BC6B9A3766D31EF9A1BB854121FB938B606462C70B2D0E26549C486A",0x40);
type = EVP_aes_256_cbc();
md = EVP_sha1();
key_size = EVP_BytesToKey(type, md, salt, key_data, 0x40, 5, key, iv);

At this point I attempted to write a quick decryption tool for the passwords using the decompiled code from the library. This wasn’t needed however because someone posted the source code for libsal to GitHub which was easy enough to find with the unique values for the salt, IV and key_data: sal_util_crypt.c. I’m not entirely sure this was supposed to have been posted there but it helped to validate my analysis [after writing the initial version of this post the entire GitHub accounts has been removed].

Given that AES256 is a symmetric cipher we simply need the key to decrypt the passwords previously uncovered. And sure enough that worked:

gs1900-decrypt-key.png

So the password for the hidden diagnostic menu is: 1900one and the illusive pass recovery password is gs1900@zyxel.com.tw. Given the name of the switch this firmware targets that makes sense.

The decryption tool is available on GitHub: jasperla/CVE-2019-15802.

Diagnostic menu⌗

Sure enough, logging into the switch via SSH as admin, hitting CTRL-ALT-t prompted me for a password and dropped me into a diagnostics shell:

sw.pwr#
Diagnostics: *******
Press ENTER to continue

RTK.0>
exit             - exit diag shell
chip             - chip configure
debug            - debug configuration
eee              - EEE configuration
eeep             - EEEP configuration
flowctrl         - flowctrl configuration
l2-table         - l2 table configuration
l3               - l3 configuration
led              - LED configuration
mib              - mib configuration
mirror           - mirror configuration
rspan            - RSPAN configuration
sflow            - sFlow configuration
nic              - NIC configuration
port             - port configuration
qos              - QoS configuration
bandwidth        - bandwidth configuration
storm-control    - storm-control configuration
register         - register configuration
vlan             - VLAN configuration
security         - security configuration
stp              - stp configuration
switch           - switch configuration
time             - time configuration
trap             - trap configuration
trunk            - trunk configuration
acl              - ACL configuration
field-selector   - field selector configuration
range-check      - range check configuration
diag             - diag configuration
ext-gpio         - external GPIO configuration
ext-smi          - Extensional SMI configuration
smi              - SMI configuration

From here one can get and set various settings, including register values of what I believe to belong to the PHY:

RTK.0> register get all

Register 0x1000 : 0x00000010
Register 0x0 : 0x00000000
Register 0x4 : 0x00000000
Register 0x8 : 0x00000000
Register 0xc : 0x00000000
Register 0x10 : 0x00000000
Register 0x14 : 0x00000000
Register 0x18 : 0x00000000
Register 0x1c : 0x00000000
Register 0x20 : 0x1b0c30c3
Register 0x24 : 0x1b0000c3
Register 0x28 : 0x00000129
Register 0x2c : 0x24000732

I decoded them but couldn’t find anything obviously useful so I moved on. This menu is implemented by the diag binary.

passRecoveryPasswd⌗

Going back to passRecoveryPasswd, in cli there are several references to password recovery passwords:

004103e4 24 84 2f 58     _addiu     a0=>s__Password_Recovery_Password:_00452f58,a0   = "\nPassword Recovery Password: "

004104a8 24 84 2f 94     _addiu     a0=>s_[P]_Password_recovery_for_specif_00452f9   = "[P] Password recovery for specific user"

The first is the string printed at a prompt (we now know the password to be entered there) and the second is a menu option to recover the password for a specific user.

Looking where these are actually used it’s in the signal handling for SIGQUIT:

gs1900-cli-sigquit.png

However as you can see from the code above, it checks access_flag if we’re allowed to enter this menu. It turns out that for SSH connections that’s not the case as demonstrated when I send a SIGQUIT (CTRL+\\) via SSH:

sw.pwr> Do not allow remote user to launch password recovery

sw.pwr>

Getting the serial console working didn’t work out so this menu still remains to be explored.

Suffice to say however I couldn’t find a mention of these hidden menus anywhere in the public documentation, nor are there any references to these passwords. What’s worse is that the encryption parameters are public, reused and static across all devices with this firmware (and perhaps other devices where the vendors didn’t change the settings).

system() calls⌗

Furthermore this firmware contains a large number of calls to the libc system() function to perform a host of functions. However input is not always sanitised to prevent code injection through user controlled parameters:

puParm4 is the fourth argument to cmd_sys_traceroute_exec()

memset(acStack144,0,0x80);
sprintf(acStack144,"traceroute %s -m %u",*puParm4,uVar1);
system(acStack144);

Same here, but for cmd_sys_arp_clear() which resorts to a system() invocation if the third argument is non-zero:

{
  [...]
  __stream = &FStack600;
  memset(__stream,0,0x80);
  sprintf((char *)__stream,"arp -d %s 1> /dev/null 2> /dev/null",*puParm4);
  pcVar4 = system;
}
(*pcVar4)(__stream);

And again in cmd_sys_ping_exec():

iVar1 = sal_util_ipv6str_vlidate(*puParm4);
__format = "ping6 -c %d %s";
if (iVar1 != 0) {
  __format = "ping -c %d %s";
}
sprintf(acStack144,__format,uVar2,*puParm4);
system(acStack144);

I couldn’t find these functions actively in use in this version of the firmware, however they could provide useful primitives for an attacker to gain a code execution on the switch.

Privileged unprivileged users⌗

Before documenting my findings I wanted to see if I could trigger the diagnostic menu via SSH as a regular user. The web interface allows for creating users with one of two types of privilege: admin or user. So I created a user, logged in and triggered the menu. Yikes.

Ok, moving on, what are the privileges actually:

sw.pwr# show privilege
Current CLI Username:  unpriv
Current CLI Privilege: 15
sw.pwr#

But that’s the same privilege level an admin user has! And sure enough, I had all the commands available to me as a regular user and adding insult to injury, I could now request a tech support dump which contains the encrypted password for admin. This password is encrypted in the same way as we previously discussed so it can be decrypted without any effort.

What’s funny is that the same user has very limited rights in the web interface…so it turns out that unprivileged users are indeed unprivileged in the web interface, but have full administrative rights in the CLI.

To demonstrate this flaw I’ve made a screen recording which demonstrates:

  • Create a new user
  • Generate and set a new password for the admin account
  • Show the limited rights of the user in the webui
  • Show equal privileges via SSH and trigger the diagnostic menu to display some register values
  • Finally dump the encrypted hashes and decrypt the admin password which was generated at the beginning of the video.

Reproducing and reporting⌗

I have ensured the privileged-unprivileged user accounts issue is not a weird side effect of running this switch for a few years, by doing a factory reset and reproducing the issue.

Also I’ve looked online if anyone was clever enough to post their tech-support or running-config.cfg anywhere (spoiler: they did) and whether I was able decrypt their passwords without issues (I was).

Timeline⌗

  • August 20, 2019: sent initial report to Zyxel
  • August 21, 2019: Zyxel acknowledges my mail and is looking into the reported issues; I sent additional details including a draft version of this post
  • August 23, 2019: Zyxel confirms the reported vulnerabilities and requests public disclosure to be postponed until new firmware is released at October 3, 2019
  • August 29, 2019: MITRE assigned CVE IDs
  • September 26, 2019: Zyxel requests postponing the disclosure due to the firmware fixes introducing interoperability issues with their ZON utility
  • November 14, 2019: Zyxel released the advisory along with new firmware images

Assigned CVE IDs⌗

  • CVE-2019-15799: “Incorrect access control” for the full administrative level access via SSH for unprivileged users
  • CVE-2019-15800: “Improper Input Validation” related to the functions using system() in libclicmd.so
  • CVE-2019-15801: “Use of Hard-coded Password” for the two hardcoded (encrypted) passwords
  • CVE-2019-15802: “Use of Hard-coded Cryptographic Key” for the hardcoded salt, IV and key data
  • CVE-2019-15803: “Hidden Functionality” for the diagnostics shell via CTRL-ALT-t
  • CVE-2019-15804: “Hidden Functionality” for the password recovery menu via SIGQUIT

Conclusion⌗

Reverse engineering firmware is fun and yet scary. It seems that manufacturers hope noone looks at the internals because what I found was simply hidden in plain sight. At least I got to figure out that my switch had much more functionality than was advertised when I bought it, but I’m not sure how happy I am about this…