My wireless router (Buffalo WZR-D1800H) failed the other day, when an apparent firmware bug caused it to go into a continuous reboot cycle. After going through the various reset options it seemed like it had been turned into an expensive plastic brick.
Luckily there was a last resort method to gain debug access, see what was going on and hopefully bring it back to life. Internally there is a basic serial port, you just have to crack open the case and connect it to a PC somehow. However the serial port is not a usual RS232 standard, but a simplified TTY variant used more commonly for hardware debugging and firmware updates.
That was the theory, next was to try it out…
Opening The Case
The pictures below are from the DD-WRT forum post by “Magnetron1.1“ which shows how to open the case:
Connecting to a PC via an Arduino
After realizing I couldn’t just hook this up to my PC via an USB to serial (RS232) adapter, before giving-up and ordering a USB to TTY adapter, I stumbled upon this Electrical Engineering blog post that suggested my Arduino could do the same job. This was confirmed directly on the Arduino forum as being a valid configuration (that wouldn’t break my Arduino). Here’s how it looks when it is connected properly (using the “tri-state reset method” from the Arduino forum):
First of all I only had the router->Arduino TX->RX and RX->TX connections and the Arduino Reset->Ground. But I was getting corrupt messages on the serial console of the PC. To correct this I found a few things were necessary:
- The baud rate must match the router and there is no flow control so they must be in sync. Setting the port speed to 115200 fixed most of the corruption.
- The ground cable totally eliminated the rest of the corruption of the messages coming back from the router (black cable in photo above).
- Although the output (received data) was now perfect, I still had to hit each key more than once to get it to go through. That maybe something to do with my serial console program or other buffer settings. But since I only wanted to issue some simple commands I just put-up with it.
At this point I could see the problem, “broken firmware”, followed by a reboot:
Fixing The Router
The DD-WRT firmware start-up sequence will temporarily start with an IP address of 192.168.11.1 (regardless of your configuration) then check for the auto-pair (“AOSS”) button press. When pressed it will attempt to download a file called “firmware.ram” from 192.168.11.2 via TFTP. So you setup a free TFTPD server, download and rename the firmware you want to try. I did that, it downloaded, but failed to start. Not good…
As the last resort, there is a command line you can use to fix it. To get into it you have to hit CTRL+C and hold it during the reboot:
This breaks you into the “CFE” boot loader command line. The next step is to issue some commands to clear the non-volatile RAM (NVRAM) then reload the flash from the downloaded copy. But this failed as there was some issue with the download mechanism or format of the internal storage:
After a lot of searching through a sparsely documented command system, I worked out a command to download the same firmware again from my TFTP server. I wanted to see what the “timeout” was about (because all timeout settings were correct on the server). And to my amazement this forced download fixed the problem:
flash -noheader -size=<firmware byte size> 192.168.11.2:firmware.ram nflash1.trx
After that I was able to get to the default HTTP configuration page at the default user IP address of 192.168.1.1 and reconfigure my router. It works fine now.
Forcing a manual download with the ”size” parameter specified worked around the failure of the “automatic” firmware recovery download. There must be some bug in the boot loader with auto-download, perhaps not flushing a download buffer or something like that. I still think DD-WRT and open source router/devices are great, but I’d recommend sticking to the older/stable firmware.
But now I’ve gone back to my pervious routing solution, using a Windows Server 2012 VM, keeping the router as a just fast wireless access point. It’s just more reliable in my experience (even though I do have some complaints about the “RRAS” services of Windows). I used to think Linux boxes were better at stuff like this, but apparently not.
Open source software seems okay, but you can’t beat commercial software quality. On the face of it the Linux boxes look like they have 100s of options, but as soon as you start doing anything more than basic stuff (like domain lookup forwarding) you find complex parameters and obscure file edits are necessary. That’s a stark contrast to the mostly point and click GUI of a Windows server.
If anyone can see why the serial input (keyboard) was out of sync (having to hit each key more than once) even though the output (received data) was perfect, please tell me! Maybe it should be cabled differently or different settings are required? I plan to look into the TTY standard more closely sometime. One thing I avoided was connecting the 3.3v power pin, because according to the Arduino forum that’s the most common way to damage an Arduino.
References & Acknowledgements
Thanks to “Magnetron1.1″ on the DD-WRT forum for providing the information to help me fix my router.
Visit the DD-WRT organization web site to download custom software for your compatible router!
Thanks to Arduino for making such a universal device!