Making work with eMMC interposer slightly more convenient

In one of the previous posts I have described eMMC interposer and how it can help with modifications of the device firmware without having to constantly de-solder and re-solder eMMC. I wanted to take it a step further and extend access to replacement of eMMC and access to UART to outside of the device, so that it can be operated with case fully assembled without the risk of cutting the operator with sharp edges of heatsink. Here is the result:

Gerber files as well as complete PCBs are available from OSHPark

Changing XGS-PON factory settings for Zyxel PM7516-R0

I was able to get ZYXEL XGS-PON 10GE ONT for less than $50 on eBay. I have no real use for it, but I was wondering how difficult it is to change the vendor name, model, GPON SN on this ONT.
It appears it is really easy.

Pop the cover of the ONT. You'll see 5-pin UART header with 4 populated pins. Starting from the side facing Ethernet port the pins are GND, RX, TX, 3v3. Speed is 115200.

Connect a 3.3v UART, power on the ONT and interrupt the boot at U-Boot by pressing Enter.

You will be at a prompt SATURN#. That is U-Boot. MRD data that contains MAC address, GPON serial, etc is stored at /dev/mtd4 in linux. In U-Boot mtd4 is at SPI NAND offset 0x900000 with length 0x100000.

One other interesting thing that is stored in this partition is a clear text SSH root password.

Use the following command to read the contents of MRD partition to RAM:

SATURN# spi_nand read 0x82000000 0x900000 0x100000

SPI-NAND: 1048576 bytes @ 0x900000 Read: OK

SATURN#

Now let's print the contents. (I've removed identifiable data and replaced with XX in hex view)

SATURN# md.b 0x82000000 0x1000

82000000: 5a 59 58 45 4c 00 00 00 00 00 00 00 00 00 00 00    ZYXEL...........

82000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................

82000020: 50 4d 37 35 31 36 2d 52 30 00 00 00 00 00 00 00    PM7516-R0.......

82000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................

82000040: 41 41 42 30 30 32 00 55 53 45 52 3d 72 6f 6f 74    AAB002.USER=root

82000050: 00 53 48 4c 56 4c 3d 31 00 48 4f 4d 45 3d 2f 68    .SHLVL=1.HOME=/h

82000060: XX XX XX XX XX XX 00 ff XX XX XX XX XX XX XX XX    MACADD..XXXXXXXX

82000070: XX XX XX XX XX XX XX XX 00 d0 4d 77 20 e8 8c 7f    XXXXXXXX..Mw ...

82000080: 40 02 67 77 6c e8 8c 7f 53 32 30 30 5a XX XX XX    @.gwl...S200ZXXX

82000090: XX XX XX XX XX 00 00 00 00 00 00 00 00 00 00 00    XXXXX...........

820000a0: 00 00 00 00 00 00 00 00 XX XX XX XX XX XX XX XX    ........GPON_SER

820000b0: XX XX XX XX XX XX 30 30 30 30 30 30 00 00 00 00    IALNUM000000....

820000c0: 00 00 00 00 00 00 00 00 XX XX XX XX XX XX XX XX    ........ADMIN_PA

820000d0: XX XX 00 55 53 45 52 3d 00 XX XX XX XX XX XX XX    SS.USER=.SN81#_N

820000e0: XX XX XX XX XX 00 00 00 00 00 00 00 00 00 00 00    UMBER...........

820000f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................

82000100: 00 00 00 00 00 00 02 00 XX XX XX XX XX XX XX XX    ........BOSA_SER

82000110: XX XX XX XX XX XX 00 00 00 00 00 00 00 XX XX XX    IALNUM.......ROO

82000120: XX XX XX XX XX XX XX 00 55 53 45 52 3d 30 30 30    T__PASS.USER=000

82000130: 30 30 30 30 30 30 30 0a 00 00 00 00 00 00 00 00    0000000.........

82000140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................

82000150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................

82000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................

82000170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................

82000180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................

82000190: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................

Now you can continue booting with bootz command.

Log in with root using SSH or UART using the root password displayed from reading MRD partition. Now you can modify the contents of /dev/mtd4 using linux native commands, or download it from the device, make changes and then go back to U-Boot, erase MTD4, load contents of your modified partition with xmodem using loadx 0x82000000 command, followed by spi_nand erase 0x900000 0x100000 and then spi_nand write 0x82000000 0x900000 0x10000

SATURN# spi_nand erase 0x900000 0x100000

SPI-NAND: 1048576 bytes @ 0x900000 Erased: OK

SATURN# spi_nand write 0x82000000 0x900000 0x100000  

SPI-NAND: 1048576 bytes @ 0x900000 Written: OK

Surprisingly there is no checksum update required when changing values. It just works. I don't have XGS-PON service, but for people who do, this may be a way to get 3rd party ONT to work with your ISP.

Some useful tools provided in linux userspace:

mrd_access - prints MRD settings in user-friendly format

app_cli - Cortina XGS-PON command-line interface (Cisco-like CLI with access to XGS-PON settings, same as telnet localhost 2233)

Modding devices with eMMC flash

A lot of networking device are coming out with eMMC flash storage instead of SPI or even NAND flash, making it difficult to read and write flash out of band. What if we can use pluggable flash modules instead of soldered BGA eMMC flash? It turns out that with some soldering skills you can convert soldered eMMC flash to pluggable module that can be read and written outside of device. Good use cases for this are replacing bad flash or expanding storage on out of warranty Edgerouter 4/6P/12 routers, or accessing filesystems of newer routers used by ISPs.

Warning: This guide describes process that requires specific skills and safety measures. I am not responsible if you brick your device, set your house on fire or kill your neighbors dog while following this guide.

Before attempting this, I have never soldered BGA devices in my life. This takes some care and patience, but if someone like me with no experience can do this by following Youtube videos, it's likely others may also be successful in it. If you want to mod your device and don't feel comfortable doing it yourself, your local cell phone repair store may have the right skill set and equipment to do this for you.

It goes without saying that you should not be modding anything owned by your ISP.

I got inspiration for this from ODROID project, which utilizes pluggable modules for eMMC flash.
The pinouts of connectors described in this guide are fully compatible with ODROID - you should be able to use ODROID compatible flash readers and use ODROID compatible eMMC flash modules with your modded devices (as long as the devices can recognize flash modules).

In this mode we are desoldering existing eMMC flash from device being modded, and replacing it with an interposer - a special PCB with eMMC BGA footprint on one side and ODROID-compatible flash connector on the other side.

I have designed an eMMC interposer using eMMC IC datasheets and ODROID adapter schematics and designed an eMMC module PCB in eMMC 11.5x13mm footprint.
This is the second time I am designing my own PCBs, so the board layouts may look amateurish. But they do work. Data traces lengths are tuned to +/-1 mil.

A picture is worth a thousand words:

What you see on the pictures above is a consumer router, same model that is used by some ISPs that I purchased for experiments. The router's eMMC flash has been desoldered, re-balled and soldered to an ODROID-compatible flash module PCB. An interposer PCB has been re-balled (to behave like an eMMC module) and soldered in place of eMMC.

PCBs for both interposer and eMMC-compatible module can be ordered from OSHPARK for less than $4 for 3 boards.

eMMC interposer

ODROID-compatible eMMC module

The boards are using smaller tolerances than allowed by OSHPARK, so while I've had success with the designs above, it is possible that the boards may not work if the drill is shifted too much from center of vias. Use microscope or zoom in on a picture taken from your phone to inspect the vias. You want the drill hole to be as close to the center of vias as possible, or at least within the surface of the via - not protruding from it's footprint. If the drill hole is too off-center, try ordering again - a different batch may be centered better.

Besides the boards you will need some tools and supplies.

For eMMC interposer you need:

• GB042-34P-H10 connector
• 0.3mm solder balls. Low-temperature solder balls are easier to work with. I've used Sn/Pb solder balls
  

For eMMC module you need more parts:

• GB042-34S-H10 connector
• 0.3mm solder balls to re-ball your eMMC IC (unless you are using a new one factory-reballed)
• Some capacitors and resistors in 0603 SMD form factor listed at OSHPARK board page. 

When ordering parts and PCBs, unless you are very good at SMD soldering, it's best to order more than you need in case you you don't succeed from first attempt.

You would also need some generic tools and supplies:

• Soldering iron with temperature control
• Something that can be used for de-soldering an re-soldering BGA like hot plate, skillet, hot air station, PCB reflow oven, IR rework station. You can find techniques for soldering BGA using the described tools in Youtube. I have used a DIY reflow oven converted from a toaster and a hot air gun (with temperature control).
  
• Digital Multimeter, preferably with thermocouple that can be used to measure board temperature
• Solder wick
• Kapton tape (1 mil thickness)
• Rosin flux
• Isopropyl alcohol for cleaning contacts and removing flux
• Solder paste and/or solder. I like using Sn/Pb, but other low-temperature solders should work too.
• HVAC aluminum tape to cover plastic parts of the board when re-flowing BGA.
• Tweezers
• IC suction tool (not necessary, but helps to cleanly desolder eMMC IC)
• LCD microscope is not strictly necessary if your vision is good, but is very handy to have
• BGA reballing station is not necessary, but makes working with small parts a lot more comfortable. I've used this kind.
• Having eMMC stencils (and PCB stencils) makes it a lot easier to accomplish the steps in this guide. I've used my own eMMC stencil, but generic ones should work too. Below are links to my stencils. I've used oshstencils.com to order the stencils in 4 mil stainless steel. Below are the gerber files for stencils used in this guide:
• eMMC BGA
• Interposer
• eMMC Module

Now let's get to the actual process.

The first step (after carefully disassembling your device) is desoldering eMMC IC from your device. eMMC flash chip is likely soldered with lead-free solder with a melting point over 217°C/422°F. Heat up the eMMC IC, but don't overheat it. Suction tool helps to lift the IC without damage, but you can also use tweezers to carefully flip the chip when solder under the chip melts. To avoid overheating the board and IC you could tape thermal probe to the board next to eMMC IC and monitor the temperature while de-soldering. It is also helpful to preheat the whole board to avoid thermal stress of heating just one part of the board.

I have used a DIY reflow oven set for leaded solder thermal envelope to heat up the whole board to the top of leaded thermal envelope, then removed the board from oven and heated the eMMC some more with hot air until I was able to remove it with suction tool.

I have also wrapped all the plastic parts with HVAC aluminum tape to avoid melting them

The next step is removing lead-free solder from the board and eMMC IC. Use solder wick and either soldering iron or hot air, being careful not to blow away SMD devices next to eMMC landing pad. Use kapton tape to cover SMDs if using hot air. I found that the easiest way to clean the pads is to use a dab of solder paste on solder wick and sliding wick over the pads with soldering iron. Not all the pads of eMMC IC are used. Most of the pads are there just for structural purposes - to hold the IC in place. If some of non-connected pads delaminate when cleaning, it is not the end of the world. If using solder instead of solder paste, use rosin flux. I've tried to be extra careful, but still got a few pads lifted on this board. Likely they are not connected to any traces. After cleaning the surface of the board and surface of eMMC chip need to be completely flat. Use rubbing alcohol to clean flux. 

Now let's attach some solder balls to interposer. Cover interposer with a really thin layer of sticky rosin flux and attach the balls. Using stencil makes this quick and easy. Some people use direct heat stencils with solder paste. Whatever works best for you. Once the solder balls are stuck to flux, heat the board on hot plate, skillet or reflow oven to melt them.

Boards come from OSH Park with tabs used for panelling. Tabs can be easily filed off.

Repeat the process with eMMC IC. Solder the connector to interposer. You could use the stencil and solder paste, or you can hand solder the connector. Check out Youtube videos on techniques to solder 0.4mm pitch connectors. Hand soldering requires a special technique and some skill to avoid bridged contacts.

Before soldering eMMC Interposer to modded device I've decided to solder it to an eMMC to MicroSD card adapter to check if I can read ODROID eMMC modules using this interposer.
I've ordered adapter from OSHPARK using gerbers from here: https://github.com/voltlog/emmc-wfbga153-microsd but factory-made adapters from Ebay or other sites would work too. If ordering eMMC to SD adapter from OSH Park using gerbers from GitHub, use 2oz copper boards - they are thinner and closely match thickness of SDXC cards.

Important note: eMMC Interposer is about five times heavier than regular eMMC IC. Use two layers of 1 mil thick kapton tape as standoff when reflowing eMMC Interposer to avoid solder balls collapse causing bridging and electrical shorts. Cover PCB with a very thin layer of rosin flux and attach kapton tape.

The step above is not strictly necessary. You can use off the shelf ODROID adapter to read flash, but it provides confidence in your soldering skills.

Now that we are confident that our eMMC Interposer works, let's solder one to our device. Again, use two layers of 1 mil thick kapton tape to avoid solder balls from collapsing under the weight of interposer and use a very thin layer of rosin flux. If you use too much flux it may boil and shift the interposer.

I've used reflow oven at leaded solder profile setting. It is not hot enough to melt the solder for the rest of components on this board since it uses lead-free solder, which requires higher temperature to melt than leaded solder.

Now let's assemble the ODROID-compatible eMMC module with eMMC IC we have desoldered from the device. You can use off the shelf ODROID-compatible modules if our device can recognize flash. You would still need the means to read existing flash and copy it to replacement module.

Start with soldering the passive SMD components, then flip the module PCB and solder eMMC IC (that has been previously reballed) to module.

Here you can see comparison between the factory ODROID eMMC module and the one I've soldered myself.

Let's plug the original flash on top of the new module into the device and check if it works.

It does!

Let's try with a different router I've purchased for experiments. Before we tear it apart, let's check that it actually works.

It does.

Heatsink off! Now let's get EM shield off.

I've used the same process to desolder IC as with the blue board router above. I've heated the whole board to leaded solder melting point and used hot air gun on eMMC IC to get it to lead-free solder melting point. I am using PCB to hide serial on label.

Remove lead-free solder and clean eMMC IC

Same with the board. Either the board FR4 rating is higher than for the blue board, or I just got lucky enough not to lift any pads on this device.

Time to solder interposer. Same technique with two layers of kapton tape for standoffs.

This is how it looks after reflowing:

Let's now test with original flash and a copy.

Success!

Time for EM shield and heatsink surgery using dremel.

Can't say it looks good, but it works.

Before attaching eMMC to SD adapter to a PC, make sure to disable automount. Don't try to manipulate block devices that are mounted - you are likely to corrupt the date.

I've used the following commands to make backup copy of contents of eMMC:

sudo dd if=/dev/mmcblk0boot0 of=mmcblk0boot0.img bs=1024k status=progress

sudo dd if=/dev/mmcblk0boot1 of=mmcblk0boot1.img bs=1024k status=progress

sudo dd if=/dev/mmcblk0 of=mmcblk0.img bs=1024k status=progress

To flash this backup to a new eMMC I've used the following:

sudo mmc bootbus set single_backward x1 x8 /dev/mmcblk0

sudo mmc bootpart enable 1 1 /dev/mmcblk0

sudo mmc enh_area set -y 0 1908736 /dev/mmcblk0

(Remove and plug back eMMC)

sudo su

echo 0 > /sys/block/mmcblk0boot0/force_ro

echo 0 > /sys/block/mmcblk0boot1/force_ro

exit

sudo dd of=/dev/mmcblk0boot0 if=mmcblk0boot0.img bs=4k status=progress

sudo dd of=/dev/mmcblk0boot1 if=mmcblk0boot1.img bs=4k status=progress

sudo dd of=/dev/mmcblk0 if=mmcblk0.img bs=4k status=progress