Reigning in the Gen2

Hey kids–

I have been frustrated quite frequently lately by the “no touchy!” nature of the Gen2 mycloud’s firmware, and am considering a somewhat dangerous/radical modification to my unit. Specifically, I am intending to do the following, but first a rundown of how Gen2 boots. I have no interest in mucking with the kernel image or the initrd, just to be clear. (I would like to USE my nas, not brick it.)

Early in the boot process, the gen2 firmware pushes several important system values to various locations in /proc, mounts a cramfs container from /boot, then passes execution there using /etc/rc.sh

#!/bin/sh

MOUNT_CMD="busybox mount"
UMOUNT_CMD="busybox umount"

${MOUNT_CMD} -o remount -w %root% /

echo "** Mounting /etc/fstab"
${MOUNT_CMD} -a
${UMOUNT_CMD} /proc
${UMOUNT_CMD} /usr/local/modules
sleep 1

${MOUNT_CMD} -t proc proc /proc
${MOUNT_CMD} -a

# Cgroup support
CGROUP_ROOT=/sys/fs/cgroup
${MOUNT_CMD} -t cgroup -o rw,memory,cpu cgroup ${CGROUP_ROOT}

echo 8192 > /proc/sys/vm/min_free_kbytes
echo 4096 > /proc/sys/net/core/somaxconn
echo 16777216 > /proc/sys/net/core/wmem_max
echo 16777216 > /proc/sys/net/core/rmem_max
echo 163840 > /proc/sys/net/core/wmem_default

–break–

echo 163840 > /proc/sys/net/core/rmem_default
echo 3000 > /proc/sys/net/core/netdev_max_backlog

echo 1800 > /proc/sys/net/ipv4/tcp_keepalive_time
echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout
echo 2048 > /proc/sys/net/ipv4/tcp_max_syn_backlog
#echo 1 > /proc/sys/net/ipv4/tcp_syncookies
echo 0 > /proc/sys/net/ipv4/tcp_timestamps

echo /sbin/mdev > /proc/sys/kernel/hotplug
mdev -s
touch /dev/mdev.seq

#for judge kernel or ramdisk if error
#mknod -m  777 /dev/REG c 88 0
#busybox insmod /lib/modules/reg.ko

#ubiattach /dev/ubi_ctrl -m 5 -O 2048
sync
sleep 1

–break–

#${MOUNT_CMD} -t ubifs ubi0:config /usr/local/config


mkdir -p /boot
${MOUNT_CMD} -o rw /dev/sda3 /boot
#sleep 1
#${MOUNT_CMD} -t squashfs -o loop /boot/boot/image.cfs /usr/local/modules

if [ ! -e /usr/local/config ]; then
        mkdir -p /usr/local/config
fi

mount /dev/sda7 /usr/local/config

chk_image

/usr/local/modules/script/system_init

Now, the total size of the cramfs’s contents is smaller than the free space on the persistent config partition (HD_a7).

root@WDMyCloud /etc # df
Filesystem           1K-blocks      Used Available Use% Mounted on
%root%                   55529     33817     18845  64% /
/dev/ram0                55529     33817     18845  64% /
mdev                    257248        32    257216   0% /dev
/dev/sda3               999576    211756    761116  22% /boot  <--- Free space on /boot! *squee*
/dev/sda7               999576      1616    971256   0% /usr/local/config  <----approx 1Gb free!
/dev/loop0              102144    102144         0 100% /usr/local/modules <---Approx 100mb used!
tmpfs                     1024         0      1024   0% /mnt
tmpfs                    40960      5632     35328  14% /var/log
tmpfs                   102400      1760    100640   2% /tmp
/dev/sda4               951192      3600    931208   0% /mnt/HD_a4
/dev/sda2            3837304328 442537388 3355764440  12% /mnt/HD/HD_a2

This means we should be able to create a folder inside /usr/local/config, and completely mirror the contents of the cramfs container into it, then replace the cramfs container with a hyper minimal one that JUST contains the system_init script, with some changes at the very beginning for housekeeping purposes.

So, we want it to do the following:

unmount the cramfs container from /usr/local/modules, so that our dummy container is out of the way. Mount -o bind our /usr/local/config/cramfsmirror at /usr/local/modules so that our writable surrogate is in the expected place, then perform some root filesystem cleanups, like putting in symlinks for important things in /etc to /usr/local/config before any system daemons get started then:

continue normally

This would allow changes made after the system is fully up to become “persistent”, in as much as those changes will get restored after a reboot, and before system daemons start. This includes changes to the dashboard UI, and pals, as well as (theoretically) the crontab, and other things.

I think I can automate the entire process of performing this kind of modification using the USB startup script method, so that I can distribute a .zip for the USB stick’s contents, which when inserted at boot, will do the needful (clone existing cramfs contents to the new location [tar is present, so we can use it through some pipes to keep ownership and permissions intact, instead of using cp], unmount the cramfs container, rename the container’s file (deletion seems unnecessary), copy the new dummy one over, then gracefully down the nas). Remove the stick, then power on-- persistent system mods enabled.

Any thoughts or suggestions before I dig into this tomorrow?

When you say USB startup script you mean the fun_plug file?

exactly. Even though it runs at the END of system_init, (and thus after all the daemons are up), it does not matter, as we can down them all if we need to, and /boot is already mounted rw by default. :smile: We just need to do the needful, unmount the old container (so it is not in use), rename the container file, and copy the new one on. Bob’s your uncle.

Hey @rac8006 , Did you ever get your hands on the gen2’s kernel source code? One thing I would LOVE to add is the zram module. That would help alleviate just about all instances where the drive does not enter sleep mode. (the kernel already supports anonymous paging, and loadable kernel modules (see where it loads one in rc.sh above). I just need the appropriate kernel sourcecode so that my module’s magic number is right, and will load. We dont need a terribly large zram swap, just one that will get higher swap priority than the actual swap partition, so that if a process needs some swap, it does not need to wake the disk. A 100mb zram swap should be more than sufficient for this task, unless the system gets very heavy load.)

Also… some hidden jewels…

root@WDMyCloud / # find / -name *.ko
/usr/local/modules/driver/ipip.ko
/usr/local/modules/driver/iscsi_target_mod.ko
/usr/local/modules/driver/iscsi_tcp.ko
/usr/local/modules/driver/jnl.ko
/usr/local/modules/driver/libiscsi.ko
/usr/local/modules/driver/libiscsi_tcp.ko
/usr/local/modules/driver/netatop.ko
/usr/local/modules/driver/scsi_transport_iscsi.ko
/usr/local/modules/driver/target_core_file.ko
/usr/local/modules/driver/target_core_iblock.ko
/usr/local/modules/driver/target_core_mod.ko
/usr/local/modules/driver/tun.ko
/usr/local/modules/driver/tunnel4.ko
/usr/local/modules/driver/ufsd.ko

Lookie lookie-- iSCSI modules! Not loaded by default, but we can TOTALLY fix that. :stuck_out_tongue:

I have downloaded the gen2’s GPL code which contains kernel code. Just don’t have the build steps. Like what is setup in the Kconfig. I would like to gen a kernel that would allow blktrace to run. I’ve got a version of blktrace that executes on gen2 but the kernel is not configured correctly. I’ve also got debugfs working on the gen2. So if I can get blktrace working I can see what disk access wakes up the disk. On the gen1 it was mostly kjournal writes that woke up the disk.

Where did you pull it from? I would like to take a look. Usually, ‘make oldconfig’ will work to populate kconfig’s settings with the oem’s defaults, if they provided any.

Here is a link to the GPL download links for 3.x 4.x and 2.x firmware.

https://support.wdc.com/product.aspx?ID=904&lang=en#WD_downloads

Thank you kindly sir! I will assault it shortly.

Did somebody say zram? I second that in a big way. Zram would be great for the Gen2 and an absolute godsend to the Gen 1 units for some compressed ram swap.

I will try building for Gen2 first, so I can actually test my modules after building them.

I dont have a Gen1 to test on, but I can also pull gen1 gpl sources for the latest available firmware (Not the bleeding edge ones, they dont have gpl source up yet, and because these are kernel modules, THE KERNEL MUST MATCH, no exceptions!) and try building them. If you have a Gen1 and feel like testing them, I can maybe hook you up. We will see. Depends on how successful I am building Gen2 sources. (Need to be sure my build env is set up right for this source package.)

If I am successful, it would be something that @fox_exe could bundle into a native application, but be aware that on firmware upgrade, the modules would need to be updated! Packaging into a native application would be useful for creating a dashboard control to set zram size, mount / unmount the swap from zram0, check compression ratio, etc. (easily done by querying the entries in /sys/block/zram0 after loading the needed modules. insmod lets you specify where the module to load is located, so this is totally doable I think. Just that web dev is outside my skill set, so building the gui goodies is not for me.)

Best way - Use OMV or Nas4Free (FreeNas?) and just add few modules for support our device (+ Module for cloud access).
Too many work for one man.

Wierd_w- I’d be glad to test a zram module for Gen1 if you manage to build one. The Gen1 unit I’d be testing on is on v04.04.03-113 firmware with a 3.2.26 kernel. Here is a link to the sources for that version. It needs to be staying on that version for now (because automatic Safepoints became broken in later versions and supposedly taken out of the latest firmware) so it also avoids the upgrade caveats you pointed out.

Most people interested in zram probably won’t expect GUI goddies with it. I sure wouldn’t.

In addition to that link one can also find direct download links for the previous GPL firmware and WD released firmware downloads in the following thread:

http://wdstagingsplit.staged-by-discourse.com/t/wd-my-cloud-v3-x-v4-x-and-v2-x-firmware-versions-download-links/148533

Ok, I successfully build and installed the needed modules for zram on gen2.

those are zlib.ko, zsmalloc.ko, and zram.ko

zsmalloc needs a tweak to its Kconfig to turn it from a boolean to a tristate, or else you cannot build it as a module. (checked upstream to be sure it can be built as one first)

I am going to try actually allocating a zram disk, and using it now. If I am successful, (you have no idea how hard it was to make their cross compiler work. Their build scripts stomp ALL OVER the path variable, which breaks everything. You have to modify them to be sane before you can use them. Morons! Given the quality of this build chain cough, I feel it necessary to fully test the module rather than just accept successful integration) I will build for Gen1

–EDIT

Nope, the old staging drivers in this kernel sources package is buggier than a bowl of mealworms. System hangs shortly after trying to write to the freshly created block device. I am going to try compiling a newer version of zram.c and zmalloc_main.c against this kernel tree. Will walk up the commit logs until I find appropriate patches to get these out of staging, then cross my fingers.

I know, there is the whole uboot process, where it initializes its ram, checks the flash area, sets some SoC specific variables for the board, and all kinds of fun stuff.

But I think I mentioned that I do not want to go that far up the chain, when I said I had no interest in mucking with the kernel image or the initrd, because I dont want to brick the device.

(embedded devices like this tend to use digital signature checking on kernel images and initrds, and get … cranky… when they dont get what they expect. Keeping all that intact, and just grabbing control further down is safer.)

This is true-- I DO have the appropriate cable for this though, and do know how to attach to it. I just dont want to shred the case off unless I really have to.

Do you know anyone that has the UART connected to a gen2 and that works?

I have done this many times with other embedded devices. Assuming they implemented the posts (and I dont have to go in there with solder to add them-- I really hate it when oems do that.), I have the necessary 3v FTDI to USB serial cable to connect with this UART and listen without much pain. I have a multimeter, and can find which is the tx, rx, and gnd pretty easily.

So no. But I can do this myself.

Reading the dmesg output of the device indicates that there IS a UART, and that the kernel takes possession of it in the boot process.

[    0.000000] Kernel command line: root=/dev/ram console=ttyS0,115200

Yes there is a UART on the Gen1 and Gen2. I’ve hooked up the Gen1 with no problem.
But when I hook up the gen2 I get no output. Not sure why.

Does it have hardware cls? It might be more than 3 wire serial. Might be 4 or 5 wire. How many posts are there? (it might have independent grounds for rx and tx as well, so that while it is raising its tx high, you are not getting conductivity because the gnd you are connected to does not close the circuit. You might need jumper wires to connect the grounds together so your cable works.)