rsync from PC to Android to pull data as root

During my backup adventures, I came across many nice tools, such as SSHDroid and SimpleSSHD that allow me to SSH from my computer to the Android device. They even include niceities like rsync and ssh binaries that are quite useful. Alas, this team failed for me on a particular older phone that I wanted to slurp to the PC "as is" for all accessible files (including system ones) keeping their original permissions and owners, which of course needs root on both sides. That should have been easy to do with rooting support in both Android toolkits, but on this phone the SimpleSSHD did allow me to connect (and with public-key auth available for free, unlike SSHDroid) - but only as a "user", and SSHDroid could not start at all due to some execution errors.

Well, I thought, if I can log in by SSH and do su easily, I can just run the utilities from command line to start the rsync session from the phone to PC's SSH server. Alas, this idea got broken on several accounts:

• First of all, even though I could run both /data/data/berserker.android.apps.sshdroid/dropbear/rsync and /data/data/berserker.android.apps.sshdroid/dropbear/ssh individually, running the common rsync -avPHK /data/ root@mypcIPaddr:/androidbackup/data/ failed again due to permissions error:

  rsync: Failed to exec ssh: Permission denied (13)
  rsync error: error in IPC code (code 14) at jni/pipe.c(84) [sender=3.0.8]
  Segmentation fault

  I tried to wiggle around this by e.g. passing a -e '/data/data/berserker.android.apps.sshdroid/dropbear/ssh' argument, or adding the directory to beginning of PATH - but got the same results.
  I finally got these to work using the system path (and note these must be copies with root ownership - not symlinks to original files). Also mind that the Android/su shell syntax is quirky, to say the least:

  # mount -o remount,rw /system
  # cp /data/data/berserker.android.apps.sshdroid/dropbear/ssh /system/xbin/ssh
  # cp /data/data/berserker.android.apps.sshdroid/dropbear/rsync /system/xbin/rsync
  # chmod 755 /system/xbin/rsync /system/xbin/ssh
  # chown 0:0 /system/xbin/rsync /system/xbin/ssh
  # mount -o remount,ro /system
  

  This got me pretty far, now running just rsync (without prefixing the long path to app instance) became possible, and it could call ssh at least as:

  # rsync -e /system/xbin/ssh -avPHK \
      /data/ root@mypcIPaddr:/androidbackup/files/data/
  /system/xbin/ssh: Exited: Error connecting: Connection timed out
  Segmentation fault
  

  So this was better, but not enough.
• Trying to connect using ssh, or even netcat, to any port of my PC seems impossible :\ Even after I disabled firewalls the best I could.
• Trying to circumvent the firewalls issue, I made an SSH session from the PC to Android with a TCP tunnel, so I could rsyncback from the phone through it:

  root@pc# ssh -R 22222:localhost:22
  root@phone# rsync -e '/system/xbin/ssh -p 22222' -avPHK \
      /data/ root@localhost:/androidbackup/files/data/
  Login for root@localhost
  Password: 
  Segmentation fault 
  

  So this was close - the SSH session from phone to PC got established this way, but rsync still crashed before doing anything meaningful. Similar experiment with netcat tunnels also failed.

Going back to the idea of initiating SSH connections from the PC, which at least works, I tried to make the rsync binary setuid as root:

# chmod 106755 /system/xbin/rsync

But connections from PC to the android, using rsync --rsync-path=/system/xbin/rsync ... failed with permissions error accessing system dirs.

Finally, I made a wrapper script that calls su and this panned out:

#!/system/bin/sh
su -c /system/xbin/rsync "$@"

I saved it as /system/xbin/rsync-su (and also chmodded it like the original programs above), and it worked!

Unfortunately, it seems that the Android-side rsync still crashes after some time or amount of transfers - though not quite repeatable (has long stretches of running well, too), so I wrapped it in a loop and excluded some files it had most problems with (passing over another time, without exclusions, allows to copy the few files by ssh+tar):

root@pc# while ! rsync --timeout=5 \
    --exclude='*.db' --exclude='*.db-*' \
    --partial-dir=.partial -e 'ssh -p 2222' \
    --rsync-path=/system/xbin/rsync-su \
    -avPHK root@phoneIPaddr:/data/ ./files/data/ \
    ; do echo "`date` : RETRY"; sleep 1; done

The sleep 1 is needed to reliably abort the loop by Ctrl+C if I want to stop it and tweak something.

---

UPDATE: in hindsight, maybe I should have looked for luckier builds of the tools as well. The Apps2SD project includes a formidable collection of programs, delivering busybox and rsync in particular. There are also binary builds for various platforms at https://github.com/floriandejonckheere/rsync-android. But since my immediate pain has been resolved by a stone-hammer described in this post, I did not look into other possibly finer tools, at least not on this phone (this is recovery after all - even adding SW there is problematic).

---

For completeness, the target directory on the PC contains a few scripts I conjured up in haste, and a files/ subdirectory into which Android content lands.
One is dubbed tarcp, which does the copying with tar and ssh:

#!/bin/bash

set -o pipefail

PHONE_IP=192.168.42.129
PHONE_SSHPORT=2222

# A files/ should be under CWD
ssh -p "$PHONE_SSHPORT" "root@$PHONE_IP" \
  'su -c "cd / && tar cvf - \"'"$@"'\" "' \
  | (cd files/ && tar xvf - )

This is sort of wasteful, because when retries are needed (wifi flakiness, etc.) it re-copies the whole set of arguments.
Another is loopcp which calls the first one in a loop - to do those retries if needed:

#!/bin/sh
for D in "$@" ; do while ! time ./tarcp "$D" ; do \
  echo "`date`: RETRY" ; sleep 1; done; done

Finally, there is a tool to help determine directory sizes on Android from PC, so I could copy-paste a lot of relatively small targets for loopcp to run over, and retries are relatively cheap (not like re-pulling several gigabytes over and over, after getting a last-minute hiccup):

#!/bin/sh

PHONE_IP=192.168.42.129
PHONE_SSHPORT=2222

#echo \
ssh -p $PHONE_SSHPORT root@$PHONE_IP 'su -c "cd / && \
   du -ks '"$1"'/* | while read S D \
      ; do [ \"\${S}\" -lt 100000 ] && \
        printf \"\\\"\$D\\\" \" ; done ; \
   echo ; echo ; \
   du -ks '"$@"'/* | sort -n | while read S D \
      ; do [ \"\${S}\" -ge 100000 ] && \
        printf \"\$S\t\$D\n\" ; done ; echo"'

This outputs a long string of double-quoted subdirectory names (or files) present in the argument-directory and are under 100Mb in size, and prints a sorted (by size) list of larger objects. The long string can be copy-pasted as argument list for loopcp; the list of larger objects is for drilling into them and breaking their contents into another string of small chunks of work.

Actually, this set of scripts is what I started with after initial failures with rsync, but in the end the tedious breaking down the list of arguments for "loopcp" and the uncertainty that I got all the bits and FS rights correct in the copy, pushed me to find a way to run the rsync after all - though most of the file content was copied by that time, using the scripts.