Monday, March 12, 2018

Moving and renaming projects in OBS

For some part of our software project lifecycle with http://42ity.org/ we use an on-premise setup of OBS aka OpenSUSE Build Service, at least initially that was the name (not to be confused with Open Broadcaster Software) ;) Having a local setup allows us to modify and manage it in ways the common cloud service can not be changed to our whim.

One such issue we were facing recently was that a few packages (ultimate recipes) were placed into wrong projects or sub-projects (a scope to group packages for the purposes of dependencies, upgrades, etc.). This caused a mess, because beside the pedantic "it is not clean" complaints, packages at different levels ended up building against different dependencies - and it this case it was not intended (sometimes it is, hence the scoping). Packages have a history that tracks evolution of a recipe, and sometimes it is important to keep it for development reference, or even as part of the workflow in our case (determine whether *this* version of the Git sources was packaged earlier).

Unfortunately, OBS does not directly support moving such data around nor even renaming packages. You can "branch" packages (to track an original package's changes and merge them with customized bits of the clone), but that's about it. Thanks to help on the IRC channel, however, we found a way to do it on the backend, fiddling under the hood.

BEWARE THAT SUCH FIDDLING CAN EAT YOUR DATA AND BLOW UP YOUR SERVER!
Take backups or snapshots before following these notes!

Now, it is important to keep in mind that OBS is a wad of scripts written in several languages for different purposes, such as a Ruby web-frontend, a perl backend for scheduling etc., shell scripts to deal with OS nuances such as build-root setup... and the backend involves (and represents) a database to keep some info and claim unique IDs among other stuff, while a lot is kept in arcane directory structures as files placed in expected locations with expected names and contents and role in the overall solution. So fiddling under the hood is tweaking the implementation detail, and might not work the same way in all versions (FWIW, ours is based on a Nov 2015 release) and might have poorly traceable consequences. Still, it worked for us once so I decided to record the experience :)

1. Identify the poorly named package. For the recent example's sake, it will be a "fty:master:/appliance:/fosspkg" which used a third-party FOSS package "fosspkg" almost verbatim - just added the pkgconfig files that configure scripts in some of our components relied on and the original code did not provide. The problem here was that the top-level project "fty:master" ended up using the upstream distributions' version of the package for its builds and tests, while its sub-project "appliance" used another - so preinstall images (made in top-level project) did not match, and our own common components subsequently placed into the top-level project did not even build (needed those pkgconfig files). We want this "fosspkg" recipe moved upwards, to the top-level project.

2. Create a new empty "fty:master:/fosspkg" recipe, using the common web-gui "Create package" link (or CLI, or REST API) so the system properly assigns the new IDs and other resources it wants.

3. Fire up an SSH session to the OBS master server, where the fun happens henceforth. If you used packaged setup initially, the data to modify will be in several locations under "/srv/obs" directory:
  • /srv/obs/sources/fosspkg contains the source code (reused by all packages with the same name located under different projects), and likely remains unchanged.
  • /srv/obs/trees/fty:master:appliance/fosspkg and /srv/obs/trees/fty:master/fosspkg contain files with references to metadata for each commit into the repo (list of source and other files with their hashes that comprise this or that revision of the recipe) - just copy the files from original version into the newly created one.
  • /srv/obs/projects/fty:master:appliance.pkg/ and /srv/obs/projects/fty:master.pkg/ directories contain some *.xml, *.del, *.rev and *.mrev files for each package in this project (e.g. fosspkg.xml and fosspkg.mrev for initial state). Copy over the original package's REV file, migrate carefully the XML file contents, into the new location. TODO: Not sure what should be done with the MREV file - is it safe replace the new one with the old one (moved away into "fosspkg.mrev.del")?
    • The XML file contains the description, title, enabled build targets (if customized compared to the project level) and such - you might transplant the description for example, if you haven't done so with web-gui already; note that COPYING the XML file verbatim is not a good idea, as it references the "project" in its top tag (can copy and edit, though, if there were many custom settings);
    • The REV text-table file contains the actual history of the package - which revision was done when and which number it was in the order of succession, as well as other revision-specific details; is not initially there before the first upload of actual contents into the package;
    • The MREV file seems track the initial creation of the component? :)
    • The DEL files track components that were recipe'd before, but have since been deleted.
  • /srv/obs/repos/fty:/master:/appliance/Distro_X.Y/ and /srv/obs/repos/fty:/master/Distro_X.Y/ contain build products of the project (including package sources and architecture-specific subdirectories). To avoid rebuilds, you can copy over the products from the old location into the new one (this might make sense if builds are costly, but if the actual dependencies have changed due to relocation - you might miss out on something here). To free up resources of the server, you can otherwise remove the build products of the original package you are essentially removing.
  • /srv/obs/build/fty:master:appliance/Distro_X.Y/ARCH/ and /srv/obs/build/fty:master/Distro_X.Y/ARCH/ contain the latest build results (binaries, logfile, reason) as well as the history of builds (when, what, how long) - move it over as well, to keep the history tracking. Beside the "fosspkg" package subdirectory under ARCH, note also the ":full", ":repo", ":logfiles.*" and ":meta" ones - move over their contents for "*fosspkg*" matching files as well.
  • Revise with a shell command like:
    # find /srv/obs -name '*fosspkg*' 2>/dev/null | grep master
    that there is nothing unexpected under any "appliance" related locations.
  • Refresh the new package's page in OBS web-gui. It may display that the builds are "broken" since it has no metadata about last build in its binary tables. Go to Repositories tab to "Explicitly disable" and then "Take default" the Build Flag - if all went well, the system should discover that it was built, and the status will become "succeeded" with the old build's logs seen upon clicking.
  • If the original recipe was "Branched" into other projects, branch the new one into desired locations; carry over customizations that the branch might have (if any) - the web-gui might help with its show of differences.
  • Semi-Finally, disable all the flags in Repository tab of the original obsoleted recipe (so it is in fact not built nor used), and perhaps use
    osc wipebinaries --build-disabled fty:master:appliance fosspkg
    - hopefully this would e.g. rectify the binary metadata we did not touch under the "build" directories.
  • Note that as far as OBS recalculating the dependencies, nothing happened (via web-gui) to trigger rebuilds of stuff that is impacted by your changed package. Probably it is up to you to determine the scope of fallout and trigger the rebuilds.
    • For greater consistency, and if your time/cost constraints permit (this sort of things is always done in a rush, right?) go into the new component's build states (in the Web-GUI) and click Trigger Rebuild. This would ensure the build products are re-done honestly, and downstream stuff is triggered honestly.
    • You might have some luck with e.g.
      # osc whatdependson fty:master fosspkg Debian_8.0 x86_64
      at least for dependent packages right inside the same project level.
    • Also a direct search in the filesystem like
      # grep fosspkg /srv/obs/build/fty\:master/Debian_8.0/x86_64/*/.meta.success
      might help.
  • Finally, when you're sure the new one works well - delete the original with web-gui.

Friday, June 23, 2017

rsync from PC to Android to pull data as root

During my backup adventures, I came across many nice tools, such as SSHDroid and SimpleSSHD that allow me to SSH from my computer to the Android device. They even include niceities like rsync and ssh binaries that are quite useful. Alas, this team failed for me on a particular older phone that I wanted to slurp to the PC "as is" for all accessible files (including system ones) keeping their original permissions and owners, which of course needs root on both sides. That should have been easy to do with rooting support in both Android toolkits, but on this phone the SimpleSSHD did allow me to connect (and with public-key auth available for free, unlike SSHDroid) - but only as a "user", and SSHDroid could not start at all due to some execution errors.

Well, I thought, if I can log in by SSH and do su easily, I can just run the utilities from command line to start the rsync session from the phone to PC's SSH server. Alas, this idea got broken on several accounts:
  • First of all, even though I could run both /data/data/berserker.android.apps.sshdroid/dropbear/rsync and /data/data/berserker.android.apps.sshdroid/dropbear/ssh individually, running the common rsync -avPHK /data/ root@mypcIPaddr:/androidbackup/data/ failed again due to permissions error:
    rsync: Failed to exec ssh: Permission denied (13)
    rsync error: error in IPC code (code 14) at jni/pipe.c(84) [sender=3.0.8]
    Segmentation fault
    I tried to wiggle around this by e.g. passing a -e '/data/data/berserker.android.apps.sshdroid/dropbear/ssh' argument, or adding the directory to beginning of PATH - but got the same results.
    I finally got these to work using the system path (and note these must be copies with root ownership - not symlinks to original files). Also mind that the Android/su shell syntax is quirky, to say the least:
    # mount -o remount,rw /system
    # cp /data/data/berserker.android.apps.sshdroid/dropbear/ssh /system/xbin/ssh
    # cp /data/data/berserker.android.apps.sshdroid/dropbear/rsync /system/xbin/rsync
    # chmod 755 /system/xbin/rsync /system/xbin/ssh
    # chown 0:0 /system/xbin/rsync /system/xbin/ssh
    # mount -o remount,ro /system
    
    This got me pretty far, now running just rsync (without prefixing the long path to app instance) became possible, and it could call ssh at least as:
    # rsync -e /system/xbin/ssh -avPHK \
        /data/ root@mypcIPaddr:/androidbackup/files/data/
    /system/xbin/ssh: Exited: Error connecting: Connection timed out
    Segmentation fault
    
    So this was better, but not enough.
  • Trying to connect using ssh, or even netcat, to any port of my PC seems impossible :\ Even after I disabled firewalls the best I could.
  • Trying to circumvent the firewalls issue, I made an SSH session from the PC to Android with a TCP tunnel, so I could rsyncback from the phone through it:
    root@pc# ssh -R 22222:localhost:22
    root@phone# rsync -e '/system/xbin/ssh -p 22222' -avPHK \
        /data/ root@localhost:/androidbackup/files/data/
    Login for root@localhost
    Password: 
    Segmentation fault 
    
    So this was close - the SSH session from phone to PC got established this way, but rsync still crashed before doing anything meaningful. Similar experiment with netcat tunnels also failed.
Going back to the idea of initiating SSH connections from the PC, which at least works, I tried to make the rsync binary setuid as root:
# chmod 106755 /system/xbin/rsync
But connections from PC to the android, using rsync --rsync-path=/system/xbin/rsync ... failed with permissions error accessing system dirs.

Finally, I made a wrapper script that calls su and this panned out:
#!/system/bin/sh
su -c /system/xbin/rsync "$@"
I saved it as /system/xbin/rsync-su (and also chmodded it like the original programs above), and it worked!

Unfortunately, it seems that the Android-side rsync still crashes after some time or amount of transfers - though not quite repeatable (has long stretches of running well, too), so I wrapped it in a loop and excluded some files it had most problems with (passing over another time, without exclusions, allows to copy the few files by ssh+tar):
root@pc# while ! rsync --timeout=5 \
    --exclude='*.db' --exclude='*.db-*' \
    --partial-dir=.partial -e 'ssh -p 2222' \
    --rsync-path=/system/xbin/rsync-su \
    -avPHK root@phoneIPaddr:/data/ ./files/data/ \
    ; do echo "`date` : RETRY"; sleep 1; done

The sleep 1 is needed to reliably abort the loop by Ctrl+C if I want to stop it and tweak something.


UPDATE: in hindsight, maybe I should have looked for luckier builds of the tools as well. The Apps2SD project includes a formidable collection of programs, delivering busybox and rsync in particular. There are also binary builds for various platforms at https://github.com/floriandejonckheere/rsync-android. But since my immediate pain has been resolved by a stone-hammer described in this post, I did not look into other possibly finer tools, at least not on this phone (this is recovery after all - even adding SW there is problematic).


For completeness, the target directory on the PC contains a few scripts I conjured up in haste, and a files/ subdirectory into which Android content lands.
One is dubbed tarcp, which does the copying with tar and ssh:
#!/bin/bash

set -o pipefail

PHONE_IP=192.168.42.129
PHONE_SSHPORT=2222

# A files/ should be under CWD
ssh -p "$PHONE_SSHPORT" "root@$PHONE_IP" \
  'su -c "cd / && tar cvf - \"'"$@"'\" "' \
  | (cd files/ && tar xvf - )
This is sort of wasteful, because when retries are needed (wifi flakiness, etc.) it re-copies the whole set of arguments.
Another is loopcp which calls the first one in a loop - to do those retries if needed:
#!/bin/sh
for D in "$@" ; do while ! time ./tarcp "$D" ; do \
  echo "`date`: RETRY" ; sleep 1; done; done
Finally, there is a tool to help determine directory sizes on Android from PC, so I could copy-paste a lot of relatively small targets for loopcp to run over, and retries are relatively cheap (not like re-pulling several gigabytes over and over, after getting a last-minute hiccup):
#!/bin/sh

PHONE_IP=192.168.42.129
PHONE_SSHPORT=2222

#echo \
ssh -p $PHONE_SSHPORT root@$PHONE_IP 'su -c "cd / && \
   du -ks '"$1"'/* | while read S D \
      ; do [ \"\${S}\" -lt 100000 ] && \
        printf \"\\\"\$D\\\" \" ; done ; \
   echo ; echo ; \
   du -ks '"$@"'/* | sort -n | while read S D \
      ; do [ \"\${S}\" -ge 100000 ] && \
        printf \"\$S\t\$D\n\" ; done ; echo"'
This outputs a long string of double-quoted subdirectory names (or files) present in the argument-directory and are under 100Mb in size, and prints a sorted (by size) list of larger objects. The long string can be copy-pasted as argument list for loopcp; the list of larger objects is for drilling into them and breaking their contents into another string of small chunks of work.

Actually, this set of scripts is what I started with after initial failures with rsync, but in the end the tedious breaking down the list of arguments for "loopcp" and the uncertainty that I got all the bits and FS rights correct in the copy, pushed me to find a way to run the rsync after all - though most of the file content was copied by that time, using the scripts.

Wednesday, June 21, 2017

Tarballing Android files and partitions over adb

Some time back I was recovering a phone, and before I made something irreversible, I opted for a low-level backup. This copy of old data helped somewhat in migration of program configs to my next phone, too -- although changes between Android 4 and Android 6 did not make this an easy task, clobbering everything with security features ;)

Of course, this adventure required a lot of googling, and posting back as I learned, so I want to save as a cheat-sheet a solution I used: https://android.stackexchange.com/questions/85564/need-one-line-adb-shell-su-push-pull-to-access-data-from-windows-batch-file

Note that this was done using a Windows PC and corresponding tools and drivers, and the Linux side might have different caveats. And of course Android side was rooted, USB debugging allowed, busybox installed, etc. - thanks to a handy habit to do this early, while deploying a new phone.

So, a slightly edited copy of my post from SO follows:

First thing to note is that the adb shell generally sets up a text terminal (so can convert single end-of-line characters to CRLF, mangling binary data like partition images or TAR archives). While you can work around this in Unix/Linux versions of adb (e.g. add stty raw to your shelled command) or use some newer adb with exec-out option, on Windows it still writes CRLF to its output. The neat trick is to pass data through base64 encoding and decoding (binaries are available for Windows en-masse, google them). Also note that errors or verbose messages printed to stderr in shell end up on stdout of the adb shell program in the host system - so you want to discard those after inevitable initial experimentation.

Here goes (note that in command examples below, long command lines are wrapped for readability - you will need to unwrap them back while copy-pasting... not that my examples would match your phones exactly anyway):

adb shell "su -c 'cat /dev/block/mmcblk0p25 | base64' 2>/dev/null" \
  | base64 -d > s3-mmcblk0p25.img

Can be easily scripted by Windows cmd.exe shell to cover all the partitions (one can look the list up by ls -la /dev/block/ and/or by cat /proc/diskstats or by cat /proc/partitions), e.g.:

for /L %P in (1,1,25) do ..\platform-tools\adb shell \
  "su -c 'cat /dev/block/mmcblk0p%P | base64' 2>/dev/null" \
  | base64 -d > s3-mmcblk0p%P.img

(Note to use %%P in pre-created and saved CMD batch files, or %P in interactive shell).

Don't forget that there are also mmcblk0boot[01] partitions, and that the mmcblk0 overall contains all those partitions in a GPT wrapping, just like any other harddisk or impersonator of one :)

To estimate individual partition sizes, you can look at the output of:

fdisk -u -l /dev/block/mmcblk0*

Unfortunately, I did not quickly and easily manage to tar cf - mmcblkp0* and get the partition contents, so I could pipe it to e.g. 7z x -si and get the data out as multiple files in a portable one-liner as well.

To tar some files you can:

adb shell "su -c 'cd /mnt/data && tar czf - ./ | base64' \
  2>/dev/null" | base64 -d > s3-mmcblk0p25-userdata.tar.gz

Of course this solution is not perfect due to increase of transferal time by one third, but given that for recovery purposes grasping for any straws that can help is good enough... well... :) In any case, if I send a whole partition for export like this, I do not really care if it takes an hour vs hour and a half, as long as it reliably does the job. Also if, as some others have documented, the Windows variant of adb.exe does effectively always output CRLF separated text, there seems little we can do but abuse it with ASCII-friendly encapsulation. Alternately, adb exec-out might be the solution - but for some reason did not work well for me.

Hope this helps someone else, Jim Klimov

Wednesday, March 15, 2017

Digging in the Git history: cheatsheet

I've recently had to rewrite git history of some internal projects going opensource, and wanted to eliminate issues that should not be seen in the wild - like hardcoded testing passwords.

For a task like this it does not suffice to just add a commit that replaces those values with expansible variables populated from elsewhere (e.g. a testbed-local config file). One should also dig through all commits to make sure the string does not pop up anywhere in the history over the years (and git rebase / edit / continue rebase the offending commits as if we were smart enough to do this initially years ago).

There is quite a bit of mixed documentation on this, both in official docs and good suggestions in some blogs and stackexchange forums, but it took considerable time to end up with a few one-liners to do the job, so lest I forget -- I'd just post them here. After all, they can be useful to just find needles in a haystack too, such as finding which commits had to do with a certain keyword.

Finally note, that this script is not optimized for performance etc., but rather for readability and debugging of the procedure ;)
#! /bin/sh

### Copyright (C) 2017 by Jim Klimov

usage() {
    cat << EOF
You can call this script as
    $0 find_commits_contains [PATTERN]
    $0 find_commits_intro [PATTERN]
    $0 find_commits_drop [PATTERN]
    $0 fix_history [PATTERN] [REPLACEMENT]

Note that PATTERN and REPLACEMENT are fixed strings (not regexes) in this context
For the SED usage there is also a PATTERN_SED that you can optionally export
EOF
}

[ -n "${PATTERN-}" ] || PATTERN="needle"
[ -n "${REPLACEMENT-}" ] || REPLACEMENT="@CUSTOM_NEEDLE@"
### This script runs from the basedir of a git repo (or a few)
### Logs and other work files are commonly stored in the parent dir by default
[ -n "${LOGDIR-}" ] || LOGDIR="`pwd`/.."
[ -n "${LOGTAG-}" ] || LOGTAG="$(basename `pwd`)"

find_commits_contains() {
    ### This lists which "$COMMITID:$PATHNAME:$LINETEXT" contain the PATTERN in the LINETEXT
    ### Note that this lists all commits whose checked-out workspace would have the pattern
    [ -n "${WORKSPACE_MATCHES_FILE-}" ] || WORKSPACE_MATCHES_FILE="${LOGDIR}/gitdig_commits-workspace-contains__${LOGTAG}.txt"
    git grep "${PATTERN}" $(git rev-list --all --remotes) \
    | tee "${WORKSPACE_MATCHES_FILE}"
}

show_commits_color() {
    ### This finds which commits dealt the pattern (whose diff adds or removes a line with it, or has it in context)
    ### Note that this starts with a colorful depiction of first-pass diffs, for esthetic viewing pleasure
    ### and is parsed including color markup by other tools below
    git rev-list --all --remotes | \
        while read CMT ; do ( \
            git show --color --pretty='format:%b' "$CMT" | \
            egrep "${PATTERN}" && echo "^^^ $CMT" \
        ) ; done
}

find_commits_intro() {
    ### This finds which commits INTRODUCE the pattern (whose diff adds a line with it)
    ### Note that this starts with a colorful depiction of first-pass diffs, for esthetic viewing pleasure
    [ -n "${COMMIT_INTRODUCES_FILE-}" ] || COMMIT_INTRODUCES_FILE="${LOGDIR}/gitdig_commits-intro__${LOGTAG}.txt"
    show_commits_color | tee "${COMMIT_INTRODUCES_FILE}".tmp

    ### ...and this picks out the lines for commits which actually add the PATTERN
    ### (because there are also not interesting context and removal lines as well,
    ### which should disappear after the rebase)
    echo "LIST OF COMMITS THAT INTRODUCE PATTERN (cached in ${COMMIT_INTRODUCES_FILE}.tmp) :"
    cat "${COMMIT_INTRODUCES_FILE}".tmp | \
    egrep '^([\^]|.*32m\+)' | \
    ggrep -A1 '32m\+' | \
    grep '^\^' \
    | tee "${COMMIT_INTRODUCES_FILE}"
}

find_commits_drop() {
    ### This finds which commits REMOVE the pattern (whose diff drops a line with it)
    ### Note that this starts with a colorful depiction of first-pass diffs, for esthetic viewing pleasure
    [ -n "${COMMIT_DROPS_FILE-}" ] || COMMIT_DROPS_FILE="${LOGDIR}/gitdig_commits-drop__${LOGTAG}.txt"
    show_commits_color | tee "${COMMIT_DROPS_FILE}".tmp

    ### ...and this picks out the lines for commits which actually add the PATTERN
    ### (because there are also not interesting context and removal lines as well,
    ### which should disappear after the rebase)
    echo "LIST OF COMMITS THAT DROP THE PATTERN (cached in ${COMMIT_DROPS_FILE}.tmp) :"
    cat "${COMMIT_DROPS_FILE}".tmp | \
    egrep '^([\^]|.*31m\-)' | \
    ggrep -A1 '31m\-' | \
    grep '^\^' \
    | tee "${COMMIT_DROPS_FILE}"
}

fix_history() {
    ### When the inspections are done with, we want to clean up the history
    ### Note that as part of this, we would drop "unreachable" commits that
    ### are not part of any branch's history (because these would still contain
    ### the original offending pattern.
    ###
    ### WARNING: This is the one destructive operation in this suite.
    ###
    ### You are advised to run it in a scratch full copy of your git repo.
    ###
    ### After it is successfully done, you are advised to destroy the published
    ### copy of the repo completely (e.g. on github) and force-push the cleaned
    ### one into a newly created instance of the repo, and have your team members
    ### destroy and re-fork their clones both on cloud platform and their local
    ### workspaces, so that the destroyed offending commits do not resurface.

    ### Note that you can stack more sed '-e ...' blocks below, e.g. to rewrite
    ### more patterns in one shot. Also note that the pattern and replacement
    ### representation for simple grep and regex in sed may vary... you may want
    ### to automate escaping of special chars.
    [ -n "${PATTERN_SED-}" ] || PATTERN_SED="${PATTERN}"
    git filter-branch --tree-filter "git grep '${PATTERN}' | sed 's,:.*\$,,' | sort | uniq | while read F ; do sed -e 's,${PATTERN_SED},${REPLACEMENT},g' -i '\$F'; done"
    git reflog expire --expire-unreachable=now --all
    git gc --prune=now
}

[ -n "$2" ] && PATTERN="$2"
[ -n "$3" ] && REPLACEMENT="$3"

ACTION=""
case "$1" in
    -h|--help) usage; exit 0 ;;
    find|grep) ACTION="find_commits_intro"; shift ;;
    show|diff) ACTION="find_commits_show"; shift ;;
    drop|remove|del|delete) ACTION="find_commits_drop"; shift ;;
    fix_history|find_commits_contains|find_commits_intro|find_commits_drop) ACTION="$1"; shift ;;
    *) usage; exit 1 ;;
esac

echo "Running routine: '$ACTION'"
"$ACTION"

I'll also post a copy of this to my github collection of git scripts, to track further possible evolution.
 
A similar operation to extract just a history of certain file(s) into a new repository can be done through a series of mail-patches, e.g.:

oldrepo$ rm -rf /tmp/ggg ; mkdir -p /tmp/ggg

oldrepo$ FILES="some-script.sh"

### NOTE: Maybe factor $FILES (hash of?) into PRJ to allow different exports
### from same project to a new repo. Or just tag this manually :)

oldrepo$ PRJ="$(basename "`pwd`" | sed 's,[^A-Za-z\-_0-9\.],_,g')"

### The following results in a patch-file series ordered by UTC Unix epoch
###
timestamp of the original commit, allowing to mix similar export series
###
from different files or projects, and then importing as one monotonous
### history in a new repository. Note that git log tends to not end the last
### line well so we help it:

oldrepo$ ( git log --pretty='format:%H %ad' --date=unix $FILES ; echo "" ) | \
   ( A=0; while read HASH TIME ; do \
     [ -n "$HASH" ] && git format-patch -1 --no-numbered --stdout "$HASH"
$FILES \
     > /tmp/ggg/"$TIME-$PRJ-`printf '%04d' "$A"`".patch ; A=$(($A+1)); \
     done )

### Can filter the patch files further, e.g. update commit message formatting...

newrepo$ git am /tmp/ggg/*.patch

Friday, July 22, 2016

Setting up an OpenIndiana Hipster on OmniOS Bloody for distcc compilation... and vice-versa!

I participate in and try out a number of illumos-based community projects. Among these, I have a bleeding-edge OpenIndiana Hipster distribution on my laptop, and an OmniOS Bloody installation on a storage server at work. Recently I was playing with a recipe for distributed compilation with distcc, and wanted to try it out. Same operating environments are very much preferred, so the remote compilation head should also be (or seem) an OI Hipster. Luckily, we can fool IPS into installing whatever we want, as long as it is sane... (And note that part of the success story below may be due to both distros using the bleeding-edge code from illumos-gate, so underlying kernel and system calls expected by userland code are the same).

UPDATE below: the inverse setup, making an OmniOS zone hosted on OpenIndiana Hipster, seems also possible - though with a bit more workaround dance.

It all starts with a LAN connection, so here goes a bit of preparation: a dedicated VNIC where the buildhost will live:

root@omnios-host:/# dladm show-link
LINK        CLASS     MTU    STATE    BRIDGE     OVER
bge0        phys      1500   up       --         --
vboxnet0    phys      1500   up       --         --

root@backup-host:/root# dladm create-vnic -l bge0 vnic199

Note about zone nuances: OmniOS native (ipkg) zones are not-linked (do not require tight coupling of global-zone and local-zone software versions). This is inverted vs. OpenIndiana, where "ipkg" is linked like in latest OpenSolaris builds, and a new "nlipkg" is not-linked.

Create a zone:
root@omnios-host:/# zonecfg -z oibuild
zonecfg:oibuild> create -t SUNWipkg
zonecfg:oibuild> set zonepath=/zones/oibuild
zonecfg:oibuild> add net
zonecfg:oibuild:net> set physical=vnic199
zonecfg:oibuild:net> end
zonecfg:oibuild> set ip-type=exclusive
zonecfg:oibuild> set autoboot=true
zonecfg:oibuild> verify
zonecfg:oibuild> commit
zonecfg:oibuild> ^D

I might go on with delegated datasets, etc. - but this setup will NFS- or LOFS-mount whatever I need from the GZ later. So far I need the basics running.

And for these basics, I need the zone created without looking at incompatibilities. Namely, the "entire" incorporation does not concern me here, but the standard zone-branding script (/usr/lib/brand/ipkg/pkgcreatezone) blindly wants "entire" to be same as in GZ regardless of ipkg/nlipkg details (if there is an "entire" in GZ - otherwise it is happy without... so we extend it a bit):

root@omnios-host:/# cp -pf /usr/lib/brand/ipkg/pkgcreatezone /usr/lib/brand/ipkg/pkgcreatezone.orig

...and apply (or type in) this patch:
--- /usr/lib/brand/ipkg/pkgcreatezone.orig      2016-01-29 15:57:40.006965818 +0100
+++ /usr/lib/brand/ipkg/pkgcreatezone   2016-07-22 18:09:49.734017982 +0200
@@ -169,6 +169,7 @@
 # It's ok to not find entire in the current image, since this means the user
 # can install pre-release development bits for testing purposes.
 #
+[[ -n "$NO_ENTIRE_FMRI" ]] && entire_fmri="" || \
 entire_fmri=$(get_entire_incorp)
 #

Now, I can bind it to my will using environment variables (so it defaults to doing standard incantations otherwise) :)

Also note that the original standard script only allows one package publisher to be used during zone installation. It could be expanded to pre-set and use more, just that such change was not needed for this experiment and so is left out of the article's scope - and additional publishers are added after initial installation.

And it is simple to use:

root@omnios-host:/# zoneadm -z oibuild uninstall -F ; \
    NO_ENTIRE_FMRI=yes zoneadm -z oibuild install -v -P openindiana.org=http://pkg.openindiana.org/hipster/

A ZFS file system has been created for this zone.
       Image: Preparing at /zones/oibuild/root.
   Publisher: Using openindiana.org (http://pkg.openindiana.org/hipster/).
       Cache: Using /var/pkg/publisher.
  Installing: Packages (output follows)

Packages to install: 120
Mediators to change:   6
 Services to change:   4

DOWNLOAD                                PKGS         FILES    XFER (MB)   SPEED

Completed                            120/120   28192/28192  200.5/200.5  1.1M/s

PHASE                                          ITEMS

Installing new actions                   41903/41903
Updating package state database                 Done
Updating package cache                           0/0
Updating image state                            Done
Creating fast lookup database                   Done

        Note: Man pages can be obtained by installing pkg:/system/manual
 Postinstall: Copying SMF seed repository ... done.
        Done: Installation completed in 325.405 seconds.
  Next Steps: Boot the zone, then log into the zone console (zlogin -C)
              to complete the configuration process.

Now a bit more repos to add...

root@omnios-host:/# pkg -R /zones/oibuild/root set-publisher -g http://pkg.openindiana.org/legacy/ opensolaris.org

root@omnios-host:/# pkg -R /zones/oibuild/root set-publisher -g http://pkg.openindiana.org/hipster-encumbered/ hipster-encumbered

And a few packages I need here...
root@omnios-host:/# pkg -R /zones/oibuild/root install build-essential ccache rsync sudo mc

Ultimately the zone can be booted, basic networking set up, home directory attached, "sudo gmake component-environment-prep" executed to get some common build deps for the recipe I'm at - and an OI buildhost running under OmniOS is ready to roll. Why should that hardware stay dormant? (once I get that distcc package done well) :)

So the distcc server on oibuild zone has been compiled and got running (not without hiccups so far - but hey, that's what I'm tinkering on):

root@oibuild:/# distccd --user jim -j 30 --stats --stats-port 12345 --log-stderr --no-detach --verbose --daemon -a 0.0.0.0/0 &

And the client build from the laptop goes like:

jim@laptop$ gmake clean; echo =======; \
    CCACHE_RECACHE=true DISTCC_VERBOSE=1 \
    DISTCC_HOSTS="10.1.2.3,lzo,cpp" CCACHE_PREFIX="distcc" \
    COMPONENT_BUILD_GMAKE_ARGS=-j20 pump gmake publish

Whoosh!

UPDATE: A similar setup, as far as mirror twins go, is also possible. Slightly more changes are needed to the "pkgcreatezone" script in OI Hipster host, because originally it requires the new "sysding" package (successor to old "sysidcfg" scripts) that is not provided by other distros.

root@hipster-host:/# diff -bu /usr/lib/brand/ipkg/pkgcreatezone{.orig,}
--- /usr/lib/brand/ipkg/pkgcreatezone.orig 2016-08-25 11:20:57.249926456 +0200
+++ /usr/lib/brand/ipkg/pkgcreatezone 2016-09-23 10:24:54.690076605 +0200
@@ -169,6 +169,7 @@
 # It's ok to not find entire in the current image, since this means the user
 # can install pre-release development bits for testing purposes.
 #
+[[ -n "$NO_ENTIRE_FMRI" ]] && entire_fmri="" || \
 entire_fmri=$(get_entire_incorp)
 
 #
@@ -263,7 +264,6 @@
 pkglist="$pkglist
  pkg:/SUNWcs
  pkg:/SUNWcsd
- pkg:/service/management/sysding
  pkg:/system/network
  pkg:/system/extended-system-utilities
  pkg:/compress/bzip2
@@ -272,6 +272,10 @@
  pkg:/compress/unzip
  pkg:/package/pkg"
 
+[[ -n "$NO_SYSDING" ]] || \
+pkglist="$pkglist
+ pkg:/service/management/sysding"
+
 #
 # Get some diagnostic tools, truss, dtrace, etc.
 #

Then you set up the VNIC and zone configuration, similar to the example above (note the OI "nlipkg" brand has the "OI", not "SUNW", prefix):

root@hipster-host:/# dladm create-vnic -l e1000g1 omnibld0

root@hipster-host:/# zonecfg -z omnibld
zonecfg:omnibld> create -t OInlipkg
zonecfg:omnibld> set zonepath=/zones/omnibld
zonecfg:omnibld> add net
zonecfg:omnibld:net> set physical=omnibld0
zonecfg:omnibld:net> end
zonecfg:omnibld> set ip-type=exclusive
zonecfg:omnibld> set autoboot=true
zonecfg:omnibld> verify
zonecfg:omnibld> commit
zonecfg:omnibld> ^D

And finally install the zone (I prepend destruction of old attempts... because... well... experiments are like that :) ):

root@hipster-host:/# zoneadm -z omnibld uninstall -F ; \
   zfs destroy -r rpool/zones/omnibld ; \
   NO_SYSDING=yes  NO_ENTIRE_FMRI=yes zoneadm -z omnibld install -v \
     -P omnios=http://pkg.omniti.com/omnios/LX/

cannot open 'rpool/zones/omnibld': dataset does not exist
A ZFS file system has been created for this zone.
       Image: Preparing at /zones/omnibld/root.

   Publisher: Using omnios (http://pkg.omniti.com/omnios/LX/).
       Cache: Using /var/pkg/publisher.
  Installing: Packages (output follows)
Packages to install: 90
Mediators to change:  1
 Services to change:  4

DOWNLOAD                                PKGS         FILES    XFER (MB)   SPEED
Completed                              90/90   24570/24570  161.7/161.7  406k/s

PHASE                                          ITEMS
Installing new actions                   37912/37912
Updating package state database                 Done 
Updating package cache                           0/0 
Updating image state                            Done 
Creating fast lookup database                   Done 

        Note: Man pages can be obtained by installing pkg:/system/manual
 Postinstall: Copying SMF seed repository ... done.
        Done: Installation completed in 526.523 seconds.

  Next Steps: Boot the zone, then log into the zone console (zlogin -C)
              to complete the configuration process.

root@hipster-host:/# pkg -R /zones/omnibld/root set-publisher -g \
   http://pkg.omniti.com/omniti-ms/ ms.omniti.com

root@hipster-host:/# pkg -R /zones/omnibld/root set-publisher -g \
   http://pkg.omniti.com/omniti-perl/ perl.omniti.com

And then, probably, to get something useful in that zone, I'd build it or proceed to install "pkgsrc" framework from http://www.perkin.org.uk/pages/pkgsrc-binary-packages-for-illumos.html - but that's another story.

Finally note, that for pedantic production use you'd create a new zone brand referring the customized "pkgcreatezone" script (leaving the default packaged one untouched) and the standard copies of other files, similar to how the "OInlipkg" definition does. As a lesser evil, you only need the tweaks when you create a new zone like this - so you can revert to the saved "pkgcreatezone.orig" copy after installations are completed, if you're concerned about this.

Sunday, January 10, 2016

Robocopy'ing a windows system partition...

OVERVIEW

I'm having some fun time migrating a laptop away from a dying HDD (as confirmed by SMART checkups, e.g. using a nice Windows GUI wrapper for smartmontools called GSmartControl) to a new one, connected temporarily into an USB carrier.

I've hit and cleared a number of roadblocks, so decided to sum up for posterity what I did here ;)
  • Windows Activation is evil ;) If you clone to a new harddisk, try to minimize the system changes (e.g. partition numbering, BIOS versions or hardware bits like memory chips should better wait until you get the cloned Windows to boot without losing its Genuine status, if you're lucky and it at all happens).
  • If NTFS compression is enabled, and especially if you use non-Windows NTFS implementation for file-based copying, make sure to NOT compress the copy of "C:\bootmgr" and "C:\bootwin" files, as well as (most likely) the "C:\Boot", "C:\Recovery", "C:\System Restore", "C:\Windows\System32", "C:\Windows\winsxs" and "C:\Windows\SysWOW64" (whichever of these are available), since the boot manager and low-level drivers can be picky about being readable and usable "as is" from on-disk bits. It may be possible to apply compression after the clone boots up and locks the sensitive files.
  • CygWin-based rsync is, alas, not suitable for the task
  • XCOPY too, at least not to just refresh the existing replica
  • NTFS-3g also did not cut it for me, at least not from OpenIndiana
  • Using VSS (Shadow Volumes) sounded like a good idea, e.g. to copy the registry files and other always-open objects, but it is a bit of an adventure to actually create the volume in Windows 7 (and other non-server versions, I gather)
  • Native Microsoft RoboCopy has an issue copying ACLs, and a Microsoft hotfix 979808 was not applicable to the version installed on the laptop, so files (like the "C:\Boot" directory) have to become owned by the current user - or a hack can be used to copy as "TrustedInstaller" (who owns most of such offending objects)
  • DO HAVE the installation or at least recovery disk or ISO image and a means to boot it (CDROM drive, Dual-bootable hypervisor installed in a neighbor partition, etc.) for the cases when things do go wrong and you need to e.g. rewrite mount paths (registry HKLM\System\MountedDevices), reconfigure boot configuration (bcdedit, bootrec), decompress some driver files, etc.
    It also seems that "robocopy" used from the context of the Recovery session does not suffer the permission issues, and of course there is no issue with opened and thus inaccessible system files. And if you fiddle with mountpoints or drive letters, you can also avoid complications due to copying of directory junctions (links). In fact, I'd now suggest to start with this option if possible.

FORMATTING THE DISKS


I just used the GUI under Computer Management MMC to partition the new disk for the OSes it will have, and to "activate" the partition for Windows. This enabled bootability of the new disk from the NTFS. (Note that ultimately another partition with GRUB would chainload the Windows one, if needed). Further fine-tuning is possible with GNU "parted" (many OSes) or a Linux "fdisk".

It seems that this laptop's new WD SSHD drive has 512-byte sectors on the harddisk part, but announces larger sectors due to the Flash layer. After the initial copy refused to boot, and a "dd" byte-by-byte copy as well, we reformatted the disk guessing a 4KB NTFS block size and it seemed to do the trick.

Native Windows tools including "diskpart", "bcdedit", "bootrec" and "bootsect" also had their moment of glory during my experiments (getting that copy to boot up is an adventure of its own, possibly because the new SSHD disk has weird announcements about its sector sizes).

TRACKING THE FAILURES


RSYNC


Initially I used a CygWin build of rsync (from the cwRsync project, maybe defunct now) to migrate the bulk of data with the usual "rsync -avPHK ..." mantra. Maybe I shouldn't have done so...

While cwrsync serves me well for networked media backups (photos etc.) between NTFS and Unix machines, including sym/hardlink support vs. NTFS directory- and file-"junctions" (which in turn are easily manipulated with the free FAR Manager), it happened to be a bad idea to use it within the Windows system. After the rsync run, all "junctions" were replicated as text files with "/cygdrive/c/..." contents which had to be cleaned up manually (again using FAR: one instance to try and copy over the changes, skipping all conflicts, and another FAR to manually redo the junctions on which the first one hicced-up -- mostly these are standardized structural links under C:\Users though a few others were of my own making).

According to docs and forums, the non-POSIX NTFS features like alternate data streams are not seen by CygWin and so cwrsync (or other similar builds). Also I'm not convinced that ownerships and ACLs were properly transferred. And, rather predictably, "rsync" failed to copy files opened by the system, such as the registry and some log/db files.

XCOPY


I tried to reprocess the existing files with some other tools, rather than copy stuff over again (dying disk... quite a few retries though nearly no actual IO errors yet... took a couple of days to get here).

Alas, Microsoft's XCOPY seems to copy the same files over - which kinda defeated my purpose. Just for kicks, this carefully crafted command-line did not help me:
xcopy /O /X /E /H /K /B /Y /R /C /L "C:\Boot" "W:\Boot"
It might still be a good choice for the initial (and only) copy, however.

NTFS-3g


My next idea was to reboot into another OS, mount the two Windows partitions, and rsync the data there. But with directory symlinks becoming full paths (e.g. "/mnt/win/Users" for the "./Documents and Settings") with whatever options I used to mount the NTFS volumes, the idea failed.

Also, again, I'm not certain that ACLs and owner/groups are properly replicated in this manner - which ordinarily matters little for me when I manipulate user-data, but can matter for the OS when its guts are being migrated.

SHADOWS, SNAPSHOTS and SYSTEM RESTORE


Windows 7 has a reduced feature set of the "vssadmin" command: it can manipulate the Shadow area and list what is available, but it can not take snapshots. According to the Internet lore, there is another program from the Windows SDK which fills the gap - but also there is the System Restore GUI which uses (and creates) the snapshots under the hood. IF it is enabled at all. Oh boy...

  • To enable Shadowing on the original drive, set aside some space (min 300mb):
vssadmin resize shadowstorage /for=c: /on=c: /maxsize=320mb
  • On a more capable (Windows Server) system you would just use CLI to create the snapshot, e.g.:
vssadmin create shadow /For=C:
  • To create a snapshot on a desktop Windows 7, go to System Properties (Win+P) and into "System protection". Verify that it is "on" for your original drive ("C:") and if you can, press "Create..." to make a System Restore point, which is a VSS snapshot. Note that in my experiment, creating another point removed the first one. Maybe there was too little space set aside, or the feature is constrained in desktop Windows versions.
  • To enable usage of System Restore (if it is initially "Disabled by your system administrator"), go to Registry (thanks to several blogs, like this one: http://forum.thewindowsclub.com/windows-tips-tutorials-articles/34531-re-enable-disabled-system-restore-system-administrator.html):
    • Start (Win+R) the regedit program
    • Browse into HKLM\Software\Policies\Microsoft\Windows NT\SystemRestore
    • Delete the values "DisableConfig" and "DisableSR"
    • No reboot should be needed, just re-open the System protection tab in the step above
  • Look up the snapshot name with:
    vssadmin List Shadows
    ...
    Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2
    ...
  • Mount that snapshot with "mklink /d" so it becomes a read-only directory (until the snapshot gets killed by the system):
    mklink /d c:\snap \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2\
  • Note that for some specific purposes you can link to a sub-object, e.g.:
    mklink /d c:\snapwork \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2\Users\myname\Documents\Work\
  • To unmount a snapshot later, it would suffice to just remove the FS object of the link (note that destruction of the snapshot is a different matter; it usually happens "by itself" at behest of the OS as it clears out some living space for itself or new snapshots):
    rmdir c:\snap

ROBOCOPY


My final spell for replication of ownership, ACLs and changes that piled up on the original system since the beginning of replication, was to use MS RoboCopy, done like this - manually for each directory I was interested in (not all of them existing ones). There is an issue with replication of permissions for FS objects that you (or your Administrators group) are not an owner of. The hotfix did not work for me, but the workaround of grabbing the ownership (on target filesystem - this sufficed) did work for some files. However for "C:\Windows" there are way more concerns about such fiddling... so Running cmd.exe as TrustedInstaller also sounds like a good idea for at least most of those files. And also a boosted robocopy logging to track which files yielded "Error 5" when trying to set NTFS permissions. Another problem is that the Shadows used in System Restore points are not quite versatile snapshots, and they seem to omit certain files so that the OS rollback would not change user documents - so copying bits only from the snapshot is not enough.

First I copied from the mounted VSS snapshot under "C:\snap" into the mounted new partition under "W:", then overlaid it with files from the live filesystem (if different), e.g.:
set DD=Program Files
robocopy "C:\snap\%DD%" "W:\%DD%" /COPYALL /DCOPY:T /SECFIX /TIMFIX /SL /FFT /DST /R:1 /W:1 /E /MIR /XJD /LOG+:w:\robocopy.log
robocopy "C:\%DD%" "W:\%DD%" /COPYALL /DCOPY:T /SECFIX /TIMFIX /SL /FFT /DST /R:1 /W:1 /E /XJD /LOG+:w:\robocopy.log

Names with spaces can also be used, as seen above - without quotation marks.

If you are in the Recovery Console and still see any errors due to files (likely bits of the Registry) being locked by some process, try to "Force Dismount" of the original and new volumes using "chkdsk /x" or "diskpart"/"select volume X"/"assign letter=Y".

For objects with potentially conflicting "8.3" names ("dir /x" is your friend to discover these), like "PROGRA~1" for anything starting with "Progra...", be sure to copy objects in the same order as they are numbered (or "move" into a temporary subdirectory and back) - for me, some Shortcuts and registry paths happened to be saved with the short space-less names in mind, so this nuance did matter for bootability of the copied system.

Note that the "/MIR" flag (mirroring) enables "/PURGE /E", the former of which removes files on destination that the origin no longer has (named "Extra" files in the finally printed stats). This might be or not be desirable depending on the directory you're passing over.

The "/XJ" (or at least "/XJD") flag is critical to avoid infinite loops while copying the "C:\Users" area. With this flag, robocopy does not recurse into (directory) junctions. Alas, there is no flag to copy over definitions of the junctions - they are ignored completely. I filled this niche with FAR, as described earlier.

Alternately, you can find junctions with a bit of CMD and some patience, e.g.:
set DD=ProgramData
dir "C:\%DD%" /q /r /a:h 2>NUL | find "JUNCTION"
or just search for links:
dir "C:\%DD%" /q /r /a:l 2>NUL

You can re-create junctions using "mklink (/d)".

RECOVERY SESSION FROM INSTALL DISC


Ok, maybe I should have started with this - but it was a bit complicated on a laptop with no CD drive ;)

It may help avoid surprises to use "diskpart" and mark the source disk (or volume) as read-only, to be sure that recursions into it, if they ever happen, do not damage original files. Also do not use drive-letters that these volumes normally hold when booted as part of an installed OS. For example, to mark the disks visibly as "O"ld and "N"ew, assign drive-letters "O:" and "N:" respectively:
x:> diskpart
list volume
(find which one is your source)
select volume 3
assign letter=o
attributes volume set readonly
list volume
(find which one is your target)
select volume 1
assign letter=n

To verify, you can try creating a file in temporary location of the original partition:
x:> echo test > o:\temp\test.txt
The media is write-protected.
Now (hopefully) you are safe to (re-)run robocopy so it recreates the links as needed.

Anyhow, in this context Robocopy rocks quite well - e.g. copies the ownerships and ACLs with no problems. If I mount the source and destination drives so that none of them uses the drive letters normally occupied in the installed OS, the directory junctions become non-recursive and so do not cause problems (the /XJ /XJD /XJF flags become not-needed), and SYMLINKD entries even get copied over.

However, copying JUNCTION entries is still a problem. These would better be remade with "dir" to discover and "mklink /d" to create (now, this *may* require that a volume is mounted into original drive-letter, so links resolve well at time of creation... or maybe not...). Robocopy just recurses and creates paths of real directories too long to easily remove (I had to rename "Application Data" into "x" dozens of times, before I cold kill the mis-made tree.

FINAL NOTES


The directory junctions seemingly name the full path, e.g. linking to disk "C:", whichever way it is defined in the currently booted OS. Likewise, the paths are entered in very many locations if the registry and other configuration files.

So make sure to mount the disks correctly, so the drive letters of old and new logical volumes resolve the same when you boot from them... I have yet to boot from this one, and maybe go back into Safe mode to redefine which partitions are mounted to which drive-letters, when the new disk gets finally bolted into the laptop... I keep my fingers crossed now ;)