I participate in and try out a number of illumos-based community projects. Among these, I have a bleeding-edge OpenIndiana Hipster distribution on my laptop, and an OmniOS Bloody installation on a storage server at work. Recently I was playing with a recipe for distributed compilation with distcc, and wanted to try it out. Same operating environments are very much preferred, so the remote compilation head should also be (or seem) an OI Hipster. Luckily, we can fool IPS into installing whatever we want, as long as it is sane... (And note that part of the success story below may be due to both distros using the bleeding-edge code from illumos-gate, so underlying kernel and system calls expected by userland code are the same).
UPDATE below: the inverse setup, making an OmniOS zone hosted on OpenIndiana Hipster, seems also possible - though with a bit more workaround dance.
It all starts with a LAN connection, so here goes a bit of preparation: a dedicated VNIC where the buildhost will live:
Note about zone nuances: OmniOS native (ipkg) zones are not-linked (do not require tight coupling of global-zone and local-zone software versions). This is inverted vs. OpenIndiana, where "ipkg" is linked like in latest OpenSolaris builds, and a new "nlipkg" is not-linked.
Create a zone:
I might go on with delegated datasets, etc. - but this setup will NFS- or LOFS-mount whatever I need from the GZ later. So far I need the basics running.
And for these basics, I need the zone created without looking at incompatibilities. Namely, the "entire" incorporation does not concern me here, but the standard zone-branding script (/usr/lib/brand/ipkg/pkgcreatezone) blindly wants "entire" to be same as in GZ regardless of ipkg/nlipkg details (if there is an "entire" in GZ - otherwise it is happy without... so we extend it a bit):
Now, I can bind it to my will using environment variables (so it defaults to doing standard incantations otherwise) :)
Also note that the original standard script only allows one package publisher to be used during zone installation. It could be expanded to pre-set and use more, just that such change was not needed for this experiment and so is left out of the article's scope - and additional publishers are added after initial installation.
And it is simple to use:
Now a bit more repos to add...
Ultimately the zone can be booted, basic networking set up, home directory attached, "sudo gmake component-environment-prep" executed to get some common build deps for the recipe I'm at - and an OI buildhost running under OmniOS is ready to roll. Why should that hardware stay dormant? (once I get that distcc package done well) :)
So the distcc server on oibuild zone has been compiled and got running (not without hiccups so far - but hey, that's what I'm tinkering on):
Whoosh!
UPDATE: A similar setup, as far as mirror twins go, is also possible. Slightly more changes are needed to the "pkgcreatezone" script in OI Hipster host, because originally it requires the new "sysding" package (successor to old "sysidcfg" scripts) that is not provided by other distros.
Then you set up the VNIC and zone configuration, similar to the example above (note the OI "nlipkg" brand has the "OI", not "SUNW", prefix):
And finally install the zone (I prepend destruction of old attempts... because... well... experiments are like that :) ):
And then, probably, to get something useful in that zone, I'd build it or proceed to install "pkgsrc" framework from http://www.perkin.org.uk/pages/pkgsrc-binary-packages-for-illumos.html - but that's another story.
Finally note, that for pedantic production use you'd create a new zone brand referring the customized "pkgcreatezone" script (leaving the default packaged one untouched) and the standard copies of other files, similar to how the "OInlipkg" definition does. As a lesser evil, you only need the tweaks when you create a new zone like this - so you can revert to the saved "pkgcreatezone.orig" copy after installations are completed, if you're concerned about this.
UPDATE below: the inverse setup, making an OmniOS zone hosted on OpenIndiana Hipster, seems also possible - though with a bit more workaround dance.
It all starts with a LAN connection, so here goes a bit of preparation: a dedicated VNIC where the buildhost will live:
root@omnios-host:/# dladm show-link LINK CLASS MTU STATE BRIDGE OVER bge0 phys 1500 up -- -- vboxnet0 phys 1500 up -- -- root@backup-host:/root# dladm create-vnic -l bge0 vnic199
Note about zone nuances: OmniOS native (ipkg) zones are not-linked (do not require tight coupling of global-zone and local-zone software versions). This is inverted vs. OpenIndiana, where "ipkg" is linked like in latest OpenSolaris builds, and a new "nlipkg" is not-linked.
Create a zone:
root@omnios-host:/# zonecfg -z oibuild zonecfg:oibuild> create -t SUNWipkg zonecfg:oibuild> set zonepath=/zones/oibuild zonecfg:oibuild> add net zonecfg:oibuild:net> set physical=vnic199 zonecfg:oibuild:net> end zonecfg:oibuild> set ip-type=exclusive zonecfg:oibuild> set autoboot=true zonecfg:oibuild> verify zonecfg:oibuild> commit zonecfg:oibuild> ^D
I might go on with delegated datasets, etc. - but this setup will NFS- or LOFS-mount whatever I need from the GZ later. So far I need the basics running.
And for these basics, I need the zone created without looking at incompatibilities. Namely, the "entire" incorporation does not concern me here, but the standard zone-branding script (/usr/lib/brand/ipkg/pkgcreatezone) blindly wants "entire" to be same as in GZ regardless of ipkg/nlipkg details (if there is an "entire" in GZ - otherwise it is happy without... so we extend it a bit):
root@omnios-host:/# cp -pf /usr/lib/brand/ipkg/pkgcreatezone /usr/lib/brand/ipkg/pkgcreatezone.orig...and apply (or type in) this patch:
--- /usr/lib/brand/ipkg/pkgcreatezone.orig 2016-01-29 15:57:40.006965818 +0100 +++ /usr/lib/brand/ipkg/pkgcreatezone 2016-07-22 18:09:49.734017982 +0200 @@ -169,6 +169,7 @@ # It's ok to not find entire in the current image, since this means the user # can install pre-release development bits for testing purposes. # +[[ -n "$NO_ENTIRE_FMRI" ]] && entire_fmri="" || \ entire_fmri=$(get_entire_incorp) #
Now, I can bind it to my will using environment variables (so it defaults to doing standard incantations otherwise) :)
Also note that the original standard script only allows one package publisher to be used during zone installation. It could be expanded to pre-set and use more, just that such change was not needed for this experiment and so is left out of the article's scope - and additional publishers are added after initial installation.
And it is simple to use:
root@omnios-host:/# zoneadm -z oibuild uninstall -F ; \
NO_ENTIRE_FMRI=yes zoneadm -z oibuild install -v -P openindiana.org=http://pkg.openindiana.org/hipster/
A ZFS file system has been created for this zone.
Image: Preparing at /zones/oibuild/root.
Publisher: Using openindiana.org (http://pkg.openindiana.org/hipster/).
Cache: Using /var/pkg/publisher.
Installing: Packages (output follows)
Packages to install: 120
Mediators to change: 6
Services to change: 4
DOWNLOAD PKGS FILES XFER (MB) SPEED
Completed 120/120 28192/28192 200.5/200.5 1.1M/s
PHASE ITEMS
Installing new actions 41903/41903
Updating package state database Done
Updating package cache 0/0
Updating image state Done
Creating fast lookup database Done
Note: Man pages can be obtained by installing pkg:/system/manual
Postinstall: Copying SMF seed repository ... done.
Done: Installation completed in 325.405 seconds.
Next Steps: Boot the zone, then log into the zone console (zlogin -C)
to complete the configuration process.
Now a bit more repos to add...
root@omnios-host:/# pkg -R /zones/oibuild/root set-publisher -g http://pkg.openindiana.org/legacy/ opensolaris.org root@omnios-host:/# pkg -R /zones/oibuild/root set-publisher -g http://pkg.openindiana.org/hipster-encumbered/ hipster-encumberedAnd a few packages I need here...
root@omnios-host:/# pkg -R /zones/oibuild/root install build-essential ccache rsync sudo mc
Ultimately the zone can be booted, basic networking set up, home directory attached, "sudo gmake component-environment-prep" executed to get some common build deps for the recipe I'm at - and an OI buildhost running under OmniOS is ready to roll. Why should that hardware stay dormant? (once I get that distcc package done well) :)
So the distcc server on oibuild zone has been compiled and got running (not without hiccups so far - but hey, that's what I'm tinkering on):
root@oibuild:/# distccd --user jim -j 30 --stats --stats-port 12345 --log-stderr --no-detach --verbose --daemon -a 0.0.0.0/0 &And the client build from the laptop goes like:
jim@laptop$ gmake clean; echo =======; \
CCACHE_RECACHE=true DISTCC_VERBOSE=1 \
DISTCC_HOSTS="10.1.2.3,lzo,cpp" CCACHE_PREFIX="distcc" \
COMPONENT_BUILD_GMAKE_ARGS=-j20 pump gmake publish
Whoosh!
UPDATE: A similar setup, as far as mirror twins go, is also possible. Slightly more changes are needed to the "pkgcreatezone" script in OI Hipster host, because originally it requires the new "sysding" package (successor to old "sysidcfg" scripts) that is not provided by other distros.
root@hipster-host:/# diff -bu /usr/lib/brand/ipkg/pkgcreatezone{.orig,}
--- /usr/lib/brand/ipkg/pkgcreatezone.orig 2016-08-25 11:20:57.249926456 +0200
+++ /usr/lib/brand/ipkg/pkgcreatezone 2016-09-23 10:24:54.690076605 +0200
@@ -169,6 +169,7 @@
# It's ok to not find entire in the current image, since this means the user
# can install pre-release development bits for testing purposes.
#
+[[ -n "$NO_ENTIRE_FMRI" ]] && entire_fmri="" || \
entire_fmri=$(get_entire_incorp)
#
@@ -263,7 +264,6 @@
pkglist="$pkglist
pkg:/SUNWcs
pkg:/SUNWcsd
- pkg:/service/management/sysding
pkg:/system/network
pkg:/system/extended-system-utilities
pkg:/compress/bzip2
@@ -272,6 +272,10 @@
pkg:/compress/unzip
pkg:/package/pkg"
+[[ -n "$NO_SYSDING" ]] || \
+pkglist="$pkglist
+ pkg:/service/management/sysding"
+
#
# Get some diagnostic tools, truss, dtrace, etc.
#
Then you set up the VNIC and zone configuration, similar to the example above (note the OI "nlipkg" brand has the "OI", not "SUNW", prefix):
root@hipster-host:/# dladm create-vnic -l e1000g1 omnibld0 root@hipster-host:/# zonecfg -z omnibld zonecfg:omnibld> create -t OInlipkg zonecfg:omnibld> set zonepath=/zones/omnibld zonecfg:omnibld> add net zonecfg:omnibld:net> set physical=omnibld0 zonecfg:omnibld:net> end zonecfg:omnibld> set ip-type=exclusive zonecfg:omnibld> set autoboot=true zonecfg:omnibld> verify zonecfg:omnibld> commit zonecfg:omnibld> ^D
And finally install the zone (I prepend destruction of old attempts... because... well... experiments are like that :) ):
root@hipster-host:/# zoneadm -z omnibld uninstall -F ; \
zfs destroy -r rpool/zones/omnibld ; \
NO_SYSDING=yes NO_ENTIRE_FMRI=yes zoneadm -z omnibld install -v \
-P omnios=http://pkg.omniti.com/omnios/LX/
cannot open 'rpool/zones/omnibld': dataset does not exist
A ZFS file system has been created for this zone.
Image: Preparing at /zones/omnibld/root.
Publisher: Using omnios (http://pkg.omniti.com/omnios/LX/).
Cache: Using /var/pkg/publisher.
Installing: Packages (output follows)
Packages to install: 90
Mediators to change: 1
Services to change: 4
DOWNLOAD PKGS FILES XFER (MB) SPEED
Completed 90/90 24570/24570 161.7/161.7 406k/s
PHASE ITEMS
Installing new actions 37912/37912
Updating package state database Done
Updating package cache 0/0
Updating image state Done
Creating fast lookup database Done
Note: Man pages can be obtained by installing pkg:/system/manual
Postinstall: Copying SMF seed repository ... done.
Done: Installation completed in 526.523 seconds.
Next Steps: Boot the zone, then log into the zone console (zlogin -C)
to complete the configuration process.
root@hipster-host:/# pkg -R /zones/omnibld/root set-publisher -g \
http://pkg.omniti.com/omniti-ms/ ms.omniti.com
root@hipster-host:/# pkg -R /zones/omnibld/root set-publisher -g \
http://pkg.omniti.com/omniti-perl/ perl.omniti.com
And then, probably, to get something useful in that zone, I'd build it or proceed to install "pkgsrc" framework from http://www.perkin.org.uk/pages/pkgsrc-binary-packages-for-illumos.html - but that's another story.
Finally note, that for pedantic production use you'd create a new zone brand referring the customized "pkgcreatezone" script (leaving the default packaged one untouched) and the standard copies of other files, similar to how the "OInlipkg" definition does. As a lesser evil, you only need the tweaks when you create a new zone like this - so you can revert to the saved "pkgcreatezone.orig" copy after installations are completed, if you're concerned about this.
