Friday, July 22, 2016

Setting up an OpenIndiana Hipster on OmniOS Bloody for distcc compilation... and vice-versa!

I participate in and try out a number of illumos-based community projects. Among these, I have a bleeding-edge OpenIndiana Hipster distribution on my laptop, and an OmniOS Bloody installation on a storage server at work. Recently I was playing with a recipe for distributed compilation with distcc, and wanted to try it out. Same operating environments are very much preferred, so the remote compilation head should also be (or seem) an OI Hipster. Luckily, we can fool IPS into installing whatever we want, as long as it is sane... (And note that part of the success story below may be due to both distros using the bleeding-edge code from illumos-gate, so underlying kernel and system calls expected by userland code are the same).

UPDATE below: the inverse setup, making an OmniOS zone hosted on OpenIndiana Hipster, seems also possible - though with a bit more workaround dance.

It all starts with a LAN connection, so here goes a bit of preparation: a dedicated VNIC where the buildhost will live:

root@omnios-host:/# dladm show-link
bge0        phys      1500   up       --         --
vboxnet0    phys      1500   up       --         --

root@backup-host:/root# dladm create-vnic -l bge0 vnic199

Note about zone nuances: OmniOS native (ipkg) zones are not-linked (do not require tight coupling of global-zone and local-zone software versions). This is inverted vs. OpenIndiana, where "ipkg" is linked like in latest OpenSolaris builds, and a new "nlipkg" is not-linked.

Create a zone:
root@omnios-host:/# zonecfg -z oibuild
zonecfg:oibuild> create -t SUNWipkg
zonecfg:oibuild> set zonepath=/zones/oibuild
zonecfg:oibuild> add net
zonecfg:oibuild:net> set physical=vnic199
zonecfg:oibuild:net> end
zonecfg:oibuild> set ip-type=exclusive
zonecfg:oibuild> set autoboot=true
zonecfg:oibuild> verify
zonecfg:oibuild> commit
zonecfg:oibuild> ^D

I might go on with delegated datasets, etc. - but this setup will NFS- or LOFS-mount whatever I need from the GZ later. So far I need the basics running.

And for these basics, I need the zone created without looking at incompatibilities. Namely, the "entire" incorporation does not concern me here, but the standard zone-branding script (/usr/lib/brand/ipkg/pkgcreatezone) blindly wants "entire" to be same as in GZ regardless of ipkg/nlipkg details (if there is an "entire" in GZ - otherwise it is happy without... so we extend it a bit):

root@omnios-host:/# cp -pf /usr/lib/brand/ipkg/pkgcreatezone /usr/lib/brand/ipkg/pkgcreatezone.orig

...and apply (or type in) this patch:
--- /usr/lib/brand/ipkg/pkgcreatezone.orig      2016-01-29 15:57:40.006965818 +0100
+++ /usr/lib/brand/ipkg/pkgcreatezone   2016-07-22 18:09:49.734017982 +0200
@@ -169,6 +169,7 @@
 # It's ok to not find entire in the current image, since this means the user
 # can install pre-release development bits for testing purposes.
+[[ -n "$NO_ENTIRE_FMRI" ]] && entire_fmri="" || \

Now, I can bind it to my will using environment variables (so it defaults to doing standard incantations otherwise) :)

Also note that the original standard script only allows one package publisher to be used during zone installation. It could be expanded to pre-set and use more, just that such change was not needed for this experiment and so is left out of the article's scope - and additional publishers are added after initial installation.

And it is simple to use:

root@omnios-host:/# zoneadm -z oibuild uninstall -F ; \
    NO_ENTIRE_FMRI=yes zoneadm -z oibuild install -v -P

A ZFS file system has been created for this zone.
       Image: Preparing at /zones/oibuild/root.
   Publisher: Using (
       Cache: Using /var/pkg/publisher.
  Installing: Packages (output follows)

Packages to install: 120
Mediators to change:   6
 Services to change:   4

DOWNLOAD                                PKGS         FILES    XFER (MB)   SPEED

Completed                            120/120   28192/28192  200.5/200.5  1.1M/s

PHASE                                          ITEMS

Installing new actions                   41903/41903
Updating package state database                 Done
Updating package cache                           0/0
Updating image state                            Done
Creating fast lookup database                   Done

        Note: Man pages can be obtained by installing pkg:/system/manual
 Postinstall: Copying SMF seed repository ... done.
        Done: Installation completed in 325.405 seconds.
  Next Steps: Boot the zone, then log into the zone console (zlogin -C)
              to complete the configuration process.

Now a bit more repos to add...

root@omnios-host:/# pkg -R /zones/oibuild/root set-publisher -g

root@omnios-host:/# pkg -R /zones/oibuild/root set-publisher -g hipster-encumbered

And a few packages I need here...
root@omnios-host:/# pkg -R /zones/oibuild/root install build-essential ccache rsync sudo mc

Ultimately the zone can be booted, basic networking set up, home directory attached, "sudo gmake component-environment-prep" executed to get some common build deps for the recipe I'm at - and an OI buildhost running under OmniOS is ready to roll. Why should that hardware stay dormant? (once I get that distcc package done well) :)

So the distcc server on oibuild zone has been compiled and got running (not without hiccups so far - but hey, that's what I'm tinkering on):

root@oibuild:/# distccd --user jim -j 30 --stats --stats-port 12345 --log-stderr --no-detach --verbose --daemon -a &

And the client build from the laptop goes like:

jim@laptop$ gmake clean; echo =======; \
    DISTCC_HOSTS=",lzo,cpp" CCACHE_PREFIX="distcc" \
    COMPONENT_BUILD_GMAKE_ARGS=-j20 pump gmake publish


UPDATE: A similar setup, as far as mirror twins go, is also possible. Slightly more changes are needed to the "pkgcreatezone" script in OI Hipster host, because originally it requires the new "sysding" package (successor to old "sysidcfg" scripts) that is not provided by other distros.

root@hipster-host:/# diff -bu /usr/lib/brand/ipkg/pkgcreatezone{.orig,}
--- /usr/lib/brand/ipkg/pkgcreatezone.orig 2016-08-25 11:20:57.249926456 +0200
+++ /usr/lib/brand/ipkg/pkgcreatezone 2016-09-23 10:24:54.690076605 +0200
@@ -169,6 +169,7 @@
 # It's ok to not find entire in the current image, since this means the user
 # can install pre-release development bits for testing purposes.
+[[ -n "$NO_ENTIRE_FMRI" ]] && entire_fmri="" || \
@@ -263,7 +264,6 @@
- pkg:/service/management/sysding
@@ -272,6 +272,10 @@
+[[ -n "$NO_SYSDING" ]] || \
+ pkg:/service/management/sysding"
 # Get some diagnostic tools, truss, dtrace, etc.

Then you set up the VNIC and zone configuration, similar to the example above (note the OI "nlipkg" brand has the "OI", not "SUNW", prefix):

root@hipster-host:/# dladm create-vnic -l e1000g1 omnibld0

root@hipster-host:/# zonecfg -z omnibld
zonecfg:omnibld> create -t OInlipkg
zonecfg:omnibld> set zonepath=/zones/omnibld
zonecfg:omnibld> add net
zonecfg:omnibld:net> set physical=omnibld0
zonecfg:omnibld:net> end
zonecfg:omnibld> set ip-type=exclusive
zonecfg:omnibld> set autoboot=true
zonecfg:omnibld> verify
zonecfg:omnibld> commit
zonecfg:omnibld> ^D

And finally install the zone (I prepend destruction of old attempts... because... well... experiments are like that :) ):

root@hipster-host:/# zoneadm -z omnibld uninstall -F ; \
   zfs destroy -r rpool/zones/omnibld ; \
   NO_SYSDING=yes  NO_ENTIRE_FMRI=yes zoneadm -z omnibld install -v \
     -P omnios=

cannot open 'rpool/zones/omnibld': dataset does not exist
A ZFS file system has been created for this zone.
       Image: Preparing at /zones/omnibld/root.

   Publisher: Using omnios (
       Cache: Using /var/pkg/publisher.
  Installing: Packages (output follows)
Packages to install: 90
Mediators to change:  1
 Services to change:  4

DOWNLOAD                                PKGS         FILES    XFER (MB)   SPEED
Completed                              90/90   24570/24570  161.7/161.7  406k/s

PHASE                                          ITEMS
Installing new actions                   37912/37912
Updating package state database                 Done 
Updating package cache                           0/0 
Updating image state                            Done 
Creating fast lookup database                   Done 

        Note: Man pages can be obtained by installing pkg:/system/manual
 Postinstall: Copying SMF seed repository ... done.
        Done: Installation completed in 526.523 seconds.

  Next Steps: Boot the zone, then log into the zone console (zlogin -C)
              to complete the configuration process.

root@hipster-host:/# pkg -R /zones/omnibld/root set-publisher -g \

root@hipster-host:/# pkg -R /zones/omnibld/root set-publisher -g \

And then, probably, to get something useful in that zone, I'd build it or proceed to install "pkgsrc" framework from - but that's another story.

Finally note, that for pedantic production use you'd create a new zone brand referring the customized "pkgcreatezone" script (leaving the default packaged one untouched) and the standard copies of other files, similar to how the "OInlipkg" definition does. As a lesser evil, you only need the tweaks when you create a new zone like this - so you can revert to the saved "pkgcreatezone.orig" copy after installations are completed, if you're concerned about this.


  1. This comment has been removed by the author.

  2. I have been reading for the past two days about your blogs and topics, still on fetching! Wondering about your words on each line was massively effective. Techno-based information has been fetched in each of your topics. Sure it will enhance and fill the queries of the public needs. Feeling so glad about your article. Thanks…!
    magento training course in chennai
    magento training institute in chennai
    magento 2 training in chennai
    magento development training
    magento 2 course
    magento developer training

  3. The development of artificial intelligence (AI) has propelled more programming architects, information scientists, and different experts to investigate the plausibility of a vocation in machine learning. Notwithstanding, a few newcomers will in general spotlight a lot on hypothesis and insufficient on commonsense application. Machine Learning Final Year Projects In case you will succeed, you have to begin building machine learning projects in the near future.

    Projects assist you with improving your applied ML skills rapidly while allowing you to investigate an intriguing point. Furthermore, you can include projects into your portfolio, making it simpler to get a vocation, discover cool profession openings, and Final Year Project Centers in Chennai even arrange a more significant compensation.

    Data analytics is the study of dissecting crude data so as to make decisions about that data. Data analytics advances and procedures are generally utilized in business ventures to empower associations to settle on progressively Python Training in Chennai educated business choices. In the present worldwide commercial center, it isn't sufficient to assemble data and do the math; you should realize how to apply that data to genuine situations such that will affect conduct. In the program you will initially gain proficiency with the specialized skills, including R and Python dialects most usually utilized in data analytics programming and usage; Python Training in Chennai at that point center around the commonsense application, in view of genuine business issues in a scope of industry segments, for example, wellbeing, promoting and account.

  4. Thank you for excellent article.Great information for new guy like antimalware service executable

  5. keep up the good work. this is an Assam post. this to helpful, i have reading here all post. i am impressed. thank you. this is our digital marketing training center. This is an online certificate course
    digital marketing training in bangalore |

  6. This is really too useful and have more ideas and keep sharing many can explore the many exciting Job Opportunities on India Postal Recruitment 2020. Latest and also upcoming Notifications on India Post recruitment are updated in this page immediately...

  7. With so much overstated negative criticism of the corporate culture in the media, it is indeed bracing to have an upbeat, positive report on the good things that are happening. Wish to read some more from you!
    SAP training in Kolkata
    SAP training Kolkata
    Best SAP training in Kolkata
    SAP course in Kolkata

  8. The content is well recognized, so no one could claim that it is just one person's opinion yet it covers and justifies all the valid points. Hope to read some more work from you.
    Data Science training in Mumbai
    Data Science course in Mumbai
    SAP training in Mumbai

  9. Data Engineer: A data engineer also works with large volume of data cleans, extracts and creates sophisticated algorithms for data business. data science course syllabus

  10. This is my first time i visit here. I found so many entertaining stuff in your blog, especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the leisure here! Keep up the good work. I have been meaning to write something like this on my website and you have given me an idea.

    data science course in India

  11. I just got to this amazing site not long ago. I was actually captured with the piece of resources you have got here. Big thumbs up for making such wonderful blog page!
    Artificial Intelligence Course

  12. I am glad to post a worthy article about the German Language Course and IELTS Coaching from KCR consultants, this may change your career growth and language skill.

  13. FOXZ88.NET online casino website Global standard 2020-2021. คาสิโนออนไลน์

    Betting online gambling reminiscent of UFASCR.COM Baccarat. ufabet

    UFABET football betting website, the big brother of all UEFA networks, UFADNA, with an update The first modern system in 2021. ufa

    Web football i99PRO online lottery casino apply today for free 5000 bonus. เว็บบอล

    Kardinal Stick Siam - Relx a great promotion. Express delivery in 3 hours. relx

    Online Marketing Company By the way we can make your website. Reach more customers directly to your business group. Grow your sales. SEO

  14. Online football betting i99club, one of the world's leading online gambling sites, provides the best prices in football betting เว็บแทงบอล

    Ufabet1688 online betting website UEFA Bet is a 100% legal website with all licenses. ufabet

    UEFA football betting, casino, slots, lottery, direct website 1688, stable financial, 100% UFABET168. ufa

    Fan wreath shop with free delivery, with pictures before-after sending with receipt. พวงหรีด

    Sticking to the COVID-19 situation: โควิด

    Online Baccarat FOXZ24 Easy to apply, fast, deposit-withdraw 10 seconds with the system. บาคาร่า